From patchwork Wed Dec 6 17:28:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482101 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AB8C0C10F04 for ; Wed, 6 Dec 2023 17:29:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=OVYUctCEH6T1eLWoJXOoPNWz/b0i3CAPT154Q1ECU4g=; b=D/CZalXVd+PJzQ w+gnOeik5IO5igXABt3++Y1QrHu8VywvcFdUGcz2ER6zi0j53FCNu29LCw97IF+66gvQokyUN+2bz PvdMMU0EgsM7um/93fNiJ5C8ps+YV8DZ+hxssg9Qd9e+oY0t+ZRCfB6gNJWXKURjtaYvhaHrOJqfY FFMndillqQmknozPeFspY3wRHDFl0ONNYR9LWVD9xD+Qq3yRgAZzL6YzJigtqlcr/q47YBhS+mMnM Ia65ZNR9pJzhAvKcD9xy/H8qEVfptlDx0YY2gFEXzLSmJfWyjmgCxAmgkF+mSynGLehG5XPHBUUQ4 2sEkErvnGA5EQ71+/f5Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhj-00AtWP-0B; Wed, 06 Dec 2023 17:28:59 +0000 Received: from mail-dm6nam11on20609.outbound.protection.outlook.com ([2a01:111:f400:7eaa::609] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhW-00AtIR-0I for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:48 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=V9x7HLZP8tkQM3r3fJ/xzzgbjMeqbiiblI1xZfLQMUbl5Dp5LfxD5yQMtVx1h6nnHv+PaI9BVd/Ye3SjuBcnuBb6b+Vc8n2lMF8XobD/aLpAjuEnqzkiJfVNgTju4RGu00wsWhLTMwJThnoplYrNAZi4Zphv0cRC7CTVCAHuEGSZP4sPt+UQB1G09gdgVV8BMq1m0sKJ8a+8tDCZuZq73Rd7YndYERk/MdNtwoPX8plxkDUs1obKj3wIK0GLRlnDI0JcEeFZbCINJ97vnF5QExK5xB6c8BFw0XI7DR7iteuFwudjC6uV9TEEP/KZ8Q9uvaXZmDbkqDkLrgcYi2sIMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4P0uOcMEkquT8nUC4TUk/YzRXYLI2mKWd4hhBUDzMRs=; b=UB2WsteKBJy7Xhp66GyJsD8phZ5yFzz92DUdsWi0b0YDm4lW9TVXp1wvsA922xs3wW26RAa2yt54PSCWLnDTtstaCR94ZI98fWykzaEqzbwirKke2dNLBmo24xcBLfQbeaInkZlZFmAx51pi7ou+P+Oy8jSMHf2o2LWuYQ8EE88590IszfBxAzWCA1v2NekRrOI3X9ZOWhl2uV9tEDB+FQCvI2PxM/zhKx5hNkDM4idMjid1OGpiQiuEHtyc9uORnb3driukNyN9wguW30OqBgzTamC3q9Z5itgkY35/6UqYx12H+GjDPs8Xv0ozplrYIUY6dDTfFc4S7J9wazmWrQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4P0uOcMEkquT8nUC4TUk/YzRXYLI2mKWd4hhBUDzMRs=; b=GMia8uFPllH631cANib1m++xOh+zrKjT1JtXlOOaMTG9Rg+RTOdiAqJ1VdXu0m6SSpS3wdwRM/zwm2Q8FWi3ZCYsS67zRFY7JLcRR6cHCPNi/PdgBvF1kbImYQmfStfR0xERiMu9KQrXJoaJETc/1op93qABL579RYM2aMI+0ByrMmMpGdmv2reIqa0786Ri6UKg3jBVUkQERE8qsPRoK5a+/mMC9uDizHUmHV+2jyTtnDUYR86aOVfnmfO2sQvGgCuLULvb72VMbd47VrJKBwYsW0n4QGwjZHxYImq55WA54eLE/JDBgH/4siyIr/ReQ+7c5lpeKF+/eqtlQcYLtg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by CY8PR12MB7364.namprd12.prod.outlook.com (2603:10b6:930:50::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.25; Wed, 6 Dec 2023 17:28:35 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:35 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 01/27] iommu/arm-smmu-v3: Check that the RID domain is S1 in SVA Date: Wed, 6 Dec 2023 13:28:07 -0400 Message-ID: <1-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1P223CA0020.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:2c4::25) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|CY8PR12MB7364:EE_ X-MS-Office365-Filtering-Correlation-Id: e88347d4-2b4b-42da-37ab-08dbf680c5db X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ejgc5lL9+XflL52xYkIxg/96rtDLQ/F2rJOozAgGAe5WEXqHP//KVBNimP0r8CDzQdrAonY2cF3NDG6VYWwpqbXPv7JP6Kzjy/PuqfNrIAxAl3TR7bIZQ0dtzq2LmiikeOwmh5eBYUKNfoQo92y8tLmOXs7atVdk+5Rc61F8oUXm3YTRVWLtZwvjNmCHfWpXbWKbR/H32k0tganvhJOy55z51j29xHIqAL/3r4djKCPRcrDNVBbf4EeblcIeU8Ls38KOG23PPFkALJI27dDTeeofNSajP9VecuavCf5z5nHWkD0OL/Gsrk5VtZfSzigUTgZt4RsYreR3+gk7xlBaOaGOCYFu1MQubvarBdb1Tm9D8BhFqjbxwYePhRPTpi9oaKGm4it/baxuZCILUlB+ufoqXbQwaku2uS/VkYIoejIqWkRHB3aO/SKe+3nm3JBKA30vIYB1qCr1iDzcqSSyBezO3K5FcztXvdqNbTkLOqzqyfCxAYja/IFdCZxC7UfjMlthvTqD/124VS5QGKbSNDSbM5gBiW1pf0GuAENPnNP4f9Cs+8uA7YH9CyH/hNtmjmjWIimC3PRWyav1Wf0w7fMM22iInGOlFmlnhenKdNo= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(346002)(396003)(39860400002)(376002)(136003)(230922051799003)(64100799003)(1800799012)(451199024)(186009)(8936002)(8676002)(4326008)(2906002)(38100700002)(83380400001)(7416002)(41300700001)(36756003)(86362001)(5660300002)(6666004)(26005)(6512007)(66556008)(6486002)(478600001)(2616005)(66946007)(110136005)(54906003)(66476007)(6506007)(316002)(4216001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 4JJ6H8PnI8TXwjavVbzolKlXyZe0/OUu0PbVsVN6xgmDNRnqVwIaRvi5hfaNOBKTvOZ4xOOr92aR6xK/iUyRkfJqO4N1KmOINtMn2b/tXWd/allKXXUJCg5GL9IOT4pwVBemlWbrTnvCdfyoAXhxGFxswl4hCyhuAvy4af8Z088Pkk1gdW6Q9xRjThhYrSUSI1LaIi6SK+UQev3EWx5JG2yqiOWW4G1AgmUYeOWEbYNIIUiyAlZOP3gTNw+v1pl11keK9D6BLnpe5eo4vFvpvsezxN3lbm+OBorao8mtVdtpdInv5DdhW3BDbVfpRwLiY1l80Dgud+bIgih3fA8tAh1E44DX4lxk9eYu3goeaZMhGCcJYCJI+arSy6Zw+8u+r86nR1FeuBmadhlEfDrXQgsu313Tz7rfoQ/TZ1M69uMn1gb+12gNyJXP5wiQt4EavFL7XpOy75LK2EHPABC4GavaQ2JONY3onFFg4fRph4IZeg4sPTY7PNjSTX/l/z3YoJmJVyhK4RS5rplu/rSKv6Ej4Zc7kSpcbRKkqLWKgoKtO8g+WLWAKddbuj3TQYKvF1hbZAvOAOHdI7mS/FYDljitfR3uiIkHuFKrQjy2FSI2uw4DH6NZ6dQFOU9pjUznp7NRstnHl7dd1B99T7tWJ3d/fVMq5HAKoejL8Mo+HcnQkvkJfTeIx0xtWWbE0GFHArPNGkN3r1P/exbetkUQWnPDoqGZ//qgt1tzjU+RWh6F/ih0pzmIbvsAvtR6v4g0VdJFbt2RHQX9pJXC8hnhkPdDSa4ODVFs6fYxjNC5LPKkc6Wewrsp1xyqeTKD+NqRB9AinNJ4qopNDzafCXyFaumVVVWmPxAOk9/ts7tEUgNmH62fGwr/64k/XLQHWsmdI1+8lcphG3I2r7pxCJEZk3VnZXqI1tpnoojzMFfwgF9O/vobOvWzUA+MyCeZr5fIAHerLFF6yWenQR2V21Fv3tkwQB2mucaw+Zy9XE4QP/Ayi991zx4WilIsMv3BuRx2gLpmz928hjIOwFqeFWSXf30BO6CVDjhBM+Tvv/VhfV8qxN0ZdQpvhsHAyV9NiOnwG0TczzB3JHmIOfb5WiXRLBuSErUiO17K4NyH4Y6fyGpyLEHQEj/KmPjn/tZb0QZXgGW7hVf8EL9h5yFQVfOvUIVWWstfUkr1UUWFJcSGPvirzyQI/GEyDYH1Hgy1LD+tedB4qFXnfTcmZDIlsw2W8M5AFjRCQcislpibK5IGVp3GSE/N45uNNkC4e/Su88sdI+LHsXyv4HR87QyGgsxN6pW5ZgUnPev/PyYD13q2Ev8/+F5Ojy6zDdC+x3uDiOMv7g03v/3tekQ0neonte6RL0xsgbMB3R3mGniQOvyrDJUe6REeO2g9JOTLb3/eD8bBAo5ZyJT53NQnzium98AlBhi21/uBdGWa31nZ/TIlW9QfVQxpKTrArA0pF5wo5rE+kioAPq5Z/gsppFYTTjZB4wN07a4nxFcy7ljBZBQ42Mw30j5SQ5vH7wQAR4qZ5m0JQt5cD7cfsoUAZ6dBLVuEf+BoitRc2ZQRFGbZqAH8+C2Ib//6HkNsXTIF+SHOotl9 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: e88347d4-2b4b-42da-37ab-08dbf680c5db X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:34.0356 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: RHveC0IHE+AmTLcsXxVWW8TfaxImugZsflEBIpNdPxAajikY8azhJJFBtuxAjLEn X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7364 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092846_153704_A97FDB9A X-CRM114-Status: GOOD ( 12.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This code only works if the RID domain is a S1 domain and has already installed the cdtable. Add a to_smmu_domain_safe() which does a robust conversion from struct iommu_domain to the struct arm_smmu_domain. Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 7 +++++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 7 +++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 14 ++++++++++++++ 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 353248ab18e76d..bd0566381c58a9 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -379,8 +379,11 @@ static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm) int ret; struct arm_smmu_bond *bond; struct arm_smmu_master *master = dev_iommu_priv_get(dev); - struct iommu_domain *domain = iommu_get_domain_for_dev(dev); - struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + struct arm_smmu_domain *smmu_domain = + to_smmu_domain_safe(iommu_get_domain_for_dev(dev)); + + if (!smmu_domain || smmu_domain->stage != ARM_SMMU_DOMAIN_S1) + return -ENODEV; if (!master || !master->sva_enabled) return -ENODEV; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index f3900a3d52524a..0f3bbc4a2addc8 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2486,14 +2486,13 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master) static void arm_smmu_detach_dev(struct arm_smmu_master *master) { - struct iommu_domain *domain = iommu_get_domain_for_dev(master->dev); - struct arm_smmu_domain *smmu_domain; + struct arm_smmu_domain *smmu_domain = + to_smmu_domain_safe(iommu_get_domain_for_dev(master->dev)); unsigned long flags; - if (!domain || !(domain->type & __IOMMU_DOMAIN_PAGING)) + if (!smmu_domain) return; - smmu_domain = to_smmu_domain(domain); arm_smmu_disable_ats(master, smmu_domain); spin_lock_irqsave(&smmu_domain->devices_lock, flags); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 154808f96718df..6f62184eaa2434 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -740,6 +740,20 @@ static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) return container_of(dom, struct arm_smmu_domain, domain); } +/* + * Check that the domain type has an arm_smmu_domain struct. The global static + * IDENTITY and BLOCKED domains do not. + */ +static inline struct arm_smmu_domain * +to_smmu_domain_safe(struct iommu_domain *domain) +{ + if (!domain) + return NULL; + if (domain->type & __IOMMU_DOMAIN_PAGING) + return to_smmu_domain(domain); + return NULL; +} + extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; extern struct arm_smmu_ctx_desc quiet_cd; From patchwork Wed Dec 6 17:28:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482099 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A6C7CC10DC1 for ; Wed, 6 Dec 2023 17:29:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7gt322HILBZKKtFsKZNvvHq9I4V+qvSZYN81B2dlMz4=; b=vwQKdbqour7idy h37XEpd4fuMy9yuIcO6dzNpoyLDGV3a+0e8fOPNiAfzaOgE2i+mYEnVyvT0F+j5KHSvpPGmSJezmz HpE4IwqIJuRhRblbCPLWXm5XrIgVP07Xxio3fj0YTCZh0O/GPR07ZP1FEHjQd2bfd+sQo2zOVmHkb UDT1oODNuD/rmI+T2AD+X6heqWeP2fCgc7VuZM6rDZDj5V2iwau8bCKJ8NRS7XZTeadeCmVxjN2AW 5+/PoFSNmMgSfJgco2vzh7F29ZFfmKmYDhe21xRDbnlM85nZVMI9zzU08cbFG8bw9032Y2VTN0DUp gC8xsftQD8vgxOsE1ZAQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhU-00AtHa-1B; Wed, 06 Dec 2023 17:28:44 +0000 Received: from mail-mw2nam12on20617.outbound.protection.outlook.com ([2a01:111:f400:fe5a::617] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhQ-00AtFo-2Z for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Wlfr4zzzuDxq6CYMIgdHBR7VC7tCpS0vIy7GFvBOIfqVzAaFqQIoBb8vYwlY1/Ma+R2b7glkFKJEYWmuc+leOKsGfvSaFynq+Brm5TGP+pszqfWZZz3DdBHaoCmnihHX0aPXzHlHXNjCw8BD08riAWuTO30GRJignvn63dOf4QwbpzacWVW0Yyc/Uy44hf4smeMwUfz3yBZtT6nB6St349kjR8BvGybEE9QsGfr4hdfUCDNVFgZ+dVDpFR3bWJyVvbyRGGz0FZUdpjsR12JNhP6IlcLmbSxVcAc755JS94mFaW4rVRUctLgk96+4lkx8kWiw5k8CMRRqFN0zUrnReQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HoDRRFaLriAMSIcRQb75wliv4+EcjymP+iY+ayLldZ8=; b=UONre6jOmXRHgRb4z/UiVvBa4c5IVRDn23bgKfbosdhwSCdqOCG493XAAoLTS52R3WpOu/ruErqYr4mmiYFKOeVbpUY6XDuOFQQNJCT+XprK6AxEKtcgpRY3Xbz8SVwkwDIY/HvvpofnA6jYc/oHFY1wxQ4G4tacNevPAvBL3HAIGfaDuDOXGlcz+2CvLuDqo8x6swRk2U2bQB6BIb3U8C/TCxOIRRApCB0rX2aAzV4o6JK1UBAuRGieEUY8/e9oQKn2U1Kp35elIRx6/aSX2CV3BCEKYOeGLGgwWKUlsR0+usEsSnZxH+hdvt0c7ljAJcQOT9HKJOw7kmHRfQ3JjQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HoDRRFaLriAMSIcRQb75wliv4+EcjymP+iY+ayLldZ8=; b=BlpPdMcueNdZWObCDJFqk7S5in6grPD8yY2cBTlm+6WotfBdvOBr4MfcMNDHroJR7a1x/O2bq/+XvgZhLWHm9zAeRXLGpXUd+gXmFI/V+WHZDIOD60Kgta7DZQESqsWYsEqcXo5VAZaySTnWdzdxIzrxn9abpvBpDSQSxS50Ro3+nRQ+sa5kOyaZPi2ExXbgbuFREOu+eyhDHHTi/o6hAbgdvaIN9LQN52tzy1HCcOAK1ywZ5A10G0MdZhgyk97JSBWtjrCaPONlEKVHeUt9tDwUUB2crErP/5AG7gPRtZNkoNwBBnzYC5w/6R/F7UhbU/Sj87P7Bo14+iw1ba7+tQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by SJ2PR12MB8806.namprd12.prod.outlook.com (2603:10b6:a03:4d0::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7068.25; Wed, 6 Dec 2023 17:28:33 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:33 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 02/27] iommu/arm-smmu-v3: Do not allow a SVA domain to be set on the wrong PASID Date: Wed, 6 Dec 2023 13:28:08 -0400 Message-ID: <2-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1P223CA0022.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:2c4::27) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|SJ2PR12MB8806:EE_ X-MS-Office365-Filtering-Correlation-Id: 179b6efb-2903-4646-d6b9-08dbf680c5b9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: CGFGZ2cK2KnG1laqQ5jp5O95UDFPZ5jGKx21/fMV9CEOtL/UZ02Hn1plfWZfEJrgjF25U62rb4aah4485wFwnFxgUT9C26AIIy26+kpanAgjNEUxraNRWdkPWMTXWU9CCg/+IBbOdbq3EW956DBuGsh5nVpg/AB1milMHPjPTX2+zZZp7+BGz9/faOtDXWA6522QC1L8OqtA7Y6JcxCfP17Khdj95cysccZsPxUO6AlVucgNJoU8HS8H6MpsUBSzPLB7oj9NZxgMB/dDagKBQ1QAbfFgjZ6Mx/Lpw44TRR2paDDFpa/7aBpYxSGgYuqOAVzmuT5QceEpZ8F0yEE5LOpAQ1W7v/tQeupItl76yWi9EpFgMLXnFQtr7Z3cwjaSUszhKxNnEL/b5qVE0q/JfTKjClkpkUPZvIu0umt2dopQ8QCpJM4mj5cANoKEcHLp8unGx6x5j4qSMAu867t27XDlorOmvHUTqDDEJivbPVegU24M7RrIkHvZ2YxAQPxLALKFSCidXLSN/ZYpPFlQMcZ0hvaxm6MYewaW/bxWNXfuJ4mNAvVGTvwI+UgIl+kKzW03tmhdQnmY+1cuxyFipLB01eoA6HpTJXf6F1EsbTk= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(376002)(396003)(346002)(136003)(39860400002)(230922051799003)(186009)(1800799012)(64100799003)(451199024)(41300700001)(6512007)(4326008)(83380400001)(2616005)(38100700002)(8676002)(6506007)(8936002)(6666004)(4744005)(86362001)(2906002)(6486002)(478600001)(316002)(110136005)(36756003)(66476007)(66556008)(66946007)(54906003)(26005)(5660300002)(7416002)(4216001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: DytV0HXlFmJR0zC4HVyJqE0WIK2e512TwdoWFOGdlGpd4re8ikp/AiSxuJaNEPL3viYakMsrOvE4C9KoAnapwP8o589qfGhR7t8IXs55pwKfuzxKxIx99hAogcW2wVxh4RVe0yoRleD0W8pG+YPZ95iagEUcDGDR6+Usepb2CbO5f6LQ9Glo1dQlMabkEJeRkyUtnMIEGcebGZPJjn5ofpTREHEzRtVrn5U6Uo5hOYmRX8WkhrQibzWSFlUqbh7A1CghahoYZ538rktRMZ51181DIyDsh6X08eUTE2tSxgOdANMoKH1RoHUobdfktJ84sNVgIBhIJX5oKtocVkZiFV5hk9k88AwWLpzkUEgvNIk8Tzn7Ai1h55x2j80gv5f9/XR8xVdx1J0RgLYyxy31pkxfKL9aEsBwXkGSm4RbBKCa6Anuc56GgGR26XX9gw3cZ1NZXAYhh9nvN/K/QnPTWYGS4BNeGH1uR81/YKfoQKLahPEnyIzLJq3P84lTgYln6cu30ma37SVLT07Lh43PDZQ7+9QM6uQ/SJKfM4yuyW9CNWA5NyXVbFvIbvlUyUNDhSnkJt1iBWQNztbfrmGhJlMcsuZsvcPVA/sfaheWGzJunAdU/CUVvopAlDo2ycqpdO8suNXor1T4+Bb1XAmmJvwaxUhv2xeN6BEHhEJp6B4MJzPB4b6eqY3YJfpT5VvfwuhhL461w8zNsrxkhd45tKxo0IvtZabihue62k1UYdv6YLA2R5Ue23GvgQbApn3GU14rRaTLwgSe7kWRYqNhU+TXcGbsthJYqqTOrd0VHDzfiz7lrkHyogh5opYU1bRnyYgiMnzruENdEALtxB2frBQDTbEX0BBmg67bdr9GrOFXB9qg4F27NKv2ZBI+HG83xXOYVkg983bVDp93qmzOmLoFWjGfiln5jHmFtCDrYHsZ/torAHqk3u+uq95Zkx0BVY9bzcIl/It78/oueWinCDpJ1YDza5pmhBUY7jKMZaGx2jF1wxtLBqUV+rUo2wxXVz/lrZgAqnSfi/sLupBha7pk8EvXSOV10aCIwKltppCuOhmNO1XFzUZaRN4YzJvlDE1dEk/lhYC/TufoGVC4nt0xz9NClwLp7lSPpN94AuE9kWAHb+k8LMc6V2ADTIKLweGp8ntCEniP2QJThKbT/VMVS85yud/o2hDd/ikmVPSRHgqnhgdXcSzyOROuZo85G4q2/AVEJjOEiiPhGp/mlxy4jndrA1bKxNMdm2TFHYMCT8fC2QFFvn54FAnTn223P09lhqmiYJ5sGqphWOcqWyWAEWyosva4McDs7TQwBFDgLL5GDmml24LAwWkEs0Mctg/8iAPkieOW/o1rsGef3nPixLONcjdCFI8YVLckA0FKl5IdE3qufedbBr+HEfPf0q1yr2rruwgedEeyjCxOoyku+/2R3Gd3ucF5PTfswH533PZXDPB6sT0scvJxoCXOoU7FsXnivkmn0659A+IELl4Zt8H0d171lOMloon2VGgPgKadLvUw1q1ZsTTrXjepchS9P5yDw8AgEOCnF7tmZQQoBtQufpAaUsxjuPpr5dze2wV5h6M79sN+93Wlgghq X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 179b6efb-2903-4646-d6b9-08dbf680c5b9 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:33.8018 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: g2HPnxKWMGVfAyFyfXE8J/r2F/m3pe7SUg9H1Y1QD9B4yH+VVw3H2y7fQFBvXz91 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8806 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092840_836693_F6B2CD6B X-CRM114-Status: GOOD ( 10.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The SVA code is wired to assume that the SVA is programmed onto the mm->pasid. The current core code always does this, so it is fine. Add a check for clarity. Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index bd0566381c58a9..ee3d148aafa26b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -585,6 +585,9 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain, int ret = 0; struct mm_struct *mm = domain->mm; + if (mm->pasid != id) + return -EINVAL; + mutex_lock(&sva_lock); ret = __arm_smmu_sva_bind(dev, mm); mutex_unlock(&sva_lock); From patchwork Wed Dec 6 17:28:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482106 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 994F6C4167B for ; Wed, 6 Dec 2023 17:29:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=VIFMKBOt0oekVmBGXDJLU5+3mWl41K5iXg1ySW9upAk=; b=NWdZbD3nsoVyZj 7UpNqT00uIVPPLHg6dN4Z4uGj34XPLW2Do1HIJYMTV0Nhjz+GNbrlADHYO2v2Gyd79zn878qYnNzL waXsYFIHbfIRa/NMkzh94x0DV1LNqGlh3dhMPnRa6weOF67iqyTIsDM2QWZdOpanr4sEIgkCvVTW9 07gJ4euEhRLzMKk1sWkRsW5lSg42glm+a4EXbesDcTk2Wx/L97fwNXxHfy50WZTbD2ph6Re2OWVhM JMsz7bq6C1/OGSPe2+7atltNYooRlz+RAN/x1WZbCocNcC/cFW1UQw/rw0YBoLw6ITYGBIdOH0Ubj yNkJmsPlM/4OmyQeAmqg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhu-00AtkJ-30; Wed, 06 Dec 2023 17:29:10 +0000 Received: from mail-dm6nam11on20609.outbound.protection.outlook.com ([2a01:111:f400:7eaa::609] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhb-00AtIR-28 for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:53 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FnaykiL+3V0CzPouXW8rypeDadb7UwXS+onhhrc+kHFTcQu9BSpcYjg5kgRkXdH2sf4qKSolKl3RFsvOU5m9xS/oXl8+a2qItlWJRuvkPGA/sFKQWbl/8oY4gAK6DlHSnyWdMpFqzFfn3EuxJk/mviEzzbUCInBShhxIDyZdjYYjxTSAG2WMfRxfcxYrgrLLckdbSeHEiHAyMRkcKpqkXftQFucrZDDs96EzpGTds250WuvkW+RHN/2Suzc8Jr/rA0TGysg9beJHeKcw1gIbIxFP3iNTdnbe+KQGRtQh0OmdV7R5bMi60zSx/jqHl1A65yf40kM8mRAPaKE+3FHkJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xDzepfs9lFCdgqCgl6iQJZ3Z0CSeZIsWdVTnV/y9GA4=; b=EXrhXzUqVYhzb6uWcHKm7dueNoqb4zfCjWQtupC66cORQpda3RHB3/ZFAIVfTv4gQj/NkORib78rOeYzrCXAP6cJN4zgfoY7vk9fRfuoZ2ioynYQFy+rqIJrYx3XRUFEGn3Rz3JOA69n1f9yGTJCjrZJzQnr35ncPNEMTK0yzYKy1U5jTHBQub47hrYyAScASvDoFiUWmctyXaKvf/ozUkO4/Wcnl2fFu4lxyJ9t1o6TVh/3DMTxB5d3QBMpzIrWOxSw7E9bmZthwN/s4/88v9qpjJqsuCB19H+9PL8I6rpbDU07k0fdrcDw9F6hJ/4K0/OFsUY115ZgOG5OE6NPfA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xDzepfs9lFCdgqCgl6iQJZ3Z0CSeZIsWdVTnV/y9GA4=; b=QV78DX+SceRPJcA1u5kpgSQGYyr2ORdLw4XPKgR12gmOqM1IgPUmW/WBwzHIUSpbsRH8uCGF+XUP29q0VG4LTv7vV6Wuj9v+UOYzC9NLOF72itVgnw+ffyYFmHpeN1Tnvb+wDPRvzvuLNE8huqaCOzH7A/iE+RBd/Lmy0o8inTJM8n5QVK+GKYI1IyCh4KdqV7/5jt/C9sexqZFPZcegkx9fccCR7kAge8HFkzCSzsThhbeAH4vLYlNoezSRtcmQa1kK8GSKTkBbI5CvdPnixxS16LaOpFIyM8FzkbygE6qbiyBrhQVAViyBnvo65iZdephiZUhHPd+Ol8xC1DFdqA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by CY8PR12MB7364.namprd12.prod.outlook.com (2603:10b6:930:50::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.25; Wed, 6 Dec 2023 17:28:38 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:38 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 03/27] iommu/arm-smmu-v3: Do not ATC invalidate the entire domain Date: Wed, 6 Dec 2023 13:28:09 -0400 Message-ID: <3-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR1501CA0024.namprd15.prod.outlook.com (2603:10b6:207:17::37) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|CY8PR12MB7364:EE_ X-MS-Office365-Filtering-Correlation-Id: 60f13c85-ad6d-45fb-707e-08dbf680c614 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: EvUj6DwVkhYM538no2SmmWC57KtSSLptmAvN0K9O651gZX8bpZBWWc3pnEZ4si/pK2CCk2gsCMdiZZKMsXzEVAAppVNmT7utIaZ9fM23p7pmHQEWxjqsRlt2SkG6jgGoEhq/kt4/pw5HtGn6IWIghHxWLcaLUVdn4rw2+2RTPf2RGMml6NLQvr2UM01J3X+gph7gdHp7H8ABrS/A0DpjEBvruj+RsAgUpxue9bfv5ioiuzu7ThBbY8GhNf63AfiFNSr4puYPYLU53dRxomJKEPN19K+30j9pktClAsCN0QOUsYd+9zS10TIms0D6AMBfmTHDkJh03vnZhZGwtPXkMa8VOnUPrNNA/ylAg1JM+tYo0Tfm+3yzPGkrMWDAU7Gij+DDuILqB2Y7FJ4bDJmOxbvTYQhL5PVM9KpNagJdMlY8manssCdG9cTt2WyqzxGoBPeJP9tAoX87Xrs4hhwjYC7FFaH4QBtTFw5f8g0fH6hIGd10FwDrUw+hYxRob7aSFemhHv95opF2wUERIm4493C5mu2GXszedFI4pCs6ioOIRCF+NJW5xAdWdJ/bixmd8P+Ij10wJYa8PH+lmEPwjbZgpANOm0+Gti8b8vX65x0= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(346002)(396003)(39860400002)(376002)(136003)(230922051799003)(64100799003)(1800799012)(451199024)(186009)(8936002)(8676002)(4326008)(2906002)(38100700002)(83380400001)(7416002)(41300700001)(36756003)(86362001)(5660300002)(6666004)(26005)(6512007)(66556008)(6486002)(478600001)(2616005)(66946007)(110136005)(54906003)(66476007)(6506007)(316002)(4216001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: yKOxg5zv/0eYxFz3TIti8zS3Le4Zs0XjsF8FV5m8aBMaaC2Acm2MV7PrbH+umIHtQO2Jfq4S4Z7bMcaBzBb16QwZdw6h7HJ3lhlW0wBVbOqOUpi+5nZARwmgAzeUJMlpwAF0ZpMQ54As4piB+R4V9Ur5FscZPICnJU5YKrlmKuPLLPkA9+aOE2YvPY0Sg/JTnj9dvrbk46v/MmVfQ7+V0pACFBYJir93rjTOhO5LiHAYs2Sm7R0fyifSnEzqjrHeZF/SYL48GjOpaSE2PXqEOCRkMZV/jWtSWx1oZTNFj8+sE8IIYjXPg6S8Rvdvvt+5Hpj9J0zzJx2XW5njGMZHPi4EFHi9Nw5z8CmSQObe1vnBQZeTafhYXkfW5Y0x/HrVhtRXMxCYNLEPDtsbXvoQ9+9uVcDiNFgyuxv9Ui4LOXPJ5nKhQlJFmxjLsXxhzTJUEqxeZiYWYvgKFj8XDgY3Pc1c1qDzxYZk73b1ZSa3MSkyEdiAQl4U4Mtp4BQO/S32UG4OxjISQKoSGB9MHvxb5ST/ByBdTsdIS/x8X/ZQuM7oA5OJHIQZQmstiueit+iIqzTp0m8/7j46p+WPB933dTL40ePyFTH8D3a+6yoiDyRMxDTk68BQZvuAEK/BbkyCJhNYbpsgj6MsOc6kN0UCHlE+ylWDjvaFsjBmEAI1Qo+mMoqMm6qlrc5IduUn1rJcBJ9kepTcsN3b87gt6oKSWS7WRTTeR9kWhoQnviNfaIFOsGrxiZ0vT4JS8g0+ectHPU9QGlnFcLWQRVMhNoKF+gaDFH/LUFlzHlSOaLKePq8CkOVTSKdh38syGxELy6w4aObpe3PIvJh7nPaD3Af/5Xfto6oe2rirJwrNbCWBkLBkplrPdU4a+5ogcNVDOcoUaV542AIRGXc8yO4BDwYQxjmqrFRuAaNlITBn7TCBEtGpLWGUgboJaQbm/fWo1TclnQ+pnWDHSWXI2f6B8HioLnjFx6lhxK+Lxfb+TH9XooZvcIeK/nTbf9pt2wITgrsU5dgPn4aLa2gQYkax+GxU5m9sJu+CglhsoqSE7fpTtBah8Dt1d8mbYSe/6xoo2FSAO6SWbmcRur51bDzpQey/oJmQwcnlPcEs9EUbEXMci4w4o+TWh1Sxkjf6zpltvpuEqd1i/311gP5YPNzOq95TB11hQXPbokk2C7VRVyVBo64hz5FsTfMvarp2WUPYEzcTzyBFTIj5kzFy7j0efA19WTEYGbnJ3Ei3mVO1Fvvt81XW9lkhKFBQkGXpib/yZ5yhhglyISM/P1/iwJ0eA1f5x2zi10hr4qMIyyObQApRWha+hY7zXWOwPavm4MwcIg9s2eIUdIvzR+sajHfIxeDmB8IPnnofwNUyw5DpnlXuq/Lybbiy04URlRTBR61BGq6wQ3Z3Qk3GghjVDq2IoeiYXkTFzYrCpFedJ1PzXw+cgMyF4VX/hS73CmlfZnUzxVQeNIyYexYXhLv/A6aMlwnS0KPz02CKGDf5w6rSgYgXpb6KOv/gCVRVtRx3WK26DZf/RXLaH65Eb4K9hvsYY3YSLlvLWCdPDi9OhKT43U22cy48JobHYaWkjKZ6mPgROSnf X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 60f13c85-ad6d-45fb-707e-08dbf680c614 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:34.3872 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jfpuS48qIyimOQbC9H7PT3crChRYEgjs9+5CUF5mp0cvh+kr6XIqeRJMerdaD96R X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7364 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092851_719088_ACBB2720 X-CRM114-Status: GOOD ( 10.99 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org At this point we know which master we are going to change the PCI config on, this is the only device we need to invalidate. Switch arm_smmu_atc_inv_domain() for arm_smmu_atc_inv_master(). Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 0f3bbc4a2addc8..39e010c838b1c2 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2416,7 +2416,10 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master, pdev = to_pci_dev(master->dev); atomic_inc(&smmu_domain->nr_ats_masters); - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0); + /* + * ATC invalidation of PASID 0 causes the entire ATC to be flushed. + */ + arm_smmu_atc_inv_master(master); if (pci_enable_ats(pdev, stu)) dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu); } From patchwork Wed Dec 6 17:28:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482111 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 49891C4167B for ; Wed, 6 Dec 2023 17:30:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=TewJQiAtWJVqqyHt3wtbeylwKWqafzSOeuBweH5VLz8=; b=wNL6ZjntAr/02S lv8poUktoHB08nMgp5/8DslxiJl6c9E/+gyfYP+yBE1/48dPxIo7RNCWTWmSVV92BVwIu4U3OPCph 32t5pTSjMVFe7YOi9MjyBYF1OVOe4rjPAzkYFVJnMYUjd/x5JxpeSdfoS82B3/WPWOYKh24b4o6+v rBxMJLIkl8c5G+9akc2WhGIlAx/HBRcvgkMauRHx6zqZPvqD+hU3gua4ew/8AA2Qc3s8Y9Y7sn++e fSnx+4jjgyQYix7rbzC+aSORDhTvH5pZwkz8LgNKzj/j06/dUsNFCYt6lQaoONbsZPNFGwG+V+TxA NQLd5SWXfh6g2Apeix0w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAviH-00Au2q-2R; Wed, 06 Dec 2023 17:29:33 +0000 Received: from mail-dm6nam11on20615.outbound.protection.outlook.com ([2a01:111:f400:7eaa::615] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhf-00AtKY-1S for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:58 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gsPj2/oCLN1EJJHGJIqZeNaCQgblP+Z1SZi/k6zVgDp6cly+0z8exz7/Uh3/EV9ncqBHVWjuGvOWdP4dqCjnrznk3s4hGpeHBlLGAei7qrKI+EbH8EkTenet8Zn33sEgqg220Ha54sCrcwNbKZLHOnfcnj8id/FLGHSHD583Zw8iEmFAT6eo3PAJriyLC+Riub5Fr9gjUnaGa/DrGT71fyJg/eDG+n00EilLuOlD25LG9riM9TTE4vzvEX9Ow1aYbZ+6YW3gLdRSq46rCn6vfH0lJDlUfXDk04UwI0+sh2M8LCjM6AK9adT2GybmtL/NZJdcGtudgENVurrapyPQIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=elGqi4l/h6kI0SCmDoB6mF7m5MDLpscgm04YWeVuV5E=; b=YM74D6M559GT9vPOvR5RAjda312bGPRJv6t+S2jMp0F96aNMPvuYszvC1WJ+FYVFRklSbq8E1SzDISJdtCi99qsJXMsTHjDPKb83lhHc31D6Le2jF97J3I1FtxMJ7XXnuIq7AvBCp7HlwjfUdm5V8KMYLO0NoX2eECNbBZcGvS98mSMQde7tFTWiswqbtlhTTgMKj71Zt/bP6xxsyFDXKAhiq4sWlqPEhDIYXZpzbXV+nGEshCaoGeC1VIGewORHn0HbuCXRJdvvQIHufFA70npbgZBHJJmqxqdV8L1iZA6a1gUJVSGaxBkdaHzOzsc/DQeoiCX/34qlVIZkq6Opnw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=elGqi4l/h6kI0SCmDoB6mF7m5MDLpscgm04YWeVuV5E=; b=MwFo6f8evhujQEg1PHWYiwPDS/hwRhjnfr/Vb/NrS4fUWDAiOYJ6k5D/kF5szzcekQ+rsPQnT5m87cAIqPNDSRpQT5IEsnzHXmwvVLffNw51fkQ6z2McGzo8EzU6L3zPWpYuSPUkyUmdjLDyCric+4IxOrBMYg57+Qc5JkdTJjof+8jlno2DkvmxiLTXilRaN5KAMHxOZQl3p7JU15CU2blS3TRo0elb5S4FFYvyq0wgAYG10YJ7pYLUhfBocY4POIljPs2ygUH+DxEo92JmoypVO0yJxB6pzAuQxbVSSH+zzEawPkTwRvMIfI4Q4lNhEWysXL3oVKjq/VN0XcOYNw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:42 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:42 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 04/27] iommu/arm-smmu-v3: Add a type for the CD entry Date: Wed, 6 Dec 2023 13:28:10 -0400 Message-ID: <4-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1P223CA0023.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:2c4::28) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: e99b7291-7908-4a26-7c54-08dbf680c6c2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: i16vf4RDaAYCAaDJH+cki4/2YZqqasxlxdY7Ld1fROmVtDNpHw0Vdf7C+yWpAf8+pmqvIvtbVXjWGMVuLibXCt8WsyUk2Es8fWOl5Lo5OyGAOpFDb7155qWiEeYcbYgeewS0dNWQNayqUL/9rdp6jL0m52L2fZ5U2pZfa6/HW15eD+RFJKLF7y3JVi+bl+R8qwPqw/tcUr2YkwjSGktA9sjQidfXOWmdeRtA5QL6reIkXlyS3WPm3emKBeIoCt39lxWC9PIIE6gNHYn/2nyX+mnWhtqRacxoT12iy2WqW2IqUiYEq78vTcsrX6ZYRgGUqEZFo83XjlzpUSNB6LyUg4ud3KySNpImHXoGSC8ivL7uR1n6vVEghL+s/ZhyP9sOiADrIWr+6rGbh4jrqD2zFFPjDtxBupYFn24BCAyN3Y4KQ6Xau73XNtZda5AWxzdzPlYFW+Li6IPQQRJaNLg38bKE0lZAp/3mpg1gpB0+fsFxmF73MkViVfd+Fet/aGDkf+HtFnnW/0HDfqtXy5tO/KwwBrDuXlvQnWlF2iEUkatz7twKQPGaPCyBQfiJotwPOeva+OtWu+YiHQD+Zf4N+JUbchg4H0MFvJTGVGqG2ednw7UXvyKFATHHBEoWCoPzOrTKW599vtG7An/7ZYq0Kg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(4326008)(54906003)(7416002)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005)(4216001)(473944003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: s0kjUqqEekrJKHsTdkkG7PIW15FVGnD3faauYhiCsmwswMqcrJdFMRoi/wXX2anDvlFkpltka0vzrNKhpZoSu7NFn0fLesMOxHcbjTIpJC2UryRQ2VOFhhrAFoaF26szDFRkV5NQM4fOyNxrbLZnHddFawz6o0PeE/2KNihbbi+tuffB1WNLpHU/6GTuT+S5ROrmiwUn3Q2uGaVlREQbVOme8Hycq+v2ofJQGvglUtRrnS5W+B3//spX2p0d9YpUsVhoVGCat1gUcreRYXH/nNdhhd2W742DyxqriQPLyRtNDILeaQO/Nd2GFhFTNms16N2gARoPPW+QF3G1Hxn6/3y6wYejY4MhD/7sZB0+cZLHe73Vo6EznHxwUMvthFN90Xf9nuZJJZmDC4InsyVXxm1/B5mZdluEQGmByrU/XwLk+A9vXyPT4YfruzEdoTgi/ATsFuuOUu2jA3xZvPXk9Iujg7lYBkr6C+dOZ6inZCUKQ3OZDtDUmNllrvdCMahqfX5o9cH4gemj/c2dxN8k5XE7MgLFkb6Ckt5G4C8qyAc9/ffEw4VirvyqbaRZ3YYl5B2dtsQDfQ5Rdxb5AV9iISGPf375AY2RE1heDqS8frt3ZLYLr3WWHKKmRVERPL2KVRd4JBW93hNsvNBL5jn6NkQeRDXXuri5sXm+d9k7xjy0kvDPcI6qZ4pqvz88fPTAzjVJYyLw5WXk62JEMxGiZ6+6VfVhw7zZy/QyK+2OSCpW+bOwQGBF9rSDrnVPvMijgi8rh/B2rXsfEvz0diad9bc106ZprvAjJg4rPPbYO43efHDe8IPbRcAhZkVOybBya5xhUm25/KRnpR5N9Lvbky7In0xVo7XVDp2AASqwtlArWOyfIJ+OKyLsQ7lLKIgHRg5LMUcaatJ7slu91SAyIEobO4v+vrTKScZFUsXMtLpGXhPnemfRmLyVD7zcE3YmdhLnTXsvK94sHe6PdTJ2FZFSgN6/LYUg0U2s9oPSz6CogRUybyegpDsMSLUD/wUiwjvqybgo8zJdkLBkGnvhHNSgvY5eG7tgRcLAarJqkVwy316+E28TruUoHadcg7ZiDFiby/DoE2RfK6t0bsCvIwz7H93kBLmFuGigJR29vkHBMJRQZ7mUktA1d/hHSktakoMvd4HjFsGF0UoV8WSef1ixBaPX1OSz3Z0Ipn/LO0yjDAAQJULw0r1vDJirFDAXPLAhgPQYKyYSs704kapGSo+fouzXHeUoDfYYZ3By1YGVG31kx3PIaCfoqfZhhmWAf33fKHiLZalKxzXTxAh7p0LfVz62ovvTPDnM9593t8QcNGR+2KtOYzUiFmjlvIxJ0iD3yg1REuSV1n28iVTQGkzR+ewKM1OXnRGFcjnn4hLwaDKcfeQZhK5Ymgyj6dqk1yS8vSspE+pRAMegoJsjKWeEwNmWFRrKKetPlh8jMZXhjPuvS3v0QYk5H3HlUeeEGbuyJ+fWgqEAp03Mo1j/IIYF6Kr70WwD2UNgVEjDc/OTMFmEqf0E38zuwogflfwkKeOpm9nmwXmSTwLLxRPtga30/r6jpDMvw627c+RQPo1lGxpl90wPhHh8bPCRN64G X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: e99b7291-7908-4a26-7c54-08dbf680c6c2 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:35.5292 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PtGleTDnioglRv/V7UzO44dYu1d9y4w+i294WLKjiyZj+T8lqI9VNZO1XZXje5+d X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092855_511774_D3CE0C77 X-CRM114-Status: GOOD ( 13.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Instead of passing a naked __le16 * around to represent a CD table entry wrap it in a "struct arm_smmu_cd" with an array of the correct size. This makes it much clearer which functions will comprise the "CD API". Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 +++++++++++--------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 ++++++- 2 files changed, 17 insertions(+), 10 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 39e010c838b1c2..3f77583144e649 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1118,7 +1118,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst, WRITE_ONCE(*dst, cpu_to_le64(val)); } -static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid) +static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, + u32 ssid) { __le64 *l1ptr; unsigned int idx; @@ -1127,7 +1128,8 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid) struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR) - return cd_table->cdtab + ssid * CTXDESC_CD_DWORDS; + return (struct arm_smmu_cd *)(cd_table->cdtab + + ssid * CTXDESC_CD_DWORDS); idx = ssid >> CTXDESC_SPLIT; l1_desc = &cd_table->l1_desc[idx]; @@ -1141,7 +1143,7 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid) arm_smmu_sync_cd(master, ssid, false); } idx = ssid & (CTXDESC_L2_ENTRIES - 1); - return l1_desc->l2ptr + idx * CTXDESC_CD_DWORDS; + return &l1_desc->l2ptr[idx]; } int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, @@ -1160,7 +1162,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, */ u64 val; bool cd_live; - __le64 *cdptr; + struct arm_smmu_cd *cdptr; struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; if (WARN_ON(ssid >= (1 << cd_table->s1cdmax))) @@ -1170,7 +1172,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, if (!cdptr) return -ENOMEM; - val = le64_to_cpu(cdptr[0]); + val = le64_to_cpu(cdptr->data[0]); cd_live = !!(val & CTXDESC_CD_0_V); if (!cd) { /* (5) */ @@ -1185,9 +1187,9 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, * this substream's traffic */ } else { /* (1) and (2) */ - cdptr[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); - cdptr[2] = 0; - cdptr[3] = cpu_to_le64(cd->mair); + cdptr->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); + cdptr->data[2] = 0; + cdptr->data[3] = cpu_to_le64(cd->mair); /* * STE may be live, and the SMMU might read dwords of this CD in any @@ -1219,7 +1221,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, * field within an aligned 64-bit span of a structure can be altered * without first making the structure invalid. */ - WRITE_ONCE(cdptr[0], cpu_to_le64(val)); + WRITE_ONCE(cdptr->data[0], cpu_to_le64(val)); arm_smmu_sync_cd(master, ssid, true); return 0; } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 6f62184eaa2434..24a77e0a97898b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -282,6 +282,11 @@ struct arm_smmu_ste { #define CTXDESC_L1_DESC_L2PTR_MASK GENMASK_ULL(51, 12) #define CTXDESC_CD_DWORDS 8 + +struct arm_smmu_cd { + __le64 data[CTXDESC_CD_DWORDS]; +}; + #define CTXDESC_CD_0_TCR_T0SZ GENMASK_ULL(5, 0) #define CTXDESC_CD_0_TCR_TG0 GENMASK_ULL(7, 6) #define CTXDESC_CD_0_TCR_IRGN0 GENMASK_ULL(9, 8) @@ -591,7 +596,7 @@ struct arm_smmu_ctx_desc { }; struct arm_smmu_l1_ctx_desc { - __le64 *l2ptr; + struct arm_smmu_cd *l2ptr; dma_addr_t l2ptr_dma; }; From patchwork Wed Dec 6 17:28:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6730EC4167B for ; Wed, 6 Dec 2023 17:29:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=5bxt3uDa6Gl2BoKPdOhmgW4CeUpIwBoESF17T4dEVCw=; b=B6h/bYkK8xXx7w FXUcPY9jEsfwQYEfnpA8zVKFHH9dZGpNP4yX5DzynXgnJVkv3u1Ra8rN9E+Z9otBLASSh6JlDL/zg faunXT/o7ISnNXQKndlN3THOAWhZuwyuidg79zGlJ2TboZcJm8Zz8UDgPZw983pwb5AQhA165bEw2 JI1UAhQZ1r86RRWyMpOPuwoe+3VE/2mXhh1QlSkR8jeVg++KPsnwZ8Be+f59oc0OIkFgID0FEH6f8 AcJ8OQALHAqgQltxNCDelaQwGO3+U8L+ZpGd1cFSX2Ec01B2a2zn2nB4j/gVa8gDlt4Y4DwmkVF6O lAlthRJo/+rFOQfdp0Eg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhZ-00AtM1-1S; Wed, 06 Dec 2023 17:28:49 +0000 Received: from mail-dm6nam11on2062e.outbound.protection.outlook.com ([2a01:111:f400:7eaa::62e] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhU-00AtFn-0J for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:45 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mdq8YF4Weqv9sQNfzXEfieZY8qSYWbvaeRbmiQgy8XGJYV+gq1SmRRd1Dq04xSflUgLdsBPMeZoQlyEGd2gaLCobrn9ydiWGMowNRRXY9sv62ttscZHQneApeDv6hAWPURZNJDIBEyTadaOWA0s94lARTHob/1PCdUUkPb0pd9rdhsr943XuSnsWb4lpOX0o28xBO7ADfFjeGSaHf549VWvt+Q4w0vWVLf0T0KKmvV8Kgttd/6R27+VrZcY7U0vkFX7VxXPtBcROzqL4MdOYXYNjaNSFeg1HDHOz1jBQ1i0RWJ3QlC4eeaicqlspjwxbmBfJKWRYDA+wf/pRfLSEyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=f4ZYaUJjMmsj5xs9BZufJhvIfG0beDtsdPFOiCXm/iE=; b=ifqIRkfmDMsumNJGov704N76xONExTeae3O19G9fM9dMKt36mrr4Cd3bK5mkK+/13G8oD8SFmDjqbueYh8oQXP7Xt4Bd9hPr0oxzu0pOyHJL2h186k1Hw7YL57vNm8kgU4+VmiZIfGNDueSCXr0I1+xHOUkGBWaKnk418MIyAT9DHERDF5ASTNZPBK4SrIAXZw4mGEBpKURz6zr4u22fjKkA1yJyX1bMzmegZmnumQ5Hg1FnJae/g7EAzHJEEQUpTrBzKj3vmIWjCJLx8BwWdzTIk/xFaYY/BoIr3B1Nix1MNK1HIdcUY9K8K1AkKb2qZs2Zx5LKucqMAzqih+vqww== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=f4ZYaUJjMmsj5xs9BZufJhvIfG0beDtsdPFOiCXm/iE=; b=fDcZjGFZZNH+blFgX+qCA761HA9V+tL2WTswax5669JGSY3V8kKpd7quj7vF6wf6iOKMI5nxzardF/IorNY9Plx1ObanOUP/R6lLIn1wvTksl/BEkRG1VwkJujYo46KnFdIEYebA4SZaCNhxF1vGKV9SVxawCD/GDoe+XcpBvOBoy0z2SF/9XKfhir0LuOzh+x4RJ4XKgqVqmGWVkmbJmwSHUd5B6Hh3ojSSgXi55FaY5ofITD8XDgBHvCKyfem/YTdc4hjQBB5aTdzmO7L3bQt/DeyAp/17+xKQHOnBzuUVVS6LjpeyzvHmes+4+o8ksWDQl2SxTmvo1W7d1AcIjw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by CY8PR12MB7364.namprd12.prod.outlook.com (2603:10b6:930:50::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.25; Wed, 6 Dec 2023 17:28:36 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:36 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 05/27] iommu/arm-smmu-v3: Make CD programming use arm_smmu_write_entry_step() Date: Wed, 6 Dec 2023 13:28:11 -0400 Message-ID: <5-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1P223CA0011.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:2c4::16) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|CY8PR12MB7364:EE_ X-MS-Office365-Filtering-Correlation-Id: 91224f27-13f7-4972-fc6e-08dbf680c5df X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2HIvLDCxSYBaAZdquZmY61u46waDi0DAPOV52dO5fLmRyhQXKMDPf3P/39nnJm0WiN+n9umoxX+lkoq991e1G5W6h+dgTGglwGMrE4IlpPepVHryHTFpFPJO1to1HyRGXcXzEDCkx5PuYREeqgELv2heZdkLIALRZaHcBarTsZhbW9h2/5htcHGMiE09dlT9qk/0LtIH9FviKXjbIInPD52Sh1QOXtd84DNtvqjiEw1TXjS6W4c4yO6hkbi45meBuIOpQ3yG+CUREsxzxVhjSkjiAAFpx/e5VZ+Ar5hAMH3J15odfcTZuvcQYqGDtEAo4pY9E+jo18tBhMGKY7qIjc6YlOoh7xBj0E+vkZu8CNgVgVmxHGW/9bVqLUkT07k58RXE6Q7mkxvwXRN740lG7bVGtdkk4qqqLGmJBGGxJMjwvuc4xgmac1zFamP2jw8ct4meBrkCUEOoW8dAOUl/W3MTp7BpG8V2IdQsTvMgYJpW61mSuu9EmWTLW9/voP2PcMhk5rfrDtKLOMJ2wg71UDP3hRUzoh/0BTGpI8OWalWZSHnK2kcyeZxTgSCVKKRDaaa62NXw+byWANvd0Ub2kaZsvY3KOiAsH4Jm/zoh2qk= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(346002)(396003)(39860400002)(376002)(136003)(230922051799003)(64100799003)(1800799012)(451199024)(186009)(8936002)(8676002)(4326008)(2906002)(38100700002)(83380400001)(7416002)(41300700001)(36756003)(86362001)(5660300002)(6666004)(26005)(6512007)(66556008)(6486002)(478600001)(2616005)(66946007)(110136005)(54906003)(66476007)(6506007)(316002)(4216001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: BWKyKEa47pn5j6jDw06KWzdZ10qj/LEN6agzvPdPphQfDJ1RzwVblUf6Ud+aZJI3CTuPQB/OEJVt7HzjW2TYKeEp0AcLkV7ktPdnfX7oMoLyTECPokk7WUDL7tvrfJUplF900QY3CL+7zAotu+gFXBTBsCUx3hUHUf9gRNBgWPgcTFbUFrqUm80pe/8ejW8j7gocuqqjsrQ/k6riFSld/Xdq/l5ET59Lw37ifzQsBg6+WpJOOQuX55jJNIXIrdwY7s+sNQmp/i14C7vA1sXcIy2Y5TiWLht5LjarIAz140jRrtgca9xI6ppE5SxxHM8qgVgde4uBnhz71MyXUOggI0ne1ExqpVFnZfsKEYYeIw+jtIljx7oXOF7NaNxxzkxRJsXvTkh9unEoqlqLu2GgaCkCoLt8ixnQ7nKcB2Z3SSgIFr5P2BCeSnKYsjqKtHrv4OP5mEB/cJgctry5ZnXIPqGU4FRHvdTgGnxzug6HNd8jumAuAvVtQgliZJRroHh3xzFNfQ/svWum/pJKojDhWKlhx05fCdfIbrdfzC/bQ2ZB+jZiTzio32FEueMoGkcvMvw8b0v7KuMCN1QGJopJGyDorBuVYMYADlpiAT0fbcKArD3TEuCu9VyighhyRgYJAep38za7Hxusdfi0GPo1Pe9igOLYmdhc+p7p3GdFLSMy3EidPClO8iCxWPLheJM0UaHAv4xQZTq/A3mIPmPRHpxXp2u+0AcJJL8ftLD3/0E0MoMJ3qxCwuRtBGg/f+iop7hboG9DWINKU9rl8UQyhvz5oV8q4Vt2HBSFtqEhhyEfkoXhJ3/Fuo33fw27XqC+eDd5wh8fPQFf+x5GW/5maohsdLrsbe3XYzn0HHlTL8wkPueHa3Rmlr0jSX8Rsvzf0k0yKspPHY/ylU8muDwd2FpOhSkg/P1Zs100xyWKFH0S4DpD4hx8B3SsMVs2azj+ionIBhk/nLwb5sEODvbe52X+S9KRv2sb0djzCg9rOVSLVgvd3he2tHqQ9/W0safOPRseDpgc3mVBVFVwab1tSB3JkZJ+ukIsYusGQ15Qvch+ZZYwcXTbnXbaLL/GW/A/frG0PkwJARUeP3gytGAMS9kGOgPWovDGH2LtaPkLGIze4UO/l3ekQC0wq/egb8LhRoznQzjEBxBaHASDQFIhZEUpRoiQVA2QIAw/MZE5wRfK+fNtZcoWPlSlxx2v0QoCu9tiOj1OUh79Lig50cWD+cx+cs56GOuiR8N4zmBKT4Bnohzaa4mkodyajwdQio+tv0B30yKiTip08cPWz31iG2bOLk26guos0v78RHXQS5/UhOfr+H1vHGy41QWlLeqyk+0weyFY6mS3h45n2SuQ9h6u+cFFcftJi8fx347T3rCz4P1vl/Qo9RstKEZBb+Xc29UVoY0/cN8OnLOQiEn0VAzI89NnGwBnzeha3dTEdmokP4nocp3kr9IWhNogixd+8s5/6MxHCttviDfXJJXnWiCAv2LCyA5oNENQZC+DxEqB/zbk69xKqJA2ZGQKVAMbmGjHPr5rNKs1PueQjXYbs2H0pbwzjPF0bFq34A0TTrJPfVqxty1RIIogfv8WvKCX X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 91224f27-13f7-4972-fc6e-08dbf680c5df X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:34.0873 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: u/iuMLY0biebi03gqT7i5PmqqK38F15GZGLvQMB2WK+6F4TdUyVmfaDJmMTzsuEs X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7364 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092844_138003_4DCC5E46 X-CRM114-Status: GOOD ( 15.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org CD table entries and STE's have the same essential programming sequence, just with different types and sizes. Have arm_smmu_write_ctx_desc() generate a target CD and call arm_smmu_write_entry_step() to do the programming. Due to the way the target CD is generated by modifying the existing CD this alone is not enough for the CD callers to be freed of the ordering requirements. The following patches will make the rest of the CD flow mirror the STE flow with precise CD contents generated in all cases. Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 79 +++++++++++++++------ 1 file changed, 57 insertions(+), 22 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 3f77583144e649..0847ce91c5cc1a 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1146,6 +1146,55 @@ static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, return &l1_desc->l2ptr[idx]; } +static void arm_smmu_get_cd_used(const struct arm_smmu_cd *ent, + struct arm_smmu_cd *used_bits) +{ + memset(used_bits, 0, sizeof(*used_bits)); + + used_bits->data[0] = cpu_to_le64(CTXDESC_CD_0_V); + if (!(ent->data[0] & cpu_to_le64(CTXDESC_CD_0_V))) + return; + memset(used_bits, 0xFF, sizeof(*used_bits)); + + /* EPD0 means T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED */ + if (ent->data[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) { + used_bits->data[0] &= ~cpu_to_le64( + CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 | + CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 | + CTXDESC_CD_0_TCR_SH0); + used_bits->data[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK); + } +} + +static bool arm_smmu_write_cd_step(struct arm_smmu_cd *cur, + const struct arm_smmu_cd *target, + const struct arm_smmu_cd *target_used) +{ + struct arm_smmu_cd cur_used; + struct arm_smmu_cd step; + + arm_smmu_get_cd_used(cur, &cur_used); + return arm_smmu_write_entry_step(cur->data, cur_used.data, target->data, + target_used->data, step.data, + cpu_to_le64(CTXDESC_CD_0_V), + ARRAY_SIZE(cur->data)); + +} + +static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, + struct arm_smmu_cd *cdptr, + const struct arm_smmu_cd *target) +{ + struct arm_smmu_cd target_used; + + arm_smmu_get_cd_used(target, &target_used); + while (true) { + if (arm_smmu_write_cd_step(cdptr, target, &target_used)) + break; + arm_smmu_sync_cd(master, ssid, true); + } +} + int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, struct arm_smmu_ctx_desc *cd) { @@ -1162,16 +1211,19 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, */ u64 val; bool cd_live; - struct arm_smmu_cd *cdptr; + struct arm_smmu_cd target; + struct arm_smmu_cd *cdptr = ⌖ + struct arm_smmu_cd *cd_table_entry; struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; if (WARN_ON(ssid >= (1 << cd_table->s1cdmax))) return -E2BIG; - cdptr = arm_smmu_get_cd_ptr(master, ssid); - if (!cdptr) + cd_table_entry = arm_smmu_get_cd_ptr(master, ssid); + if (!cd_table_entry) return -ENOMEM; + target = *cd_table_entry; val = le64_to_cpu(cdptr->data[0]); cd_live = !!(val & CTXDESC_CD_0_V); @@ -1191,13 +1243,6 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, cdptr->data[2] = 0; cdptr->data[3] = cpu_to_le64(cd->mair); - /* - * STE may be live, and the SMMU might read dwords of this CD in any - * order. Ensure that it observes valid values before reading - * V=1. - */ - arm_smmu_sync_cd(master, ssid, true); - val = cd->tcr | #ifdef __BIG_ENDIAN CTXDESC_CD_0_ENDI | @@ -1211,18 +1256,8 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, if (cd_table->stall_enabled) val |= CTXDESC_CD_0_S; } - - /* - * The SMMU accesses 64-bit values atomically. See IHI0070Ca 3.21.3 - * "Configuration structures and configuration invalidation completion" - * - * The size of single-copy atomic reads made by the SMMU is - * IMPLEMENTATION DEFINED but must be at least 64 bits. Any single - * field within an aligned 64-bit span of a structure can be altered - * without first making the structure invalid. - */ - WRITE_ONCE(cdptr->data[0], cpu_to_le64(val)); - arm_smmu_sync_cd(master, ssid, true); + cdptr->data[0] = cpu_to_le64(val); + arm_smmu_write_cd_entry(master, ssid, cd_table_entry, &target); return 0; } From patchwork Wed Dec 6 17:28:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482100 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5CC11C4167B for ; Wed, 6 Dec 2023 17:29:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Ia16zqs9m4xiTOSzkCe6Uuhs64GPmBPiuoUHfrCtmUc=; b=ISyTAkzr7Tr0Ae OqKCEc7chQqhVe37Ac+f2L3rK0SWlJxbEM6yq8Oq9GG8h8fZwfkX5lCOOCWjf0Jj3RVeHx3wGeVyh mH0bCJ89zaOHW3Kc14w9hkDM1z9T0Hv45VCA3wMd42Oaln1VTcLQsI6BDIlX8LfB/XdT6aGIfgql9 frIx9i5cuVSH1j0zg9tUoymuHVvSFnfVniX1UrujdqWF/B8w3PSV15Q6Wqr1b7UNTtbs5BX6grQxu UxqyVQQ040u/TbXCcyNU7hf8YdL37eb2RLBaoNQSykFnb/1kdz2C2DI8tAg30QQf9mKzrAKvI1Y7H nPgy/thsbbzzzcIDLZag==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhk-00AtYt-0m; Wed, 06 Dec 2023 17:29:00 +0000 Received: from mail-mw2nam12on20609.outbound.protection.outlook.com ([2a01:111:f400:fe5a::609] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhY-00AtIF-0R for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:49 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=B71BOfSc8X6v2R0QKsDTAuaf3ZtyOrZDh2WEsk3YuC8OGHUyVYBk+XyYHpLGkDy6RcPx7VFfg7Z6cKF+Rx4Uv0M6747b+1yNQIWwlzrbg12FX83LD9IKzxPmW6JbxNq73KRXt7/PgpeMUcPeim9/AOAy6FBKmfCrDNBBNDsNMSakA/rYh6c7AHJm3qP8Mtk82rEfKPbZ7Bei+Y4EfvRys2nrNJUV0170VGNRHTKAm4pkR26gbiEEqi2oXUJcw8ya9XdY/Y23Ua9lb/UtREMa/mVlvnbTodeJuYU2bfRB2CS8pTsuN7Ou/lJrRdUhUH8Is0JDiw2wouISv+8Wlo0gkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DJt7TVjS1AM9FM/jiszbXAAXo61ZvURxUOsjWtsE+H0=; b=TsHNCQFlkJ5x/NSIqvYk5iV3GK9PhXP+GDe9x2JgRmLAZD66hSvsVYdcdfGRZMfmI6c3FzEoUBukOVTTvRksUuIDR33myPKKzr4dY3/YMLfMPWPHv1SFTZ2UpMgBgW99AtgaO3CD+dGD4+3S1pN9fKt7SjfI52Ql39IhcDkfuml/YTWAKZ1ftUYX6FTATMmPaf3dk2f4T+KCB1tEgQxSvi01ry7y6aOqcUpADFWh0ez15Y+KwgExQG66NDw45ULrt4e722oYE8ALep7YyS6SDvymtIjAPmKPaKOxeJB3MXekfu/hrbhfHog5zYEq9XOkQdAx38FGbWTNCWwzdf+r0g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DJt7TVjS1AM9FM/jiszbXAAXo61ZvURxUOsjWtsE+H0=; b=qyFx8aPyO87BzK34wQAcgqGtR7yfXfH3G2Tu2f9q7dtyBfRt4lG/9RIpYJKyUMBgxvV8IJaGvfsgJDzdR0GIkaHgLtRLCalRyyFc8YEqq86jl7vM0PQtrF53v8y2NMTzGpsRAcaVlf3aVRUJ3qzieKgmSM5bwR/8IgdqrWauo+kLT01AmWC9BlXdQf1ve0fq4h64wkgHoOnTRyCn7OP/vwMev7GShREEh2VXpwMUEL9s7GX6CYGj8/qrfJcRbpfcQB8qsFIuozDO/lxD08AN3O8c5d0rVvxj4FLyzaqoHWU0uhqlshevtQcGvRZXicCV5ycW8i7va1Io2S50ff7vZw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:40 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:40 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 06/27] iommu/arm-smmu-v3: Consolidate clearing a CD table entry Date: Wed, 6 Dec 2023 13:28:12 -0400 Message-ID: <6-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR22CA0029.namprd22.prod.outlook.com (2603:10b6:208:238::34) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: 34538971-27e2-4d1e-321b-08dbf680c657 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zsagU0suiEmnKD0Rwj6NveQf93kBxQV2n+hA/Q8RpDdZKg9FDiDyFLHvuNtY58s4P0FCZV4yTCF5gXMXTUDyHKXrL+XHF8b1xzN+X89nUOF93CWsak12cCQkNXEBfNfq+XZL4WReMarUlM4uUIwvIsPDsf+tlFxS8hrr/iv3DLxHBrowYva7jyD0HQXstYPlB6VFpY0S4HYEZsn1I9xM9vbhSs3m5jiAaMsmalt6eYBORxMkR4uCserdD6kgpvIY9bKCTEGn1NGWIN+2MWYjPpaBqOkF5KUPJuHZ4CXNgmA2ACf4xJ//KofIDhWfyo2714M88DgYfZuPu9kCubR79bnnvlSmmjNNDr6kOg1P1wAVucifDExpbcHF4TJbBBvvRRrLZ1efPNOCu4y1QyvtOzPlnQJXHFw4gZGCj6JoI+fabHSlDYwVsd2PVJYcy9skGMAx4CIjfXt31w64xhhAi5eexKWR/FHne56oQcbOSQ9QY8Ttv/nVLbCqCcJeESGdTsUuDI8/396qhsr3PSLmihFA/WdxzVrghzP0gmeAc8+9xGsIXmrgzTUNjzgomQ1qbkoVVkuaZeb5m7HhdKcbpXc7zbCrTA79Rh/t4cK/kFQ= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(4326008)(54906003)(7416002)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005)(4216001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 0wbknoBHG/uAf/dPIi3vc0o4mYqScU5noT2ACodwzgv8dzcgytKc7CQZsS5qxFBL5/Sf25jxY1gi7vPMuTAkzzZyvCXefZStALWKl6Wu0H/64xjvoqBEhj/Xul3oCEeAthX6wirhaNQVgsaMamI/bnuZtbiexO5FdwTpcNGeGQeo0fqtB3cqsFJhe0Oby5DdtS6Khq0ZPsSwcLjy4rxvokTMgRvMEeD7CFT2hWe8XAQ87CRnlU/+4l7UzuNxH+RgeRQU+cAfROzI1STINRHtspc0Us5BclJ8ONWKqoBPNz7jWYrvnvqkZ0sFVCVmykgD0go2Vq/fWaxx9FVxKyYQNTLKWkwfCLng3qTXwIVkMxER3RxDqnFiAqNhydaRqNGz7e6VXd8XNH+lHhCAGxdf4bkDVWPKcGDe9osF+5FvpO3ubSQDG7vP84BlppyOATcDwdzUP9zuzXCjOKeduCxnLxjYgHNdR/cea6F+GkGtms6717pTCXFIJrZ38v4MoetBOHQiwBRYRh9aus2ztdr8lNcpC+YbaF8Hkh9hnjjafOKO0CbaEU7zMy9f11Ud1U4zVfriao4ID2KZxfkHx7fT0574EwzBuJVdo43R3EK+LALwGd7IErCdPwn0W8wjk5Z5zia0V5YlNfDIz4jrcHj+PTCnadOsCoHkSAPoNo7QI9hz2OMG+Kcn53+G9PP4WPoIfkzQqLU4R+Hsk62LozbrIGvwZry+hKALrYbTW9j/h+vwYCJSn0mq9PZtrGRgzGRKtGYqXXELdme/9Ww1i0UXZIGeqQ8Y+nw8QFCRRE4WO0Z2uPtCi30SAJv0+IcBfLqQ69qlu8N0E/TJfQJcuipInr7Qfgqm17+msSB3qtkWymiqAAz+meeXU6AJElXJcgA7/ftYwBWFRH4c5YnZQUBmyoxKu5ci/xdMuNbdO5kHqhq9MfwCgnssmuiIPwfUAiGUtxIXNw8N4wJO/n3HsfkTevtVt9JwQn5Y48JNcIctqzalDZs2hsw7qs0zMzUvSBABEX2+OvMmAkaNdT51o28Ot24qzyorjjk1TLMQeX62AjzZJny73Cfi/uqNKRGOkw3FXl8GiMAP5Wqkf58Qay9+WGsM98Fx6q4MFyMm1LLcqdhn6Kk0mXImJE6x1OGNCIAxA2cTMbJepr8VQ724YXd5g+boT6ECv5rpbLI/tQp6L1N/N2Y9ngpNyGgRSJTOHmxpZIh2Rv2cAuMbN4vehaU5khfKF+aY56Frpj/iqxpZpQdff0lhS5GguZvqnw5PnorQg8q7niFtmTV8lTFN/MkgffeCeXjtNKPc+XDuMjM+xsSqvmFBUot+0t00lKCrYdKn4V+nn3TGTcHodsw5Ya52Q50grp0PXO8xFOYmGxlwYEnslDQ2UWXTpaTD3TZ0L4rshnTHb4Rnw9znecwTso+NDAPOsD51Ucrmxso1FwoKhm5yIsGiC3Vap78CxdDc1lgkoUXBRXtn7imviFt0suhHvYI9xqpVAaU4wL/bCaoYtux/Yl1l/D9e2v3niYO1m9rBfMFq1cPW4rRW9xs74erUJmKxyYR9Ald3auxQwBHSjBKRwiqhDOXIg7T/fk9iEi5s X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 34538971-27e2-4d1e-321b-08dbf680c657 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:34.8212 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: emCw/Dz38Zx70dNMnz/fNKdjq3Qzagx0f+cLvfb0L2QypoMHckyJgtuq8WTsWUpU X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092848_178190_F4BA6385 X-CRM114-Status: GOOD ( 12.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org A cleared entry is all 0's. Make arm_smmu_clear_cd() do this sequence. If we are clearing an entry and for some reason it is not already allocated in the CD table then something has gone wrong. Move the two SVA flows that clear the CD to this interface. Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 9 +++++++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 ++++++++++++++----- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 ++ 3 files changed, 24 insertions(+), 7 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index ee3d148aafa26b..521bfa18879f90 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -328,7 +328,7 @@ arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain, ret = arm_smmu_write_ctx_desc(master, mm->pasid, cd); if (ret) { list_for_each_entry_from_reverse(master, &smmu_domain->devices, domain_head) - arm_smmu_write_ctx_desc(master, mm->pasid, NULL); + arm_smmu_clear_cd(master, mm->pasid); break; } } @@ -352,13 +352,18 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) struct mm_struct *mm = smmu_mn->mn.mm; struct arm_smmu_ctx_desc *cd = smmu_mn->cd; struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + struct arm_smmu_master *master; + unsigned long flags; if (!refcount_dec_and_test(&smmu_mn->refs)) return; list_del(&smmu_mn->list); - arm_smmu_update_ctx_desc_devices(smmu_domain, mm->pasid, NULL); + spin_lock_irqsave(&smmu_domain->devices_lock, flags); + list_for_each_entry(master, &smmu_domain->devices, domain_head) + arm_smmu_clear_cd(master, mm->pasid); + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); /* * If we went through clear(), we've already invalidated, and no diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 0847ce91c5cc1a..7fcd578b762afe 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1195,6 +1195,19 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, } } +void arm_smmu_clear_cd(struct arm_smmu_master *master, int ssid) +{ + struct arm_smmu_cd target = {}; + struct arm_smmu_cd *cdptr; + + if (!master->cd_table.cdtab) + return; + cdptr = arm_smmu_get_cd_ptr(master, ssid); + if (WARN_ON(!cdptr)) + return; + arm_smmu_write_cd_entry(master, ssid, cdptr, &target); +} + int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, struct arm_smmu_ctx_desc *cd) { @@ -2622,9 +2635,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) case ARM_SMMU_DOMAIN_S2: arm_smmu_make_s2_domain_ste(&target, master, smmu_domain); arm_smmu_install_ste_for_dev(master, &target); - if (master->cd_table.cdtab) - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, - NULL); + arm_smmu_clear_cd(master, IOMMU_NO_PASID); break; } @@ -2672,8 +2683,7 @@ static int arm_smmu_attach_dev_ste(struct device *dev, * arm_smmu_domain->devices to avoid races updating the same context * descriptor from arm_smmu_share_asid(). */ - if (master->cd_table.cdtab) - arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, NULL); + arm_smmu_clear_cd(master, IOMMU_NO_PASID); return 0; } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 24a77e0a97898b..a8e7574ab8e154 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -763,6 +763,8 @@ extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; extern struct arm_smmu_ctx_desc quiet_cd; +void arm_smmu_clear_cd(struct arm_smmu_master *master, int ssid); + int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid, struct arm_smmu_ctx_desc *cd); void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); From patchwork Wed Dec 6 17:28:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482145 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E95B1C46CA3 for ; Wed, 6 Dec 2023 18:31:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=HuejLJdUpIG3tDcqQ3pQ1IzTgsSYzoSslGfA+Nf1gZo=; b=Fhci5D6lEdvbjr D1En9ngJodNCjcwdZYJjwghKn8trzMZALdcivm/2LcUGTztBWJE8z5ILKDKiV7oHt096OwN4G055K /u17GaLo4C4mJXRPIln/n3REYP2TYTEOD2L6Qgi+qR2z+t/qWqoBw9CvOvdMSJz3SFqpJgPCGpfnp lN4unDwcnkM1Z5q9hWbxk04ahqQxtDoDpwPDaDz32/MMMLdL2S7pS3HHXq0oAIu6RMXo36P3uwOD6 faoh4Qq7SFirGrFSeA0K1oXzD7CQOlN+9HS3fWh9iLx4goLTBZyYNhl1JoXrkPQJioWNmMRJqIbbk IkVjsDfpn4X/ZX/ICJSw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAwfR-00B2C5-0d; Wed, 06 Dec 2023 18:30:41 +0000 Received: from mail-mw2nam12on20609.outbound.protection.outlook.com ([2a01:111:f400:fe5a::609] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvht-00AtIF-1Q for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:29:22 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=asXIw4RT5rvlezIO3i2GnBW1L6Lr0fKxd1xE3CLel8vGvb7pXD3Yq6erMHJOgxtqc+X2xD7yDXvM/X2ocFAvdy/sKq2hBGLTpscAUAt839gcBm5X30jOAlvW3j781wNw/qaLuCMX01G0B1FgekuBt1ZOpDt4R2AuO8/Nx9Mzw/H9auLrb/VNVO+h2LjhjMVqU0LT0uyH6UJsJ11ophQif31Xz2rbRjIokBfGWNJPI/+DyZlOWqrSimPc2jCuYfslik5KRAqKxqRYGdaV/h54bHjPIgMkTA6g1NvMtETS6gec9glKJNXQIHQHT3Jt3aQz+vc9NQB44hhrtX0NYmuqpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4gwelGD56wJhfomawNHVmawSMgQOAg08hMp+ta9/3K0=; b=Sjo7liyRaNiE3a0q500LJkzvcRsCn7Ev7inEmuDLDpweVDyqxnAb3dW+Yq+o6tnJM8mQt4uQKxln4O63zdVQATDqN34ZUme4sSMCylRBs5cJ55Pmve27N1Ecu7+1a/G1HdhKqCAhY5fZPuC6Q/x3hxDlWBLipPCKDl5V6oriqBCOjfq+hWPA6pDBFC5+b2IODiI05WppS2IUPFTXXxsI2jyK5dWDYNamAwlTVDGBz5aDpfVLI0CwkqTDBIyXt4dDeUPswN7ZC6gaTBevjQW2OA+rF9CpNY2ywfmzoSR/2t51Zv22eTwnwAMrs3tSqRVQj8pHNf0DjWridGMDd5hVCA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4gwelGD56wJhfomawNHVmawSMgQOAg08hMp+ta9/3K0=; b=ncK6+3mYiITzzvcOaPWaqnoOozUDZFbmh5724IvdQTJxcvuhGWGkBwTDQfOeCoZ7CSPvEEVlOYho1/Xb6p4z9WoWhRgXGcUOnhpC6eXgrMPz+aWfl/ILhEL2lwpAU/by0ITXnD9Yb8HlvNKoNMPqGRNphQpTIUChLjsfX4wRh3wkS/SIoEmRwE3K0ErLHFalVOtJB8GgS2R3qv4cKjglpJFVNONELcUXMswNrLw5bf2ideFyhGKKHWzSnvtGiztpblAe+r26ksgC2/JIfFXgHSmSzUOszn+azKqH2Wce6e+phMms7RANKOghxAkIZotj4Qa0XRVmy9i6wmk5QQ1zDA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:45 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:45 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 07/27] iommu/arm-smmu-v3: Move the CD generation for S1 domains into a function Date: Wed, 6 Dec 2023 13:28:13 -0400 Message-ID: <7-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR1501CA0023.namprd15.prod.outlook.com (2603:10b6:207:17::36) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: e8a96e6b-088f-4ef8-3a15-08dbf680c762 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: tHlHwwJk1TARexUHbaKuoL5xFVygfInj0P35hRBjrOpEOLcMXfPK+TxbquZh+FPQaUK8lPx1DNU8wSaAek2xnzFvPKCBF30RMyXKynnU6EpA6mhMmfBC9ZeIyJvvMgc84qHTRkA5QcifCWlW8T/axau9WH3ZoafJJAvK9TfrVWbS4n3VVNLInhY47PSS9pGM6eMb4AUJg0LlUk1P6B+m3XTY+HZQXu8qMnGTEhcgVrNntmkeg/B0NwIf3yR2KBqvJzQM9KVu7YZIJmZ28t3NNnpB8TmqtFPyQGrmz0UpjRvuD3x++QNHT8c88u6oLldLuLlshAmabck6z734eNabC3r0js6DybwTnxpaagyTw9vv4WU7vbgoFle+KdLL+FnxSQuWNsgszfV0MrWNtZ8t+vZKnuhTPT1Bvl1hx24gXj/7BtzSoJz09837f1JOQZv7qu6viRDVkIJ+nvR83MErGfAnPVaFr8rbJu0lbtWO2dYpsCONpXSG0MtmNcvBjyTzWoQVCXAjWdK6jKKCWwwPo3b1JCqazthDgKiTniFhrgWb9RPsi9auM5759n7+wF4OKMzoxH6LTYMmAa7PLVRz2iKFqVg60J7dn9ZXqhnapMLmZMw1f8zUcE+5B5mltUhjTY9XZ49yzAoWORf/XBn4ag== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(4326008)(54906003)(7416002)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005)(4216001)(473944003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: JSFrOeLNXuSaaq0MsODFtEUZEGEzuuhHyG/qSxv76kLg1Pl5vTVgM96U45ikSB+JJqlxmQrxtwUefi+f34JjVTYht3RJ3a433LIS/Ok9Jlv7ccpQHgSx4JGbpLc5xzGAudm2qP7fwZIL2XDZAKTvZQ4tg67wFXQyTFtkPxF2ZgDj+Yj184MzqldckfjE4xJBf3quo2dC+6UO+b+miwrUxmxNA6P16BfuT2U7a4Y8rJ7qNDeLMYwRW6zEfyVAnT4XzIpxNfCz2XhQnSCGLzcaDLrvh8HOfdBjJY5GggNgTsDOftVyxkM1Qu+2VQQF8cjlWO/oo/RhHU/UOVGL063EOksRZWLoAH+jkeaHYTaDEx/Ymm4KXSLJ1eCsQE4VAEJigS5MpLgusnLnUX2cyxCKTpipVDcYyG6y6OSQTPYAiYj2CnXOMMVsFixkB/Ui0JP9+c4RQ6xUpTrdO0zN4ct3zuLkLKaFOPyV2UFgsHOhpP0r60xlf4O9lx89TpJ/wxFoY664AnNe0U4fZTa9Y+8MoEWGJZabXsqYspEVPW54PbTJBhIrHPZSrCMfMnvAJ1wM1441g4lI6Zhezj10Exa3E71xlriZpZJryjcPyZHD/FGSrQvGVmmIvDvzruW6iGZjWMZbMer9nMzgmX33LDdIwG9t4VrL8AQNgtahk2yT2m3A0LGCYGBJQNA6M8vTrEq0uaAhrgA9ebIZ5fValOm8KQFfvrDwYp7azoBeLiACx5oMPf/STzkb80tFyY70wBdhyKUyv+HPGyZqb41mSvdpYXOX4NiKMWBp+quQQsEM1u2ay9HOPGGKWg8OXkZDzR2gIHjEC87OKBQt1hteJbOY/LFXamsR4rXQM6PwhL51sbcFJq5tjpaPSOsg3UXibxqpKRNIXUeuXzFw3yLGOVmeA2XVrJGtQWKTKdpKsVbagKvN5Mh9M/nTfzdI9Gpv15InwY7OeTtYqNEIV1dotuWxstZmNFaCTQFjudhAZRxk5HZyXe9FtoCbq51OmShEDnIZkxrRptTSA7b6FO37ZV//SykhDH9mlhzzM2VKQv4NweEbkiAHu6TERQeH0yJBKCopyKReuaeFBEUqd3CZ8yIVBRePfw5RYPI7VhBjW3mVeN42sbPFDjY1XAoYkUJyaRgrDqXce/MxHK/00JR43Mplet/IOdQNmra8TcJXVTZavJv0UOjzJ1JVIyOHe5h5DdwtUorCznZHftJbGfFe0saFcDgTAs+71la+Q9+4DhKcU28CKxQr0oDUzD4cMaeDKDq33ByV5ozffzkJ4ewSwNNEYa7JzcjHbJPSy2yWx/9bZaRT5Kd7ZNHmkA9crwCZp2kWKL8OzhHiccjV+l0PFybEKWsbbHOsvC7wtPCbDoYR5r9Kqfw+Hu6HUZNkH1tWJfPqMR9W72VbTlJRaVYuoMnaCHYxoWKUaEbSFxoTaVlD+FheFYA5mD6QCLsyrlipE6Ji09KJhm8yQP+scwIM3izm37pieiLhgEWDEPIf9EQkPggWLGsFVldCemzHp/A7Q1gSXCli9rohrdDWwN55L5fl/C/TvjrVH3n4UX+oxGnfKOuxTZDSzV6BCRxTUFQdgdwL X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: e8a96e6b-088f-4ef8-3a15-08dbf680c762 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:36.5805 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: x1HieTRDTTS6NKfkqSO+Evsl6yh+0kWfMBcuLTyYmVNuSQGpgZUb64aFui/4N2om X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092909_535042_20FFFC22 X-CRM114-Status: GOOD ( 19.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce arm_smmu_make_s1_cd() to build the CD from the paging S1 domain, and reorganize all the places programming S1 domain CD table entries to call it. Split arm_smmu_update_s1_domain_cd_entry() from arm_smmu_update_ctx_desc_devices() so that the S1 path has its own call chain separate from the unrelated SVA path. arm_smmu_update_s1_domain_cd_entry() only works on S1 domains attached to RIDs and refreshes all their CDs. Remove the forced clear of the CD during S1 domain attach, arm_smmu_write_cd_entry() will do this automatically if necessary. Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 25 +++++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 60 +++++++++++++------ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 +++ 3 files changed, 75 insertions(+), 18 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 521bfa18879f90..04a807774402b2 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -54,6 +54,29 @@ static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); } +static void +arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) +{ + struct arm_smmu_master *master; + struct arm_smmu_cd target_cd; + unsigned long flags; + + spin_lock_irqsave(&smmu_domain->devices_lock, flags); + list_for_each_entry(master, &smmu_domain->devices, domain_head) { + struct arm_smmu_cd *cdptr; + + /* S1 domains only support RID attachment right now */ + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID); + if (WARN_ON(!cdptr)) + continue; + + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr, + &target_cd); + } + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); +} + /* * Check if the CPU ASID is available on the SMMU side. If a private context * descriptor is using it, try to replace it. @@ -97,7 +120,7 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid) * be some overlap between use of both ASIDs, until we invalidate the * TLB. */ - arm_smmu_update_ctx_desc_devices(smmu_domain, IOMMU_NO_PASID, cd); + arm_smmu_update_s1_domain_cd_entry(smmu_domain); /* Invalidate TLB entries previously associated with that context */ arm_smmu_tlb_inv_asid(smmu, asid); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 7fcd578b762afe..696f837015771f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1118,8 +1118,8 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst, WRITE_ONCE(*dst, cpu_to_le64(val)); } -static struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, - u32 ssid) +struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, + u32 ssid) { __le64 *l1ptr; unsigned int idx; @@ -1181,9 +1181,9 @@ static bool arm_smmu_write_cd_step(struct arm_smmu_cd *cur, } -static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, - struct arm_smmu_cd *cdptr, - const struct arm_smmu_cd *target) +void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, + struct arm_smmu_cd *cdptr, + const struct arm_smmu_cd *target) { struct arm_smmu_cd target_used; @@ -1195,6 +1195,32 @@ static void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, } } +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, + struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain) +{ + struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; + + memset(target, 0, sizeof(*target)); + + target->data[0] = cpu_to_le64( + cd->tcr | +#ifdef __BIG_ENDIAN + CTXDESC_CD_0_ENDI | +#endif + CTXDESC_CD_0_V | + CTXDESC_CD_0_AA64 | + (master->stall_enabled ? CTXDESC_CD_0_S : 0) | + CTXDESC_CD_0_R | + CTXDESC_CD_0_A | + CTXDESC_CD_0_ASET | + FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) + ); + + target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); + target->data[3] = cpu_to_le64(cd->mair); +} + void arm_smmu_clear_cd(struct arm_smmu_master *master, int ssid) { struct arm_smmu_cd target = {}; @@ -2609,29 +2635,29 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); switch (smmu_domain->stage) { - case ARM_SMMU_DOMAIN_S1: + case ARM_SMMU_DOMAIN_S1: { + struct arm_smmu_cd target_cd; + struct arm_smmu_cd *cdptr; + if (!master->cd_table.cdtab) { ret = arm_smmu_alloc_cd_tables(master); if (ret) goto out_list_del; - } else { - /* - * arm_smmu_write_ctx_desc() relies on the entry being - * invalid to work, clear any existing entry. - */ - ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, - NULL); - if (ret) - goto out_list_del; } - ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, &smmu_domain->cd); - if (ret) + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID); + if (!cdptr) { + ret = -ENOMEM; goto out_list_del; + } + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); + arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr, + &target_cd); arm_smmu_make_cdtable_ste(&target, master, &master->cd_table); arm_smmu_install_ste_for_dev(master, &target); break; + } case ARM_SMMU_DOMAIN_S2: arm_smmu_make_s2_domain_ste(&target, master, smmu_domain); arm_smmu_install_ste_for_dev(master, &target); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index a8e7574ab8e154..950f5a08acda6d 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -764,6 +764,14 @@ extern struct mutex arm_smmu_asid_lock; extern struct arm_smmu_ctx_desc quiet_cd; void arm_smmu_clear_cd(struct arm_smmu_master *master, int ssid); +struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, + u32 ssid); +void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, + struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain); +void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, + struct arm_smmu_cd *cdptr, + const struct arm_smmu_cd *target); int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid, struct arm_smmu_ctx_desc *cd); From patchwork Wed Dec 6 17:28:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482103 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F2787C4167B for ; Wed, 6 Dec 2023 17:29:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JKIRSrT/ZSIlE9LYeaweMRhMdMNSfwUuIMcL6CFVVic=; b=zj5OD3cqoj/lh1 VJQfgWsb07AByR6rrzVfwOhIAxihieXogFg0YzR7Wp1LPgQFYwEOIvipe5VUjQIulJJbHTfgf1M51 8Ui44utSTpoKzYZGTWlZuZeErUjLanvkeIEBOlRF9lDFMhfWZ389MlyMcX8iJrwuZDUNVXsVKbs5A 0HRIGAka1ss+jxGVU/XFKfQHkiG2tSrWs8S14qAm3/OhKRclTpDYXVj6r6FL7zfFGwd5yGEn/H4gL vN3jUQaJKkEX2wHPiWb+Vu8FPxZSfB6zs2RMxSuGDc4hmespMw8mJbanjY+KDW14bcyr8sC5+VGh0 R6RLysAYfX+wtX/j35zQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhl-00AtaP-2t; Wed, 06 Dec 2023 17:29:01 +0000 Received: from mail-dm6nam11on20615.outbound.protection.outlook.com ([2a01:111:f400:7eaa::615] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhY-00AtKY-1W for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:49 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hZOuaKdeJsWxyc7ZCRGuwOpUjEGvDRLw89KVAakO/xJW3pprApqjM+ijAmDnXLaw+OLn16cbcI+gRdyFBF84g3nGfUzSZEZjrJ15JX2HyQ58cdibm2IpKpoXQlHCQXjmsH3WlYjVJ/zoDRh8LFM7KXsC+lAoY+J2Fe2NjGe222Wa7/pfQGj1BntYysjcvguHPf+iRm3Hl6r6CS4h44r0RBDbc9SzaVpuLSLdVwvOCWYMIPX5yhaIpefZzNK+Vwm2MTZ3aBkut7UzWhBKfJvto/p3fsXXNQjAuVJeMVPkIV1HLsbDomEq3VXDEFFyHPWfUcZuWC0ExvpSC/4bMd8SVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vtrl5dtWYNbmQqQnStN45gQKdSw1hWTiwRqlc7s1JAk=; b=g/IQJ8hFMhUE0djCiFv2gPq0DAc0Mp7e31bQFiRwsYueWdu0G9GA/QAHkLdrmZcLZSuYZQtu6wDpInob41dTRDGpWGD1OxOIx1/++7ROcVIAoCeJVHOK3L6XytHFxfm6bdYMQb3pQH2s23fjklZ+N4CF87pCUBI+4eF0NjkZEHgpBraQCQ3sCnlXpmm3kBNHisTRFLP878A+PUJyfiS85a473on7anQnjBmPhiB5ybhifa9nEU8flEjjhsIeEzGkFP8uOWKuAarAIehvt+v97ytedWW2BI7A4MbK9t5X+rUFP7w0x9M98IIOiXtchk/5ZvbnWu1HEShT88GAqO3uOQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vtrl5dtWYNbmQqQnStN45gQKdSw1hWTiwRqlc7s1JAk=; b=fZ+jrTxlTMJxIJeGYIEovE+7TO9qjdHDJUXTLCdfalLyMlXOeW3BounAZsLGAMpY5dEqgsdj2MxNwA01wHXLzbDF4/Ijc6UKrUZIg+y2+vNUAA0GgCmvuNoYhDWw1Ik7wLm95FNtlVl4VNXRknSiSHt46tRUhlVR+4LsYV+5+3B4F5I5AISS8sV7K4hz/Ut+uHh0YRTqx1VnZIapPcm6bZk34TbjyEISTbccUoK3yCQs9YbYaUluLgouWhKGPv5VZlId2aFrG5e0gBchNnemoBG157k2itIjJZieGn0v5p9jJEN16cRnBjefa8dJZiJH6Ibw0cnR2oxrZbI1Ds9oug== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:40 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:40 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 08/27] iommu/arm-smmu-v3: Move allocation of the cdtable into arm_smmu_get_cd_ptr() Date: Wed, 6 Dec 2023 13:28:14 -0400 Message-ID: <8-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR22CA0002.namprd22.prod.outlook.com (2603:10b6:208:238::7) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: ae7fab7f-ee34-4ec4-1f58-08dbf680c644 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /oMlTop4DS4zUuizuvzH558I46CHzDScbGaWueR76pF514BnMprJiPc1ov79sDXXcJwLC+G6KikRGRmIma7QqT9lhxOvyvt7ZEYfklWuXDw28H5p9Sfh+svkS2Usnfxp+gkoFFhHz9F5hnaoBDyv8VWYCML6nGV120VmpYueHvxSuzbQpbIOsT6F+CBLCYv0/F4sVyfnK59PsnzYIQFvSBQxAj2JqyU95o7iR40u+OJa9Duftle1RRWaop6U0bb1Y4IyYopuvdzego8hl2MbIfAUZas0ZT6xxzQGg7sc7TBvVZLuDgKtLQIv6CxFt2KM4t0xxnsnZxPR1gRm/lYuLzyayg0sZy+Neq9QkyjkiDjpvFjiYlApeGywEJHxdnjq9UbbhjSmdbIis+8L1F2gaGYIQnQXmnZaqrm7uoAhd/kayTE+sTd3rpzYJqFo0n6Y24d0RaKdj2XcHD4OcfFFnvHLFnuwwdASkqbGJbjHJZ/bekZXu9chmzAg9DnWwfRzSScD2Tzg4mURihSN/Lm0E+VCLHZcgHc5LMV/TvbpPgv6q7xaLbidgCsj4e3d7rSGyYER9SOnqdG3JjNd3nQNCx9vcZPZFFwBHgpQaXtxhe8MaZ5V8dGKRJeF0BAJqsvPyLrGqPZwopJ2h5oK/Ywd4Q== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(4326008)(54906003)(7416002)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005)(4216001)(473944003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: /bmY6TupidND3Q/lL9IkstpO+q/L+aIHskeYY71fh2yh/ZyYH9WqrC22NaulTbOBouVf+oIwP306MxmXhaBEqDVK5HmhiGDULk8j3t1q32NGngxLWhQeuH8NMTTihWSoAWjOOP5FxaYHtHIk0cRXMrpVWYfum//qixfOHwjSRmJDgvbbJMxP2IPZM5Fxdf8knMvBgHg+h4baEZIaJpJfNACBmxeQGMC6tHXQOXHBDQBqpa+RsNE8ArqQK2ZclzjNZg5IVgEyGvM55ToMQbRfvFshF8FA3ZoRsJ6GdlkGc3QkGykRd8pTd2h5WpVjt6tJ3F+/0xq2/mQA1NHC/RGfeA8bYcpbEqCfLYf097aZeGBrDe41oBrFxC6plzmpih0fvA3VXlmbkZf/kaphLCvP3sjfqmq+k4/vi+nsomMYeokEZctM3PoyUl8hLD9peJybaCl1dLn+Apy2e/Q1JokDtKVaobtJ6A3vfeF3fCBM9TAxWnXtfZTINGM7iSi7WzmLpdnoxD9PWxzBlj5M/QW2wnFz03ukKWBRtSlNjGv8iZKFax7Afb9jR72R3kSYT0JBPophU1XnG1y7f9SHpi9g2cZdUrdEg0WvwKvUwmu9s6XDtcHrALP5mAQ9id6+XHfT44sSHA/3thI1rxI2Z30QobTJ5S699YPaahLAXZrADzYA1eLpZh/G6MLQVSllxp+clKKtUQ583emNYSdyZHnhEGFRg5kYrdh7uVGeI9WV6eT6T3tTe5UbJx+OjNCKD/2NUKfJghEWV4Y9/NpU/Cmm8YW0Jsc1ZlYl/WuiC0k5xuU4XrHJp0RLb1eBRqgE8iHEXf9QgE0+Fy/BqWRtALHDf6njfg1F1PEig5y03syPm3OYRc2OEEFLIuwb/f9OZy5ccOhy6Juu/cYJFVjaflDORybkRGSRPR/B7HN740mhdvpIUdZ2VYLpmmPcwdhG77tQXqyNy/rT7aebECbWSs8+en/KkKxMWINPBOcApuySn1f20qozoufvvSGOlozMBun2l4zoyCJeMemD4I1ZuXomXqupoNbbVB+z3FXsI9C/f2OTEaEA0kcyE7VzV10C0WPOhG4UJmt0e+KwsF3Jn8BpChWziRjgW+qhJ6L6l1rty+8Xeq0ZFO9gNGuhoIu7cK2bUQ3XsZ76xBjJUOPqHwDT/gAo8B5De96o5BuTcTA9nZj8rGquj0oif02xHnl6maRc0EzuPM5qMQ2JqqSKqpM8q4aRY2ouSyruyry5Q497Q2HDUbpJ5hp9DkOhTA55WZ4z1VTFbofw/r/jm9ocdTLISj/y5AwGi2YWC/ulzerK9Qe7z4DYsneE6dGad1ChyO+QrpW7MqSLiBbAoHTpaUAzgxLnqnvdxOTtqrrbtzyZvdxJuZ2h5JeU+fmvX21HIVZ8cYl5t54Gwmjz4JO0wDMVviaAaWudO/F7QMY+Gf34taZsdu6I4ZCPPzmjoXRCDdGbWR6JBi25Qw9cliPoFS/rNTIIbzdE6pc9raRKrBlMJX3AQy6eJeaAeyBE08Z2x0qeJaaDjuPQrxHA3/9WK6Z1mukrPPftI5+cp6UVSBaYaZmMlpk8NpeY3oq16De2yTZS X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: ae7fab7f-ee34-4ec4-1f58-08dbf680c644 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:34.6973 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: VgJyL88QKSls1CNoArxT9E6BEGw21vw7KDpfM/6X0GjXv7p10ZbE+8ul8fRzLY4c X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092848_508247_1B9BA62A X-CRM114-Status: GOOD ( 12.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org No reason to force callers to do two steps. Make arm_smmu_get_cd_ptr() able to return an entry in all cases except OOM. Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 696f837015771f..a21589f8729176 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -89,6 +89,7 @@ static struct arm_smmu_option_prop arm_smmu_options[] = { static void arm_smmu_rmr_install_bypass_ste(struct arm_smmu_device *smmu); static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain, struct arm_smmu_device *smmu); +static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master); static void parse_driver_options(struct arm_smmu_device *smmu) { @@ -1127,6 +1128,11 @@ struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, struct arm_smmu_device *smmu = master->smmu; struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; + if (!master->cd_table.cdtab) { + if (arm_smmu_alloc_cd_tables(master)) + return NULL; + } + if (cd_table->s1fmt == STRTAB_STE_0_S1FMT_LINEAR) return (struct arm_smmu_cd *)(cd_table->cdtab + ssid * CTXDESC_CD_DWORDS); @@ -2639,12 +2645,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) struct arm_smmu_cd target_cd; struct arm_smmu_cd *cdptr; - if (!master->cd_table.cdtab) { - ret = arm_smmu_alloc_cd_tables(master); - if (ret) - goto out_list_del; - } - cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID); if (!cdptr) { ret = -ENOMEM; From patchwork Wed Dec 6 17:28:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482098 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6B199C4167B for ; Wed, 6 Dec 2023 17:29:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=oFz85q1o2CzoyoaGqeKfN8V0LuedrLeD4gtcqh1JzP0=; b=Tc7c2L0IYfR7Sp u5sxMou8Rx63mJiijjjBazczqQ0EwPGb32TXxKI1OiyeXH6W6cOAiPLcd1dWQ96br/IqXpnMCVbdx y2hc5SWaG+XQZv8q13BW69O/2kXVfiOZqBlOGdlIQJJsOHJsfSaUp06Bf/pLV7Xlhj29ViiXK0HGU dYS8CZ1bdEwWq3QmYk7G3QvVnNbr+zuaTpoYwdNkcjGeHPZOskamIB0k8ET/YLN4v8tU3eOcY3wJZ BmQ4LKdUdWHV4Qeow4NpFnYEmu4EoVLkjLuqRiGl5+8+aPKJKnWyvjQAXceKAg8YFperfjqugRPH+ GCztL7F63M0YmHmu4chw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvha-00AtNa-1p; Wed, 06 Dec 2023 17:28:50 +0000 Received: from mail-dm6nam11on2062e.outbound.protection.outlook.com ([2a01:111:f400:7eaa::62e] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhV-00AtFn-2d for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:47 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ddq0M1K42FMTw/2/VvcqkMAFcl12cYHhRAQjoT0lYfpNWNiNi6kfll16qJXKtUWOqijntuvVf1j2v4KkGIuKhnOJXfgm2H8SYSOKOeu4hPauWCTUt1CkcrOzB7cd7ChQ54XHGDZ1lXMqa/DS1zHwaJRVV0k0cWIXRX/xAlPsEw4i3POluAgG2cfMzBczt/h+UBffmYLNlxJV664BwJlI3jXVZThDsyqDAQYzx4cX1N6HXcb4Qv5g3+yOq6x8SKa6/UVvYlPu3mwywD5LYiQayMNYOH0R/mdR2jbfnO3mvw2QxmhzT0oaMrzfl73+q5LQL1FtvOA7QLbf+dZIHowMcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bq9A//P/Fz/X6TWAwXikaNRN+4cbmGhSjzDHwPANg3g=; b=DKuQuvdJYEAAoAWILpBpv2TwHGe+RNfC591HHcIAcqYYMfh/VgGNw/MHuC3q5+bUqOi2dRXMHaXv2FI/UjgDBW1a61pI5GoeFk/ksP5cUjA4CSwqcwiukiwkQ25KPzSFtE4UXtNhVHbH5Ndb7oTW3J44D6J/KL1coMjoio0HQQHdOA3ID9frZ4EGf0uFM+k+Fo0QR9x5aVUYcpgzcRrqVWy3FARlIwwterTt9vCXm7q7ZXNEsBqa6MT8qTumtxpTqA339rYLs30myumwxRAen0dzl/axEJnSQNlPVrj89uwVMxCbwQx/ETd33eGw9TqLOp7NP8NUEqL6JaJewB8tVg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bq9A//P/Fz/X6TWAwXikaNRN+4cbmGhSjzDHwPANg3g=; b=PVtzJv8VJG3Jq3RxceMHbFO6wQ63+05IIpjA7+aqALaYQx5/0CgFystubo5UA0gFw5q+eKZRm+9Xqs8wZ3u97k7LS5Y8F3tQ1AoyMTYVW6OzT+iCAGctYfsxMbcDWYK9cySKzPfGrc4tux3XwtnSQHzAE1lWNXy9z/Hzq7/IUpSYamUPuHtIBTcenGJxV1YwuxwKQ8bfBCpKCe+ZWuLifLYp1t3fUzV1eeLU+8Ud0xrFVKLbyMwFOGrk1/GxfrbBLvVo5A+bt5wUQ4NU9Nsjjrg0VgsOMwClZFKrb0DntvbNuBlGJgBy6HNPix9baYvy1Vx/lcijZ2mIQVe9Y3kS3Q== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by CY8PR12MB7364.namprd12.prod.outlook.com (2603:10b6:930:50::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.25; Wed, 6 Dec 2023 17:28:37 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:37 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 09/27] iommu/arm-smmu-v3: Allocate the CD table entry in advance Date: Wed, 6 Dec 2023 13:28:15 -0400 Message-ID: <9-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR1501CA0003.namprd15.prod.outlook.com (2603:10b6:207:17::16) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|CY8PR12MB7364:EE_ X-MS-Office365-Filtering-Correlation-Id: 4340af18-e384-46f6-c116-08dbf680c5e4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: GcYm2GlSrQP7MJ9nxl7SsRGgK1qU9cZfCSnXZBAKl41W4s/4b4/j1Kz5qstZJuiV8php/cAWEzhdAh1VbBiaE0XfiUewjXLbvloWaaDr5FEOKHqdmK0ckuhsiJdLwugXEoVxj/xl80pu5fWQ1JHp2cA0eJAnVwbqq556iECfd+3/QOMZBrzLKKgEXt8oRg2zpemZcgCTCxQWtt0HDHP0NJNx4Y0FLhpUjiUW34L16WrZXkF2iHmicPm1EwLLTnQVucJoCaeRh0ZscFJOub7NdvK2Me0aybPxnGdvLTjZv5A5ByPEQOOAy9HQPyCOL0uRK4ztHKhqslj9bpY4kQexLO52eXgLm0SDx24p7pPNRi3iY2kk/x7niYlFUnxoyvJBwPJMWSoYCeLTg9WvD6XLUgjhSGHKaizxSAyWSrEsM6V2/GveVdmIPOf/ocuBBbrgn2gSfd3ITf7IjZ/pC3fGD7oTan/crQaJ6g1ppIH7fne3IFzi1NwblKMHbHtDY2PSDW/h2R5bSBoniOMAQbMmViYEHh6HpcUexolE904XdjaPDsPy36i4t6wRqurgqnBYN1CcysjUpfL+knJkeW6gfX6Eazx4gDUQ88uBkKs+gnE= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(346002)(396003)(39860400002)(376002)(136003)(230922051799003)(64100799003)(1800799012)(451199024)(186009)(8936002)(8676002)(4326008)(2906002)(38100700002)(83380400001)(7416002)(41300700001)(36756003)(86362001)(5660300002)(6666004)(26005)(6512007)(66556008)(6486002)(478600001)(2616005)(66946007)(110136005)(54906003)(66476007)(6506007)(316002)(4216001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 8+fkJ0trdFO+wLYj7z7wZs4HMtFu1x2yvis9aOl5a/vBIjF13BlMQgWn/yhd3TcQ8IedyS1FETpJzCnwVaM0U4Ijagwwtn1QpBjwPBV8v788I4XWGErEKwfzRsdxKtSrUuLGFPSFzdNpn0JrYbtUM3LuhmXOy1w31lp1zpT988/0lsGFsjyqyFkx9ysHyu/5fzhGKWLeuqFuIOeO/AqAkt4i1axh4phsM8nSaETdUMHsjvzQaWNw+fgKGdvEo7lGAqyQHZKH2/aUvmq6RyagUDwn75WCF6heMfbEfzw2qW72oa8HBsf7Myjb3oelaxo8PLgTvLQHGl7+UT5X3L0B7PfNR07+fQ1hPZzqh0+8F04cqG1CHZhjHYbYCZtEP/swLhfI0keGOcC6BletcpQblh+bA+Plymxv/vMBz9KRrP99lksUxAUP1nHu+KV8ii2kTrJPRNzfgSnD1g5de0Un7A50kqi1c1yBwXO7dl/Ko4BRKnvW93naa8zII6Qlj3YnkKLj3Qp6vV6gLWdlScABPHfHjeWbEonXi/i+8ZdZcbCbCKvMTyUpuGqwN/Vi/XrwEgtw0pDbbGnR+wBzVQEU3+upp4VFT51eU0zWeORIMVA4euOxI6ntln/Pz5Y1zXmjjPha9oFOI2NidqCC1hcG1ASIpFFQH0Dpupn6Zj5rwE7b2S+RjqYPpM79gxnz/fmxkRTo3PyVkrJZCQ9r6Hu8ikzmNJMp1bxCRMHNXQ7/S31QZIEIRqHqdwwUvPpFZyox0paee+SjIhJDDmlQY/QX5dbp4LQ6JY/qadDCX2stU0BH+5z36N8LHIr2V/6akK0L15ukztQCnvMq5VudjVB1qZftE1qVgRmw7rs07d4TYolwemNAM6z1aVsQQIhu+z6C6scWJGfIlfvZLaIwp4fnrFiiD7LY+Un3BvV2tW96mMax10tLrJOLjGYGqOjHHxxRzfXIE5HC0AAJ7YhW8nnNe/yIk+Uvf1Z787RiVG3H+qUHCyemdf9pOxe3+CYrCbvM+VjT2GGeBSll5q4hyZnx2twhFX/X3LL12kBaco4wZFrFiICkUyzamH/c/f1V7oaSWcpc1MzL+neIaZBBOTxeRuKiL4nj6Xzg+s2U/03hqXjDGiYxgpFwOxOXylETZmMQ2dRRpIUqPx2FuLo8FKHirmSARV8k2hZgevrvSrvga8/bNjZbNfEEKdWEfmKmVbD7vNn3Sn1QMzmBqnQjPCVHwKK45uF9WXT9c2liQsW5bHsHQSl6mAhhyQ38zOO7U5K93HFcPpGd0m/2kwz38fviyS1Aath/2097+hSdu2TRqYfVdyf/Mf5kNR3zeN95WD0+IDfBBD2Au7cVlzDL9g8/cWZZj7tTELZSKko7MmejwmKnupyCFAJqdnoQVI9Gdp9Z77U5gBy0jVG6hInHztW7uZkqJnQKR8CsTrgLhXrqCuJIT7gLTJ1dp+U8BvD8KdZkg/Sriq43qB/S8g+4Wy/iRTHpN9GnAiujcT2lFqcH2AFHeqoNI1RtvArzbFi08daM41F4FAOXoYH3Kmi9Sz1JG3vKKQU/1u7gUhL6TMJ5SQlxwjByT95oimUHQ5Fi7nAj X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4340af18-e384-46f6-c116-08dbf680c5e4 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:34.1230 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: fkXHiPBwYzOjHGDACeNKQfFQB8kBrF7GpjsRVGxt60kqWwGkW9fKQzh5PGDEPC3o X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7364 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092845_914473_62B50C88 X-CRM114-Status: GOOD ( 15.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Avoid arm_smmu_attach_dev() having to undo the changes to the smmu_domain->devices list, acquire the cdptr earlier so we don't need to handle that error. Now there is a clear break in arm_smmu_attach_dev() where all the prep-work has been done non-disruptively and we commit to making the HW change, which cannot fail. This completes transforming arm_smmu_attach_dev() so that it does not disturb the HW if it fails. Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 24 +++++++-------------- 1 file changed, 8 insertions(+), 16 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index a21589f8729176..eea561edaee908 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2596,6 +2596,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) struct arm_smmu_device *smmu; struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); struct arm_smmu_master *master; + struct arm_smmu_cd *cdptr; if (!fwspec) return -ENOENT; @@ -2624,6 +2625,12 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) if (ret) return ret; + if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { + cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID); + if (!cdptr) + return -ENOMEM; + } + /* * Prevent arm_smmu_share_asid() from trying to change the ASID * of either the old or new domain while we are working on it. @@ -2643,13 +2650,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) switch (smmu_domain->stage) { case ARM_SMMU_DOMAIN_S1: { struct arm_smmu_cd target_cd; - struct arm_smmu_cd *cdptr; - - cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID); - if (!cdptr) { - ret = -ENOMEM; - goto out_list_del; - } arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr, @@ -2666,16 +2666,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } arm_smmu_enable_ats(master, smmu_domain); - goto out_unlock; - -out_list_del: - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_del_init(&master->domain_head); - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); - -out_unlock: mutex_unlock(&arm_smmu_asid_lock); - return ret; + return 0; } static int arm_smmu_attach_dev_ste(struct device *dev, From patchwork Wed Dec 6 17:28:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482148 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C208DC10F07 for ; Wed, 6 Dec 2023 18:31:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Y2N/P4orBy9dZSPFV/3yVFLUZZWP0cDJXN7DmvXH9LA=; b=DhLOnodDl/QmM0 XKiOf1ycGBmo9Qx+JuYoLpILRz4S/IR8zKFbCaL7Wk8hLe8H4TU2az1d9113KL5HTUpNBLjtV1NSA yef/DxrBnc8NMT6aJuQOs1CLgwnA/sqet8cLYPL5ojnpBQqhRSZcMKe2WZYUORR48g4qydQ+b6mwt ZTIS64Vv37+xzpT4O8WjcdXJAAScmPgRImkLdG+NY8dNwPkl3lKRvXtEQ/lbKDJT6FGYXsqwaiUN3 RLrnoVChAyOjBY6lyIemF5gvygZkIF5ACRvShpXj2VBdRjSd3+t5wc8ok5xDuMjWTAORIDimlYisc UKw/dwzL8aXZS9SU3VaA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAwfN-00B29u-05; Wed, 06 Dec 2023 18:30:37 +0000 Received: from mail-dm6nam11on2062e.outbound.protection.outlook.com ([2a01:111:f400:7eaa::62e] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhZ-00AtFn-1Z for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:52 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nVo+8foc5j+GTUetqKclUDV/L1lmlLnYxr789bEwEAY378dkWWoRMHWyrezjJEUpTRRe6QUZY0OjlcYx7IIiHaAlZ6kO0W/x1U28sGbzbHwJ1mi3lqqd5/cw4wz38IsF99dMYvKsDcRATOWFYjMbidAiFhiYtnf5XuecVBnRhumL+c1pJz60UXFQJ0kZM+VT8vgEcfUgMGkSvQgT6JPeemJplls7pBV+llL71mlB96rNbcFK6kNZEdd8mRcmNbHD3PXtA83FmjE9l4NCbL5hYppI5zmXXc6+D14EHGNA6Jcr7egh5tsOe3sapMZ9RaaiuXAjq1yUzRjMOi2t3olz2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=EFCRnO21BqzYV20TDmy5w+rK5Gewr81V6zaew1sl+S8=; b=KejOpt6fwnWRnAYEgje3XRWGUXrlNIsGGNpwM4GnZhXJAFVD8lBkYVKaNyOYgJOw9bvaGreqZ+NEL9r7medtH9MrH3KimXVICMiMd+C5/lxUxweETCbU5g61/A2TCU8mTCY/puuVX90VjbplYOY5XhsrhU9OwTkXviTgQk0kMB9jGomBjxPEeTND9nwve85/DKCsSDcgAveFBRqps4Culm26XrEm5nBZPCMa+EoaNOhoCwzb1s9reU69TcNgV2LulIt0We7ImAXC8cYJ0x6dMp8TH1BNX4ucC7bxfFUv5NUvEAif/36TZ5mw27p2NQcr9F5wuk1q/kE2kkFnmbShPw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EFCRnO21BqzYV20TDmy5w+rK5Gewr81V6zaew1sl+S8=; b=ElQuqOjjj8V9sXwtlWJlM4Hy/JW36CqdxsjFxP0c73RacyaVxEy7FEAKflm8yN7L2DXfKFj/MoJ2zTIetRdfejLkFbVUoeHck4g7IBQU5wF55xQ95+nBHlYWv7ZPtcPp4g1myrW6+QQGZgzumOE6XO4+c5HuZCRlIn93wESHIYmPMvZvFihIaoB/ZfZcuk2MuWDohjRlz0tivrOmTsTiNoxmkGkoZSQZneRW7oIjfHLy+ow4C+yHHjWbfadh56CMyOF+I8AXixJc5pxOv59WpgHZTZoWUqFLxEHrWHSmand4vrcfd7/azZSJIbQqkUKcgnE1lt0OY4nX+YcyIP94jQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by CY8PR12MB7364.namprd12.prod.outlook.com (2603:10b6:930:50::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.25; Wed, 6 Dec 2023 17:28:37 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:37 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 10/27] iommu/arm-smmu-v3: Move the CD generation for SVA into a function Date: Wed, 6 Dec 2023 13:28:16 -0400 Message-ID: <10-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR0102CA0059.prod.exchangelabs.com (2603:10b6:208:25::36) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|CY8PR12MB7364:EE_ X-MS-Office365-Filtering-Correlation-Id: 86050df3-9827-4a3e-98c9-08dbf680c5fc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: a4imQmv3CiLR8Tz9TpT7aI16qZtUfuv0v8VjRgz8ImEf0WFRdom6P/EQXSqiPNCSNJKKfhNgHuY4I5ZXogjhDLL9hkpE4hZvax0wlLKDBeSpwb+wfQHJAgpusREZ9TcjnhM28y/0wTvRFw6z2oiXakQPDvSwFid+Kr9yPlQ5lY65eZ6cU2xQU+k8FI9UvgHMyuCyaLaWEXh15OXk4NrZkatPYhBDUqUS/MgY3AaLYFZwnj1yK+MGHyakqo0xJyFOwudSveRo0iKiEympXeRjkGKFPCeft7U1S9M/GeNYKTme+7KG6GK/KEIBZ2RKv6t/Mg9C2VsDERR0ynXD7WK/gQrKl56WFKL5QkUAUMpwu+OH8OqLdf8oQJNtC6JblpgI3/sxG9Z/IpJAfajpBLG5oatH5gUy3yFYnKTJ+MKw4hkHEJ79dDDWM6WeDTBi8jJKIL+kK45sNKhrohYbUQeGddbL0MBIhawtwoBRj1zed1HlLkHLeR17Nng8Z56e8gm40OSuvP1kTRLsPJUK7rpo7lAGZSn5LvX+nWdEK39iVKmYis48tGE//CuENOeQsBtKeA1qpxQIbxbnsTuZVoaEmS9WGiEMJ0uugrqGdVwYEBu881qmUTEK3yildZuIerY8aon0zj+QkGze/HD4Y6yKuA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(346002)(396003)(39860400002)(376002)(136003)(230922051799003)(64100799003)(1800799012)(451199024)(186009)(8936002)(8676002)(4326008)(2906002)(38100700002)(83380400001)(7416002)(41300700001)(36756003)(86362001)(30864003)(5660300002)(6666004)(26005)(6512007)(66556008)(6486002)(478600001)(2616005)(66946007)(110136005)(54906003)(66476007)(6506007)(316002)(473944003)(414714003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: PWdfn0/fmU6p9Jy9LVSW/YIeGli950ZCUdUV2p+TZI9EvfQ4X0fB4xTYg406dcJoFKLOkW4sE5kcGUcveftNhQxhgnJYex9YeyTdbGf6OzrV+zza/NOH2g7Xr3wgdAtfsMSV2JX1c9g2TJTjSurN0KFZe9c4wlIjwewMu/g1u1YjIZc83Pj1GoP5KhHy78Q//PQOwce928/kpXbQinylv18VAJ3Q/vo4zVvhSxG0JT+a73ujm1W6bQOoD0Q5i8gkbmx0ZScslO4UOWlRNjZtf1WJx8UqiPGy3M8uPSu/0HB9TrCM8wDtItPBxcDRMTfkzNxsWNYe1XZflhGtlPdkSJvtteG20NvUt4/BzMPqPYlgPBb44KxdU6MqczVfZ6bxC/msQXdO2/jxjhtAgab51LbS7RHztlJ2bSvSoYHPaGKGYGo4NZMwv7r4ajtTiQzR/n/NL8ng/2pgK2bk2+m+JK44LxNKSuhA0GyKDPyVqoCl9Qg+M6ZDgtExb4HWNB71ueImah0Xc6lLNFh6r9ja+JzElt/Jd2ygVaYasJO0A1G0dsb6nrgxzYzpfvFbuGLUpuIAyYFaALYWXJQcBO644QsnK2xu4SdW0iC9mLtKwFmmtRPikH2XskulyRFpqRlTaGYlKOecckU7hYsABV5YPNUesPoenbtHCTuPrlWKaJyiMJlTybggYZE/CMRTBbefF85UWzIYvkrpihFHvUVx5fAlKscFv0ZYlQgmx5v/99q1UPnXI8lK+LzvkrkES6X/fy86XyDHC2Jx8LhWu/go3LLh2Nam8uvqMzZb17uNi2A14gk8J3YiQ5P4Bt3eqXkH1ySYCHxPZrB1J8SNOg/AQrMyH1q1fMsoTPeuMDAXgeSMreIq+JZNIT+swGQqOUlmqFDrbXdMhpxdJcVvmXk3gBN/TxzLPYTJmmpTfLYo3y3XJVYxqMvV2mwnJOu4v4LRNdRlkVeMWfBveKR/ykUk4azDG8veXjQR4YxMhO1mGUL1Fpmn57d8ZBbRMybwzKe17zof1yuWtWpWgCD55aODXS0kJf1H5TdfYWsKqZkUnWSpEvZjiC/g36OGpmyYZoaUmNHtUZNhcAKAy1V/C/xdseaxvB71qrN5I3M2S6fsZMjiwQjuWBZupuqsH01EvjrITWHFO8XGt2vrqOmuM3SK6s42Cpz8lvc/kc8lsRoQEnIjG4T6lvYMGgEc/jxxHfaCnjZr0t3TTUI0bTq9LW3SvEmE4TY3QxKAAd+a6Xq3yHFQymhRP2FLwHCDed42Q7N56dQ3ZrP84p+qRJRstGNyVdceunU3UsTgT5SHsgSRA3vypUHwbK0hy+HErUXn7LgOvGLdoMR8DrRG4K7qY8DO9i7AqGihs8Ed0OTJAgDpRWy8DysHSNFCBBITiMzaVo+uBy+eSFVxpUO4G2Ffiep+kAfZ9i1V37QVZqK+mSGV2vc+3VAQwQIsh379tKFKX6NyW6dZFw6cHf3gl3drBcpBTh6hMbST8PcdLpI42D4Zh+cQ7wwa2s/5A6zTnS5GKhgT69iEiUS4izqxIFcxSQtDu4oWnNfiNGf4h2sohD3IbcKnxR2rq8RBnxVXwlAYOY1y X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 86050df3-9827-4a3e-98c9-08dbf680c5fc X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:34.2500 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9CTgAROdMx/WjLciow7oO42e9d+FRptXji5vKllbt/FsGueqmUuayUbTHV5nLanT X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7364 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092849_543767_87CD2A61 X-CRM114-Status: GOOD ( 23.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Pull all the calculations for building the CD table entry for a mmu_struct into arm_smmu_make_sva_cd(). Call it in the two places installing the SVA CD table entry. Open code the last caller of arm_smmu_update_ctx_desc_devices() and remove the function. Remove arm_smmu_write_ctx_desc() since all callers are gone. Remove quiet_cd since all users are gone, arm_smmu_make_sva_cd() creates the same value. The behavior of quiet_cd changes slightly, the old implementation edited the CD in place to set CTXDESC_CD_0_TCR_EPD0 assuming it was a SVA CD entry. This version generates a full CD entry with a 0 TTB0 and relies on arm_smmu_write_cd_entry() to install it hitlessly. Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 145 +++++++++++------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 81 +--------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 - 3 files changed, 95 insertions(+), 136 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 04a807774402b2..1f0940b497f3c6 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -35,25 +35,6 @@ struct arm_smmu_bond { static DEFINE_MUTEX(sva_lock); -/* - * Write the CD to the CD tables for all masters that this domain is attached - * to. Note that this is only used to update existing CD entries in the target - * CD table, for which it's assumed that arm_smmu_write_ctx_desc can't fail. - */ -static void arm_smmu_update_ctx_desc_devices(struct arm_smmu_domain *smmu_domain, - int ssid, - struct arm_smmu_ctx_desc *cd) -{ - struct arm_smmu_master *master; - unsigned long flags; - - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master, &smmu_domain->devices, domain_head) { - arm_smmu_write_ctx_desc(master, ssid, cd); - } - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); -} - static void arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) { @@ -129,11 +110,76 @@ arm_smmu_share_asid(struct mm_struct *mm, u16 asid) return NULL; } +static u64 page_size_to_cd(void) +{ + static_assert(PAGE_SIZE == SZ_4K || PAGE_SIZE == SZ_16K || + PAGE_SIZE == SZ_64K); + if (PAGE_SIZE == SZ_64K) + return ARM_LPAE_TCR_TG0_64K; + if (PAGE_SIZE == SZ_16K) + return ARM_LPAE_TCR_TG0_16K; + return ARM_LPAE_TCR_TG0_4K; +} + +static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, + struct arm_smmu_master *master, + struct mm_struct *mm, u16 asid) +{ + u64 par; + + memset(target, 0, sizeof(*target)); + + par = cpuid_feature_extract_unsigned_field( + read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1), + ID_AA64MMFR0_EL1_PARANGE_SHIFT); + + target->data[0] = cpu_to_le64( + CTXDESC_CD_0_TCR_EPD1 | +#ifdef __BIG_ENDIAN + CTXDESC_CD_0_ENDI | +#endif + CTXDESC_CD_0_V | + FIELD_PREP(CTXDESC_CD_0_TCR_IPS, par) | + CTXDESC_CD_0_AA64 | + (master->stall_enabled ? CTXDESC_CD_0_S : 0) | + CTXDESC_CD_0_R | + CTXDESC_CD_0_A | + CTXDESC_CD_0_ASET | + FIELD_PREP(CTXDESC_CD_0_ASID, asid)); + + /* + * If no MM is passed then this creates a SVA entry that faults + * everything. arm_smmu_write_cd_entry() can hitlessly go between these + * two entries types since TTB0 is ignored by HW when EPD0 is set. + */ + if (mm) { + target->data[0] |= cpu_to_le64( + FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, + 64ULL - vabits_actual) | + FIELD_PREP(CTXDESC_CD_0_TCR_TG0, page_size_to_cd()) | + FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, + ARM_LPAE_TCR_RGN_WBWA) | + FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, + ARM_LPAE_TCR_RGN_WBWA) | + FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS)); + + target->data[1] = cpu_to_le64(virt_to_phys(mm->pgd) & + CTXDESC_CD_1_TTB0_MASK); + } else { + target->data[0] |= cpu_to_le64(CTXDESC_CD_0_TCR_EPD0); + } + + /* + * MAIR value is pretty much constant and global, so we can just get it + * from the current CPU register + */ + target->data[3] = cpu_to_le64(read_sysreg(mair_el1)); +} + static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) { u16 asid; int err = 0; - u64 tcr, par, reg; struct arm_smmu_ctx_desc *cd; struct arm_smmu_ctx_desc *ret = NULL; @@ -167,39 +213,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) if (err) goto out_free_asid; - tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - vabits_actual) | - FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) | - FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) | - FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS) | - CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64; - - switch (PAGE_SIZE) { - case SZ_4K: - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_4K); - break; - case SZ_16K: - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_16K); - break; - case SZ_64K: - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_64K); - break; - default: - WARN_ON(1); - err = -EINVAL; - goto out_free_asid; - } - - reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1); - par = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_EL1_PARANGE_SHIFT); - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_IPS, par); - - cd->ttbr = virt_to_phys(mm->pgd); - cd->tcr = tcr; - /* - * MAIR value is pretty much constant and global, so we can just get it - * from the current CPU register - */ - cd->mair = read_sysreg(mair_el1); cd->asid = asid; cd->mm = mm; @@ -276,6 +289,8 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + struct arm_smmu_master *master; + unsigned long flags; mutex_lock(&sva_lock); if (smmu_mn->cleared) { @@ -287,7 +302,18 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) * DMA may still be running. Keep the cd valid to avoid C_BAD_CD events, * but disable translation. */ - arm_smmu_update_ctx_desc_devices(smmu_domain, mm->pasid, &quiet_cd); + spin_lock_irqsave(&smmu_domain->devices_lock, flags); + list_for_each_entry(master, &smmu_domain->devices, domain_head) { + struct arm_smmu_cd target; + struct arm_smmu_cd *cdptr; + + cdptr = arm_smmu_get_cd_ptr(master, mm->pasid); + if (WARN_ON(!cdptr)) + continue; + arm_smmu_make_sva_cd(&target, master, NULL, smmu_mn->cd->asid); + arm_smmu_write_cd_entry(master, mm->pasid, cdptr, &target); + } + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, 0, 0); @@ -348,12 +374,19 @@ arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain, spin_lock_irqsave(&smmu_domain->devices_lock, flags); list_for_each_entry(master, &smmu_domain->devices, domain_head) { - ret = arm_smmu_write_ctx_desc(master, mm->pasid, cd); - if (ret) { + struct arm_smmu_cd target; + struct arm_smmu_cd *cdptr; + + cdptr = arm_smmu_get_cd_ptr(master, mm->pasid); + if (!cdptr) { + ret = -ENOMEM; list_for_each_entry_from_reverse(master, &smmu_domain->devices, domain_head) arm_smmu_clear_cd(master, mm->pasid); break; } + + arm_smmu_make_sva_cd(&target, master, mm, cd->asid); + arm_smmu_write_cd_entry(master, mm->pasid, cdptr, &target); } spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); if (ret) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index eea561edaee908..824e725f905d71 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -74,12 +74,6 @@ struct arm_smmu_option_prop { DEFINE_XARRAY_ALLOC1(arm_smmu_asid_xa); DEFINE_MUTEX(arm_smmu_asid_lock); -/* - * Special value used by SVA when a process dies, to quiesce a CD without - * disabling it. - */ -struct arm_smmu_ctx_desc quiet_cd = { 0 }; - static struct arm_smmu_option_prop arm_smmu_options[] = { { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" }, { ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"}, @@ -1115,7 +1109,7 @@ static void arm_smmu_write_cd_l1_desc(__le64 *dst, u64 val = (l1_desc->l2ptr_dma & CTXDESC_L1_DESC_L2PTR_MASK) | CTXDESC_L1_DESC_V; - /* See comment in arm_smmu_write_ctx_desc() */ + /* The HW has 64 bit atomicity with stores to the L2 CD table */ WRITE_ONCE(*dst, cpu_to_le64(val)); } @@ -1192,8 +1186,12 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, const struct arm_smmu_cd *target) { struct arm_smmu_cd target_used; + int i; arm_smmu_get_cd_used(target, &target_used); + /* Masks in arm_smmu_get_cd_used() are up to date */ + for (i = 0; i != ARRAY_SIZE(target->data); i++) + WARN_ON_ONCE(target->data[i] & ~target_used.data[i]); while (true) { if (arm_smmu_write_cd_step(cdptr, target, &target_used)) break; @@ -1240,72 +1238,6 @@ void arm_smmu_clear_cd(struct arm_smmu_master *master, int ssid) arm_smmu_write_cd_entry(master, ssid, cdptr, &target); } -int arm_smmu_write_ctx_desc(struct arm_smmu_master *master, int ssid, - struct arm_smmu_ctx_desc *cd) -{ - /* - * This function handles the following cases: - * - * (1) Install primary CD, for normal DMA traffic (SSID = IOMMU_NO_PASID = 0). - * (2) Install a secondary CD, for SID+SSID traffic. - * (3) Update ASID of a CD. Atomically write the first 64 bits of the - * CD, then invalidate the old entry and mappings. - * (4) Quiesce the context without clearing the valid bit. Disable - * translation, and ignore any translation fault. - * (5) Remove a secondary CD. - */ - u64 val; - bool cd_live; - struct arm_smmu_cd target; - struct arm_smmu_cd *cdptr = ⌖ - struct arm_smmu_cd *cd_table_entry; - struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; - - if (WARN_ON(ssid >= (1 << cd_table->s1cdmax))) - return -E2BIG; - - cd_table_entry = arm_smmu_get_cd_ptr(master, ssid); - if (!cd_table_entry) - return -ENOMEM; - - target = *cd_table_entry; - val = le64_to_cpu(cdptr->data[0]); - cd_live = !!(val & CTXDESC_CD_0_V); - - if (!cd) { /* (5) */ - val = 0; - } else if (cd == &quiet_cd) { /* (4) */ - val |= CTXDESC_CD_0_TCR_EPD0; - } else if (cd_live) { /* (3) */ - val &= ~CTXDESC_CD_0_ASID; - val |= FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid); - /* - * Until CD+TLB invalidation, both ASIDs may be used for tagging - * this substream's traffic - */ - } else { /* (1) and (2) */ - cdptr->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); - cdptr->data[2] = 0; - cdptr->data[3] = cpu_to_le64(cd->mair); - - val = cd->tcr | -#ifdef __BIG_ENDIAN - CTXDESC_CD_0_ENDI | -#endif - CTXDESC_CD_0_R | CTXDESC_CD_0_A | - (cd->mm ? 0 : CTXDESC_CD_0_ASET) | - CTXDESC_CD_0_AA64 | - FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) | - CTXDESC_CD_0_V; - - if (cd_table->stall_enabled) - val |= CTXDESC_CD_0_S; - } - cdptr->data[0] = cpu_to_le64(val); - arm_smmu_write_cd_entry(master, ssid, cd_table_entry, &target); - return 0; -} - static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master) { int ret; @@ -1314,7 +1246,6 @@ static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master) struct arm_smmu_device *smmu = master->smmu; struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; - cd_table->stall_enabled = master->stall_enabled; cd_table->s1cdmax = master->ssid_bits; max_contexts = 1 << cd_table->s1cdmax; @@ -1412,7 +1343,7 @@ arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc) val |= FIELD_PREP(STRTAB_L1_DESC_SPAN, desc->span); val |= desc->l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK; - /* See comment in arm_smmu_write_ctx_desc() */ + /* The HW has 64 bit atomicity with stores to the L2 STE table */ WRITE_ONCE(*dst, cpu_to_le64(val)); } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 950f5a08acda6d..6ed7645938a686 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -608,8 +608,6 @@ struct arm_smmu_ctx_desc_cfg { u8 s1fmt; /* log2 of the maximum number of CDs supported by this table */ u8 s1cdmax; - /* Whether CD entries in this table have the stall bit set. */ - u8 stall_enabled:1; }; struct arm_smmu_s2_cfg { @@ -761,7 +759,6 @@ to_smmu_domain_safe(struct iommu_domain *domain) extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; -extern struct arm_smmu_ctx_desc quiet_cd; void arm_smmu_clear_cd(struct arm_smmu_master *master, int ssid); struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, @@ -773,8 +770,6 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, struct arm_smmu_cd *cdptr, const struct arm_smmu_cd *target); -int arm_smmu_write_ctx_desc(struct arm_smmu_master *smmu_master, int ssid, - struct arm_smmu_ctx_desc *cd); void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, size_t granule, bool leaf, From patchwork Wed Dec 6 17:28:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482108 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DDD9DC10DC1 for ; Wed, 6 Dec 2023 17:29:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WGueZVxJSpwcLYVax32CEEomrcMSGubcvtzbgogPejQ=; b=YHkb/QDOLaDwG/ SrnBkmobwkNUHQyowkuK9wHs39fXfbl9oNn54JFBqrpnNzsIBP7HYzRr7xsRANXW8+rGTtdo2r4pr wd1nTY9D/sFk3xlizaBJi5jqPgCjq6+WXMHB197humAHtTFeFodOwJVEB9neoHvVBVZ5YZaVmhVbg Q2vJdVEwtSCrW7s0NwUxGyfBrsLzJFbe9ziubho8BJwlwJnTZDiz3G0IlPLTz4wE7aGuz6of9lVIi 9BmlXsxmhZNUkTcuZPR5y1+C4k91u70Z3FcSvd3SV+1LbLXSeNe6dR60LC6dhlK+N7UxDniIc+TEL Ir8LWMgUqjwgB8pkvs/g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhz-00Atoy-2d; Wed, 06 Dec 2023 17:29:15 +0000 Received: from mail-mw2nam12on20609.outbound.protection.outlook.com ([2a01:111:f400:fe5a::609] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhc-00AtIF-1D for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:54 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mP622xyar8iIqjSl5eK/ixG0TV4HanHocGHvczz6bRaIVeBuq/BuOda5TTQG1aESYV7NSg5SBZVo5v3oIq8byXGh33If+xpfUpGyv7nDTytvfapIXob78BcSs7DivNoa3uefTnL218wIvBBkjutwp3AWxE1Bq3XaACX4ey4bxpMlX7O9dvD4YCp8Y6by/CdcHZXdFaPO+cE/vGInf+Ab+5ogqMTQlI0twg5/xlmQoQBQsj90l0bc60Zhv7xkfESqr+wMLBEY3snTiRIHvhpP3zPZHu7ClGu7Bml3xhE1piNBqiLPkyX/vF7yk8ilfNPdW0XHnWe9DQ8WzpAG1J8I5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+env2I9YCegAfxYMJeXMpcHiXCL7djy9sLV+jg/N4QA=; b=m7c5/0TdoOhjG5GKkUSWpjm/DbsR9wgewVrrC/CVGDsmlxeqTdLcuPwfAabKmyluVFBKJdLZ+fpdW4rqozJIlL+d9YtCsY4CSPwmkqFbFmTNnIDw6lxg43ThWG++1iJixQOsdDPrJnIUBnyPHuRqQDbLjjcFAWwwqi5r6ArCgbN04qBXa50rj9+pfZNI1tgLbZzmONgu1eMbIue/bd13IX9yabSTEB+gKBR31dlIfCF1zoilRdfT4klMSYJ1Lpk1VLaneAHfngK/vJUmXGSJFELCAzvfo+B+C7mfniV63rXkQC0BX5UMEK3C5vNlsCa4eR+fWFB0BAUKSRFjQZGdzg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+env2I9YCegAfxYMJeXMpcHiXCL7djy9sLV+jg/N4QA=; b=J7R/fNYln5WJ+qtH0F8+2Xv5h1e9pdD/s961rKfsHuE44BdBN3DfSdZZS3Rkk8IgPAl8LX8sPqTlPFOTTM0SVELN+MEHS5P0H5ccMkhmeFMq+NO/T/xB5kAD47VApNW5l3Yzxn/ZAAZjRKaLZj0zktKECPAKl/kpqRzUFI0sjVgt1W7ezrAqgQJUdWEzB3ag8uoDHSBogiMTMzUnJIB8wKuDCN/N1XIS/2d2bbmpd+nbaQPO7l45yKX+QYWCPxAIRK9Kci7QBE8uM8l/h+ev57dxZMkQkTcMKuMPMjgtod//oH2NMJp0GhMWjT4+JnFz8HvCNKbkYHWjl8mOex7etg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:41 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:41 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 11/27] iommu/arm-smmu-v3: Lift CD programming out of the SVA notifier code Date: Wed, 6 Dec 2023 13:28:17 -0400 Message-ID: <11-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1P223CA0016.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:2c4::21) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: 0ead0505-0c60-4315-43b2-08dbf680c689 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: OkdhI8RzFM2nkGwsomY8Tikw52ssTD5ajYuaUQEI83PXBjayggqSCDHv/hRdMQsDm8gUa0/rrRv7qi/UrJrvIYjI5hxkjOQ4s+7zFu9/9fgQqQxpz7IQaIfa9MwipysyQXR/Th6DzDltEUfi5cW0cctX7/I6lcvm8aP/RuAdSDXFULJWhgoDjdK+tc2L75NDVNd2ywm/5l01AlTPKOLYdFq0XN/Bn/45Y/Shnjyy3UjqtIWpOq4l3nPjJPi/RaqoyohZCEJqthxh9ZcjzFUssG5/5SiY4vc3EyrwFIU4QaSsQuOqZiaoiSxHLr3ysQ1YwVR+Z/DcMR35BpxsYaDmdfxm9w+I/SGloMqM8QZJSiNTk8bK4xT4P2IkUSUeOD8afvm6A9a2No0mf7KfhpwrGzebk0vDOyPjb4SJAQRwXZHSN70Wn8WxVVLmUZe3PaAVqKSrI2Luooa8ObSsiAJRXlqN+CrDpSWPtpFiq8HUKAbiLcZX/hBF/WwhrMZoInMk9vE69FnoR6bZci37+d9H7/EK5qWQ6voxBT8RF0qniYyCdYX3GPbsZ7/K8shTtPi+ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(4326008)(54906003)(7416002)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 4OBG8dZRBZoDMV1tTFcm1x1i6bKYIlKlVKGipIiYzjxzHULnCSKb53LbGYUr3YJr+N/lphLeITyBQZVr7QDtr388n5MCz8WHTYCxvC2WyJZFSPVOwhNuBC1gLnMTJ96EtUX8nsB9QuFd/BUJFhW9PbAbyCQYUldRHwqgeJ5pzWeA2jV9BIZL4rEKakMWao7b6oOZ+ZJdRetrSMhsi1kXq/FhYfE0I4OxTrXWmqHHN+zZjN2JtOD1kET06ggdD0RRBHoAfe8XKCE9C1j25tn66RQkscTcJrYw8GQlparBJI/557r30GAw7O/TlExZeA2N1Ic9VG5FYslLm+1tPWAzJllfAdXJ5pJI8QMqU2oGD8oZxMCyGnmzIzzTTJRSKZak3IBn0RCHqo8ohadilVy4e2BDIAAGNLShTeyTtkpEc9XuFceG2hc5fo7uOmBmS1tl9sRYYGk7TiLks03FqoqgeHvyhqp5lFY4Q6PCGayrijJ3Aif9BIG+w0t9Oc3d3ylVAaCxSpr3Tv7bBJCU/PdkawaKUYapLuulwCJqhavXk0qioiSsZ9aWzG/6W0ouF5ECyzKbKAtL2e+AqWHwAiOVvoXCveHAPI+oz5/eEYkpljhNZj5YqVV3HoSxKzw2rqaqsf9FDTI5DCkqox79JGnkyrjOBWI0dfIUfFVp74LyH5x5dAiE+0qkPu2P0VCIU4WIFAWfMFgG40fLI4OaFr40wlaTMEQd7jyDCM4hu9NLq5QRaUE5PhB3NvtTMEXqFRUxhMX3LY7ZzmfCOLdy5uJ5y2bB6djnJgAWfym072cR3gkquWvWhFcx6h9/wLDq+8oSn7viLFEvGnJHDz6Y4vUxWDMzzH7XPRL/1aRGgnQH87obfkPun8Rdof+6k2RP+jagTfCGF5OPVg/jRjOXHfj5NbCRECHZOL7xxTp/KoaRy78gufb5z8nBAgVYi8fTSQEmDyCqKW1b2v57EHrvsgsYAXKcwkY4f2XCcrOQqbZzHl5TxpH8Pn1Ma+ppf/6z2pdkPiwoVcWZVVljkaAaeZDzkTsaM/c2VmA8hfaWHRfQz8KP6ztELLbOmwGFUsqpe8pALL/XyA05lWgxnsQwlyKEWtN0gOgdBHYnKRdk9rBiSM7On6VyagOM6f38elEpb03NqzeVPW1DK5BaDB+Fylj+FXY09ndEnwfhlPY71YWfWv2CF6mTy1sMYNiTsOw2sJ28JXnLNhnPX5yi4zA6UZdoUYNThbJhfyaILRpeCam5fUFC558zfBR9u1o8d6SnacmpBWVGvdDnO+uBN21xzeL4qrjbnDNPJ2KDB96uaP1wLqZStry1X4S5+N1W1VFIpogCjKyCNmtl75ExlM5TsChAHhbM6tutnMQCDfaK5oGM6T8fMkaeMjKXr1tdEr1ljtLezgoNaaq7sCHFt4hPM/0JkvZ6AGtIBxZcj81e313Vm9lmxR+5kk+19SMMTVPXQSIFoML2rZ6c6bS2kuvhNtA/BfP/ogrD1ealuUOPhS3PDn+o6AuASVZVKxP9wZ2ItoCfAwU3LDrB5lBAwaIQAcyQCpqSaGzJdIdX7YR+RG+gEJ4Jj6nHWJK8Xt5NJY2qGFzk X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0ead0505-0c60-4315-43b2-08dbf680c689 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:35.1802 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cgFyGERNe2Btwe/ihyPRvUMgHOQMyevGbgfYIrdHdNzp9p0OldS8Kga9a5kx8JyV X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092852_476197_F81F1AC9 X-CRM114-Status: GOOD ( 17.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org ops->set_dev_pasid()/ops->remove_dev_pasid() should work on a single CD table entry, the one that was actually passed in to the function. The current iterating over the master's list is a hold over from the prior design where the CD table was part of the S1 domain. Lift this code up and out so that we setup the CD only once for the correct thing. The SVA code "works" under a single configuration: - The RID domain is a S1 domain - The programmed PASID is the mm->pasid - Nothing changes while SVA is running (sva_enable) Invalidation will still iterate over the S1 domain's master list. That remains OK after this change, we may do harmless extra ATS invalidations for PASIDs that don't need it. Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 55 ++++++------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 24 ++++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 ++ 3 files changed, 48 insertions(+), 37 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 1f0940b497f3c6..29469073fc53fe 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -339,10 +339,8 @@ arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain, struct mm_struct *mm) { int ret; - unsigned long flags; struct arm_smmu_ctx_desc *cd; struct arm_smmu_mmu_notifier *smmu_mn; - struct arm_smmu_master *master; list_for_each_entry(smmu_mn, &smmu_domain->mmu_notifiers, list) { if (smmu_mn->mn.mm == mm) { @@ -372,32 +370,9 @@ arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain, goto err_free_cd; } - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master, &smmu_domain->devices, domain_head) { - struct arm_smmu_cd target; - struct arm_smmu_cd *cdptr; - - cdptr = arm_smmu_get_cd_ptr(master, mm->pasid); - if (!cdptr) { - ret = -ENOMEM; - list_for_each_entry_from_reverse(master, &smmu_domain->devices, domain_head) - arm_smmu_clear_cd(master, mm->pasid); - break; - } - - arm_smmu_make_sva_cd(&target, master, mm, cd->asid); - arm_smmu_write_cd_entry(master, mm->pasid, cdptr, &target); - } - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); - if (ret) - goto err_put_notifier; - list_add(&smmu_mn->list, &smmu_domain->mmu_notifiers); return smmu_mn; -err_put_notifier: - /* Frees smmu_mn */ - mmu_notifier_put(&smmu_mn->mn); err_free_cd: arm_smmu_free_shared_cd(cd); return ERR_PTR(ret); @@ -408,19 +383,12 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) struct mm_struct *mm = smmu_mn->mn.mm; struct arm_smmu_ctx_desc *cd = smmu_mn->cd; struct arm_smmu_domain *smmu_domain = smmu_mn->domain; - struct arm_smmu_master *master; - unsigned long flags; if (!refcount_dec_and_test(&smmu_mn->refs)) return; list_del(&smmu_mn->list); - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master, &smmu_domain->devices, domain_head) - arm_smmu_clear_cd(master, mm->pasid); - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); - /* * If we went through clear(), we've already invalidated, and no * new TLB entry can have been formed. @@ -435,7 +403,8 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) arm_smmu_free_shared_cd(cd); } -static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm) +static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, + struct arm_smmu_cd *target) { int ret; struct arm_smmu_bond *bond; @@ -462,6 +431,7 @@ static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm) } list_add(&bond->list, &master->bonds); + arm_smmu_make_sva_cd(target, master, mm, bond->smmu_mn->cd->asid); return 0; err_free_bond: @@ -624,6 +594,8 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, struct arm_smmu_bond *bond = NULL, *t; struct arm_smmu_master *master = dev_iommu_priv_get(dev); + arm_smmu_remove_pasid(master, to_smmu_domain(domain), id); + mutex_lock(&sva_lock); list_for_each_entry(t, &master->bonds, list) { if (t->mm == mm) { @@ -643,17 +615,26 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t id) { + struct arm_smmu_master *master = dev_iommu_priv_get(dev); int ret = 0; struct mm_struct *mm = domain->mm; + struct arm_smmu_cd target; if (mm->pasid != id) return -EINVAL; - mutex_lock(&sva_lock); - ret = __arm_smmu_sva_bind(dev, mm); - mutex_unlock(&sva_lock); + if (!arm_smmu_get_cd_ptr(master, id)) + return -ENOMEM; - return ret; + mutex_lock(&sva_lock); + ret = __arm_smmu_sva_bind(dev, mm, &target); + mutex_unlock(&sva_lock); + if (ret) + return ret; + + /* This cannot fail since we preallocated the cdptr */ + arm_smmu_set_pasid(master, to_smmu_domain(domain), id, &target); + return 0; } static void arm_smmu_sva_domain_free(struct iommu_domain *domain) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 824e725f905d71..27c0a468958dc0 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2601,6 +2601,30 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) return 0; } +int arm_smmu_set_pasid(struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain, ioasid_t pasid, + const struct arm_smmu_cd *cd) +{ + struct arm_smmu_domain *sid_smmu_domain = + to_smmu_domain_safe(iommu_get_domain_for_dev(master->dev)); + struct arm_smmu_cd *cdptr; + + if (!sid_smmu_domain || sid_smmu_domain->stage != ARM_SMMU_DOMAIN_S1) + return -ENODEV; + + cdptr = arm_smmu_get_cd_ptr(master, pasid); + if (!cdptr) + return -ENOMEM; + arm_smmu_write_cd_entry(master, pasid, cdptr, cd); + return 0; +} + +void arm_smmu_remove_pasid(struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain, ioasid_t pasid) +{ + arm_smmu_clear_cd(master, pasid); +} + static int arm_smmu_attach_dev_ste(struct device *dev, struct arm_smmu_ste *ste) { diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 6ed7645938a686..1c756803b7f963 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -770,6 +770,12 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, struct arm_smmu_cd *cdptr, const struct arm_smmu_cd *target); +int arm_smmu_set_pasid(struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain, ioasid_t pasid, + const struct arm_smmu_cd *cd); +void arm_smmu_remove_pasid(struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain, ioasid_t pasid); + void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, size_t granule, bool leaf, From patchwork Wed Dec 6 17:28:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 407B4C10F07 for ; Wed, 6 Dec 2023 17:29:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=dwhQlPpCzS4P/T9bUscAK1IpAe5IEk0UBxn/ku3DEcM=; b=NTuMX3t+prROQI urD3i40l8YFt8c6IhFNaApoMYhhhRGahs488/6OqGIlrv6OG7gGErX/cCXVVWiqx+s7RiKB/nqYsV qGey0wIln2lhY+6eOfAatSjJh7UCjc7PHDxD1a6H63LbnuCEz9SLNjCRx5FI2eakuGc+NHHECqxjb wPv9gOZoIULR6VrGD87EI30jqRF1b4pJfLi/7GqzSEJU2x5Gkg57OjG5aaXnLnDhgwkWfJ5cHd4Ip A4Q9Gt722emRcWjVlSJJGwnWA6bnBn7tV16lCExutZRPPUaTHzcfuaZDZ5AIpFQ96nYuiAxu/vVwY 71edtgoTSgXP519ztFww==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhY-00AtLI-2F; Wed, 06 Dec 2023 17:28:48 +0000 Received: from mail-dm6nam11on2062e.outbound.protection.outlook.com ([2a01:111:f400:7eaa::62e] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhS-00AtFn-1P for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:43 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CYQDWEXSjGenIBkCXgJzxIuAd9jY0ISMv6s4Bk4NnC+4J9L/ijoL1bnc+ZdFhzhx8rOoXifKnyedmpoukZvKKMJJwbCiSxaA1FfG6+GygBlC/wA8VjJN+3cjt3Fmk9VtCWoRKIwIVC971VunpQxEhhd74Ybxb4SjdwZxbmarGNZ1isdcUMYNvjOTtMw1R4mkJEuG7gZIPtTCyyN6/MRPJ2afXWujE2TY7bA3Kw0IDDQtLdrSxR2juauOoujCZV8EYUk8GzPgndFBjb28+lShIwV70ktmMuvXpoP/AcSxeo2lwhtJerlSTogZrKc258Jt9fVSjTWeGkjIe8c03XNn3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AzTs54M4R0dJXEkA7e9eEUaQVjze2XQHFwn806D0TLA=; b=Y4ud5YzCwYzTdlEbYp3cV46AfUZduFKmu4BG4mawLwSCqKQvNdU28pLt553EB75f1ibGYr8QEayMammC9TYwpD/JA8+4EcMDN4nQmZlPvUFufo13hx3u4zKnOSHhPHA3jMpGm4O6tQnKlhBJ8O8DnFWF1dqxfZcxvGDgxVG0Ji7+UKwBUvo1zQajDJThZepv2lS4PUBRYv/GBgY+LV0P9JEQdd15vGDAByzUSYb4KY/SGmoozf4dmUD3fTZs3RDaI+lO4GG3geSaeRXP3sqBSQa9kmccE5BglGepbxFJRc08ZjFNDbIfSKuTfPt7+SPlnNw7WCfra+AVEISD9X0J3w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AzTs54M4R0dJXEkA7e9eEUaQVjze2XQHFwn806D0TLA=; b=bJv132Qz4lgNx/twnPJbL4SkCBuNOdpCmd/nETn8/HVGdz94YiPhuA6cCqa/um7DaeDBuwAhlW23fyCXFtTpjGVpaQxP878fbLp1jbVFUotxjK3y+4nKLlcIlclLhVZiAPDC3TYTM9Qxkh5Y+PNU+bHYVdoWtgVQ9j4xXoy6D1reKZRfCXxp4XtMyUqMQznoEfArUwGR8GpFEfZDFtVlWv0WO4pLEuqjVPAp9CHW8UobWbsCotK3l8DelYi6emAQZezq/FbGFIpy/DE7/o/rO6WxtoZSHd48jBOp8EhaiIZmi64yQSvsauoTJh2Ke/OZf8xgrgwoyz0b486ACZJn2w== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by CY8PR12MB7364.namprd12.prod.outlook.com (2603:10b6:930:50::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.25; Wed, 6 Dec 2023 17:28:35 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:35 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 12/27] iommu/arm-smmu-v3: Build the whole CD in arm_smmu_make_s1_cd() Date: Wed, 6 Dec 2023 13:28:18 -0400 Message-ID: <12-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1P223CA0019.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:2c4::24) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|CY8PR12MB7364:EE_ X-MS-Office365-Filtering-Correlation-Id: 7973dee0-1f38-416b-24f7-08dbf680c5c8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: jtmy4q+58Wf9vAW0yTZtgYiEUcL4Kvvo9b3dR4KxzBFjDa/nnBGan3Uscstp35HF9uh/6UejFxMtuhvnZKeNu0KWkkMxaWtoLWQDAhK7u2STGvwmx6ntOBNbo2HCiDvKCtDYaHSuzyd4JeYnsZ6UJd7sLOAPwxKn7ABcdaaLEO8TH/2Wi3IsENT/AObvgB4STNbS+mrxtNlzmDeGp3xnjFgDbS7mH9DBdiDV/XUC/qN5RmsD2/cqCqH3NJrjJdO/i+8jRYPL/ekaKpoffFwjVIsHrKCJy4QZn0p7EN7n8PYw6db8kT8ZYIOw1xNcvL+ckFea1HG48RDRVWGFs1c9SJjW8o3swquSLdwdyT9uiWTyD6ISgxx015tJA6N7vCDQB4/xNlxB1gZna8aAuRGBl9+7Ip6y9mRWSRLkO56ZJJrrOhy8QXZTwbAbTHPD+eAD+pnNRzZkMzkX59yY1+pnsTKyFAsbBfmTYMQYXRIUWViUQ5a0L809+9+96isNcT82WpYteYPYUquBBZJl5jJ6axbkBKZ24ObqIlEgEAow1rJCD8HtDLsHw93WB0ovYCmV X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(346002)(396003)(39860400002)(376002)(136003)(230922051799003)(64100799003)(1800799012)(451199024)(186009)(8936002)(8676002)(4326008)(2906002)(38100700002)(83380400001)(7416002)(41300700001)(36756003)(86362001)(5660300002)(6666004)(26005)(6512007)(66556008)(6486002)(478600001)(2616005)(66946007)(110136005)(54906003)(66476007)(6506007)(316002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: zhxUOnFVIKQs9567PLgGThqVOIkfVvMrSSMgyhIS8MqEcULhFX1SZvNE8OhwxYl2q9TbpoRz6ofOIB2HOmGd40u2gajV/TxrD8oaKnjPcYPo7jt1OQad2W0tCEa9c6JX63HaYaXM/Kts2L8eWJ24xpxaNUFy4BFSaOjiTtfSh/HC+f9oQhe82hm1sdXLxfd+IxiVnI13+yVkI0oTbELX2q5R0YGmoYu3576mGDPelVvuy+E9QTwYZAAhT0cCGijr/MZyzxfz5YGVNYiLOumLB6mJiZycefyvspUB3NgwSIu88LFWI2TRBQme0jfGD+dtMZXYfoUgSZHn9nkRMUNg4DlDcHwWGwARFrCzhKAgn46NzsPvhhdZk5YkjJRcv+xQIocjgcTfkA1+++3JjGePNBIv4NNh9R/mUchbSU2GlHINJu79dcL27vKVxIn3wphyMtoZmrm9/XnrV8MyZkXOhyMitqjqMnbb8c1zJKMTSdHamaEnKML9yWymBdNIocWsEuZgcuFo9N4CmP9o8TauwE59ig4yUoMOu/DQTJT6NQdDojvD25cSA71CHh+/0o7c9Jj9PEt5bo5DtESiVthcOwM31/vjWxgC7meugs917YJFUA4fvd3aH2cmlKKRXEYQ0WQX2GFjEo8CRdzUMlx4oervDk0qKJVCI4lhQmvJBKERYtL2/rtfWDDdz7giVF0cuK41cHbFYZSRJWYfDhl2mQBGheFxYsD7MP39ATE8ArjQy1pFnEEuMqHahyZH73fBF90sETJTwEBepzGrn4JAKqla50Hd6oVudWW8xsSnYtKuB5YFztUgfT/y7c6n32rb7Hid29AKfgfMHqkinm0Fg9PTsWoWKRFFPQbMxiPrlQ1LNFlm+lVQikA8K6lrOiywGQUDyZ5XLMm8/jJ4WrPDu+2y50UxRj+Nqhs2QDWAv8KiIkBqVfFR+36VjIMAWMobh2BvHWNajc67frmS8VrIJXQ6jU4qt1Su6/Xl778RoyVZs83IQJjLFtn5gd/le04QHXL2XJbnJy8+fVw4w8bKukugQnbrvMDTxfNoh8ldPY86LOmX9eMg/g/5b0RuOksDU4tYqOh6fIo4WEQBJzcKWba8Ws8pwQQReOllaNxhtSCMWwhpvmOXswYGHFKqFmkpy48aRW23bNFuXbPDYsmtyUQpwat7v9BDjhBOWWMup+rnwfjne+U/p/pYLitB/JtZdXAdIkQC3blh9ipveXH70f/GUEBJt43cQuYXuX0rfglHHiun9L5kv2KeTf2mmO1/LL4Ga84X4sJ2sbFv65n53VRxHwbLrnWupT8Rhwa/3XGHKIoEhOzyqfc1y4CJWj6piKUqTJ4vl7/EXi+dp4l3gwxY/45cZnBlrwYN860KSGDSZu7f7Csr278FTS/xqMLdZ2hCWCyAxcSJk6FCCWWiyJPbtQwpIpzFv3d/3pM8L9iNhIcNcxA68z6wOKA1Nz7MzpCy9yWSw0tZBw96tQ9rBVMOGWwA6s7yBUn8Hnzn2rIdkVU8O872bu5nOotCiqAnBCKKNcG5CMuR8nD5+dB2gQXGB4+lN+oGPwFuSOXvoqvg79ILqeWMNyIaOFUqa3/C X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7973dee0-1f38-416b-24f7-08dbf680c5c8 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:33.8611 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XgCmqTgib5tiYJPvK9xMpH24pY/Vkir5OKx67pS+bBtgH+qgXvUHyPfdZdc306XU X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7364 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092842_481974_49844742 X-CRM114-Status: GOOD ( 13.73 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Half the code was living in arm_smmu_domain_finalise_s1(), just move it here and take the values directly from the pgtbl_ops instead of storing copies. Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 47 ++++++++------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 -- 2 files changed, 18 insertions(+), 32 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 27c0a468958dc0..38e83386f631f1 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1204,15 +1204,25 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, struct arm_smmu_domain *smmu_domain) { struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; + const struct io_pgtable_cfg *pgtbl_cfg = + &io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops)->cfg; + typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = + &pgtbl_cfg->arm_lpae_s1_cfg.tcr; memset(target, 0, sizeof(*target)); target->data[0] = cpu_to_le64( - cd->tcr | + FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) | + FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) | + FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) | + FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) | + FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) | + CTXDESC_CD_0_TCR_EPD1 | #ifdef __BIG_ENDIAN CTXDESC_CD_0_ENDI | #endif CTXDESC_CD_0_V | + FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) | CTXDESC_CD_0_AA64 | (master->stall_enabled ? CTXDESC_CD_0_S : 0) | CTXDESC_CD_0_R | @@ -1220,9 +1230,9 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, CTXDESC_CD_0_ASET | FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) ); - - target->data[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); - target->data[3] = cpu_to_le64(cd->mair); + target->data[1] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.ttbr & + CTXDESC_CD_1_TTB0_MASK); + target->data[3] = cpu_to_le64(pgtbl_cfg->arm_lpae_s1_cfg.mair); } void arm_smmu_clear_cd(struct arm_smmu_master *master, int ssid) @@ -2240,13 +2250,11 @@ static void arm_smmu_domain_free(struct iommu_domain *domain) } static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu, - struct arm_smmu_domain *smmu_domain, - struct io_pgtable_cfg *pgtbl_cfg) + struct arm_smmu_domain *smmu_domain) { int ret; u32 asid; struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; - typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr; refcount_set(&cd->refs, 1); @@ -2254,31 +2262,13 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu, mutex_lock(&arm_smmu_asid_lock); ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd, XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); - if (ret) - goto out_unlock; - cd->asid = (u16)asid; - cd->ttbr = pgtbl_cfg->arm_lpae_s1_cfg.ttbr; - cd->tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, tcr->tsz) | - FIELD_PREP(CTXDESC_CD_0_TCR_TG0, tcr->tg) | - FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, tcr->irgn) | - FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, tcr->orgn) | - FIELD_PREP(CTXDESC_CD_0_TCR_SH0, tcr->sh) | - FIELD_PREP(CTXDESC_CD_0_TCR_IPS, tcr->ips) | - CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64; - cd->mair = pgtbl_cfg->arm_lpae_s1_cfg.mair; - - mutex_unlock(&arm_smmu_asid_lock); - return 0; - -out_unlock: mutex_unlock(&arm_smmu_asid_lock); return ret; } static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu, - struct arm_smmu_domain *smmu_domain, - struct io_pgtable_cfg *pgtbl_cfg) + struct arm_smmu_domain *smmu_domain) { int vmid; struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg; @@ -2302,8 +2292,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain, struct io_pgtable_cfg pgtbl_cfg; struct io_pgtable_ops *pgtbl_ops; int (*finalise_stage_fn)(struct arm_smmu_device *smmu, - struct arm_smmu_domain *smmu_domain, - struct io_pgtable_cfg *pgtbl_cfg); + struct arm_smmu_domain *smmu_domain); /* Restrict the stage to what we can actually support */ if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S1)) @@ -2346,7 +2335,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain, smmu_domain->domain.geometry.aperture_end = (1UL << pgtbl_cfg.ias) - 1; smmu_domain->domain.geometry.force_aperture = true; - ret = finalise_stage_fn(smmu, smmu_domain, &pgtbl_cfg); + ret = finalise_stage_fn(smmu, smmu_domain); if (ret < 0) { free_io_pgtable_ops(pgtbl_ops); return ret; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 1c756803b7f963..b7a65e37c51e9e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -587,9 +587,6 @@ struct arm_smmu_strtab_l1_desc { struct arm_smmu_ctx_desc { u16 asid; - u64 ttbr; - u64 tcr; - u64 mair; refcount_t refs; struct mm_struct *mm; From patchwork Wed Dec 6 17:28:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482110 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 66410C4167B for ; Wed, 6 Dec 2023 17:29:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WJEgpfPf5cuwb9OpyD3ps6EWb6g4BgWOPVEYRBMAQaE=; b=nhDvyN24lOHaUi GEfLz5az+KlOr572iMh0dbsSTGwiGtSBsaKaJ37e+jtarAPdGfhnxENAHRt4cI2/m7Kdkwo3olxW4 6UGNHZto2GV6wDzqGvy6jPI0LhvG8d+JagxL8pmzSMFLGYf+/mhgvNBRK0HBl6NGo+vJUVPFWuyrO M8X54t3Te32/4nN5BJHLk/JsAsRfM60K9QuXVgF2sG2F6eLKYJ0WGrUO3s8E4pIK2PoWlbpfgQtbG GYB46524GocIn/Ghe3Vz30ACXPW/B9Wpj2A7eofPN3NUvPLJPJB/qXVJG+QhEgCWq8vgz9v9+uBD3 c1zWydu4ZqgtR09E5hJg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAviC-00Atyn-2u; Wed, 06 Dec 2023 17:29:29 +0000 Received: from mail-mw2nam12on20609.outbound.protection.outlook.com ([2a01:111:f400:fe5a::609] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhe-00AtIF-2d for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:57 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=heCJ92T/gIihu2+qC/VjtewFQLJqJhssfFCJygBMuDnBpKZOOdM+YvZVFyKeOdVltdzZseoij+ZfF//2AICY4zb17cFjsXmf7x5Zyy7nIFzN4ziJjvkRw2pr3WZsOhDLbBdADicMBAVvi3U2l1tVEfrk5SX5B1vFJisgRcLjeMeXdnUW5AvLW3CgB6rbNoZILLtMAWCrJGKJdbDRiFEak/0jJkaSpUrQk8U5TSbdwW3c/ppAjuOtcXetDmF3DAdwJqWmDocwUKNqf8c1XQd7cdeg6q1nlCvMmvjIuF/WfkrbKW3Jaro+NOzlTRczdgiNXaoSemKAxFONfT0ExBB6Zw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4vl6lp3QuHYqbaNJWw6aeMZow/i48DkCMDwU6zuHchs=; b=mTqTbwia8Mxsc16Er8TpY+iTjcuXMqOn2pbDQMBk4zMyrOG3cjk0BUA+Q3olx+OwPc1ISioHYhG37ZQreIUMxwbRlpq1ptwnwrAZ/MhEJfbKk5WRu2r8s82AQn8ZK21snTGF5tnc3oD+8oC2tG7Yx5yFgXP1rKHwfrZM99h73/T+BYaNQUAIa4XVxnDvbkKWwKpSm6C147txdCT6mz/qWyHyYKhsSeNnAQS70JgkAWFy+lH0HuA5vsTdFHpXvC5WGo8CwLlSNRqOcuqnnEXMDOrq9TwpjSbPngdG46KuPQHDYC5+PW4GsZ2nZaqphsJv4rJ/QU6gE6zFKoyezKB2Lg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4vl6lp3QuHYqbaNJWw6aeMZow/i48DkCMDwU6zuHchs=; b=c8l6htWNkIKwGynZNzNc4rFlHeQpo415zDO0msgaVUNnvKnyODkNK5ulZT29ZoK2gaHsnbI3z0lgQsUGx1UvAQa3h8FCbHq+YhMMRrOjDxBTvI8mo5oUM9OQ1dBzBVtQNRxPahvRKWKoTqOfRx+5JTjzQ960mx+VdJtQUlwGQ1vJnoEaHYsITAqLNSY32BV6FnCjn+T9bqeJRCx2oYhvOcdrgRQfM27Ti3TWSDpRp825qNSpEG7TdnTEu6C2EaB912GL70xfg860Qv0e+ufvdP26pegISTv20AjHxZX5Xppbca+GQxjvjFMtdqmWww/C+LDKV8aiOg0mlaunGPRd5A== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:42 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:42 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 13/27] iommu/arm-smmu-v3: Make smmu_domain->devices into an allocated list Date: Wed, 6 Dec 2023 13:28:19 -0400 Message-ID: <13-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR0102CA0053.prod.exchangelabs.com (2603:10b6:208:25::30) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: 4c1242ed-ee52-453c-2e27-08dbf680c6b4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: dN+YB2wtAPb5f6obOuMcGbmPWrzxyOqjtsGs9iyp5P+aZJAcgp1yaORZQ0ydxTpdYU3f+WVbhvKwH2O3zy28S+LvLickeQYM1G9b2k2UiC7FNkyGgaaaDDEbcUJBZwLhQrq/y+ThY8r3C6b9GKMmbJaZMBlzxFcQlTP5QWXegDn3taFM3i5wcVQ7CcDC+Kk77RoEnQ9X+4FOeafhg7nLoBMcMfCf1dN393nBPHiLeuPXKir4K+Zidc7z1Z/aZoy+pmbPVkUDMGHjVvZvTZY8Ktkll/T8u1bfUFppIBv9mDP1/QjziNPfhI8ExxpWWf/lD11eaDx1M6ArxciiKGmJoShP5UbBIaDfGfDdgmmI+WfWcCY402NHjXQdnEsqSoTtumu6uuPGSSosQo4RZ3ZfkYkFr8oSkWX4HZCQqmzskI1shPAlIrtUuXb7BPy7wr2e3yGLgWTEa0ORxtkfJW9S0PVxSN4DGzgXnmoQOBsl/pp4az8QDXxGihtnOBcSbXjaX+VIcWY49aeC6U1bCLlTHj9rZumMD6UyrvbFA29UwLBbXvA+8xG79j+9ig9qd0X6 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(4326008)(54906003)(7416002)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: vKYVZbIotAEcw6s8mLlw34t8s7CIpwZGo+UzWYD9xDPcR1OwwiZSawKc7SfAWlOHCLwWx5ksBGq35bawFNrmkEDtZW5437YJ+LZWwVkJ7zhJkZMEQk4zgM65G1jCKCeXanGIb686Ob5LDtG/uaUY0TtWuwJ9oVUqtXfCy7s0/dn4soUgtQRFtfAlevw+B8IH2QfP+okS82fXFQFScFSxaH6y/8msVn8KCy545gx+KqYQ7kKR1mejD77vfpJklloNHDGkXxOpxE/46LY2EMtKDF7xoCydq+8UdaUrhT5qKHnfeMm3GPu51MzqHAG78fcYS9J438/4In+rsr0KrnjtJ3GdBErqR325L77Myuv7Jf0aX3u7gd3F8EG/9wsq+YBZQ9wP3ShOy85Qux2yaS5Ftl/WvgcZc73LcTfkkhJ1Z9Hiv/Icjnb70tT9bUhbJYhdzuWB4cDUXSDcTb5OZhnXNzHvxa4DVrTFe0stUSxkOjyQHLZCY/I9sq+ZtgnS4zrVkUz3v64/y13xDtafFFjeRrWMndCqhVQ8VmRD/SjRL8AOJQVxNnmYPY2l3079MMP6KwArT+K0l6fF3g6cJCu045MWE8vyTFNkAzmiPOorbfgALQU5s7yzyG9GE7UVhkME/22VQtyHYdLxW7JErhKIX/F+mSVFA6TbtVm0MjiSDDX/XjEC+mfRY2A7ysHYEdpFda7A/0YLWjd+zNjRPigucEZFRQaUgtAqbZkSozYUDgtQ9rvN7Jw77btg3og/ZCnUdC8nWKP5uVqrYyHaZxcgHY2T8FKgjLoLyqb9zugcnNC5RszImtQxmkwtuse68ewk9gw6gDxGUxmJaezbUZqIZiNFnZCNn9eHGA5iihld7WAoQ+jpejPg1vBbo0I8UP354Gf/TYPTFK0KZ34q9VXpJon6XcjNRcHZLlA8z1lVvwbVVoCVGlfnGUyCI8EaG5AgjdFeghsm3r96d5VUlWD0EnObckt6uT1UpUdDL8NBVasmMeU9yj9ySk/stu73o5wQGx16Nt5ZUyrieoBN6XFVpiY1+olQH/29rsaqWCsFw/W2SZ71KuTcmR4Cz6jDN9DkOgQgjaaqbEeR6CRiVyCLw1LSPk2dAlpGNCo5SicDhi+9ZrLjw7xV+dCHrrtXMTnDk+UeadnKR9WJDAOdQ+HEUaTW6QIyz2LqIu4pKEeV8UgfI2dCxgfljm0PuON+Bc6mWqcOY81kzPMqUmHpU6BHwNb9xbsh+VBxECzCFSwSkxaPLuRYDwrnOgvLJqN8MdhKKaMUqcDea+QtQh7JuKKXEUv9Mt32Yh3QOuG0t1PdouKrY2S/ZHtQ+9YUxhckYnzOEOspbTfJcdnOmTNyGVRvvYIkIpO2cooCgHf60y6ZekXtTwF5QcSFYDAE7xXZ1qfur6q7GNPfOvJdb94hsjYVCnGmccREb4Kj7x7m2D/prmi5Tdu16loNDkmtvz47xYYVWY2gkUFP+zPexerRHhe0MeihB3Efo4ZRrJeaM6ym7acTsEFNw0ZFChNl6jqoRYwM42zVnbd3oCNvo3rEDF/HNkbW5PYLLmGuTGqq6xroUoZPQjMb2rUD7Au3bj9T6IKa X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4c1242ed-ee52-453c-2e27-08dbf680c6b4 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:35.4629 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: EWmEIwV31ETzr520bCF5O8hysEuyK1c9y3AU381hO9eie38jfOfxV8awq/a5RHpR X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092854_920291_E951D6E4 X-CRM114-Status: GOOD ( 16.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The next patch will need to store the same master twice (with different SSIDs), so allocate memory for each list element. Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 11 ++++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 39 ++++++++++++++++--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 +++- 3 files changed, 47 insertions(+), 10 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 29469073fc53fe..c541b94917d036 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -38,12 +38,13 @@ static DEFINE_MUTEX(sva_lock); static void arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) { - struct arm_smmu_master *master; + struct arm_smmu_master_domain *master_domain; struct arm_smmu_cd target_cd; unsigned long flags; spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master, &smmu_domain->devices, domain_head) { + list_for_each_entry(master_domain, &smmu_domain->devices, devices_elm) { + struct arm_smmu_master *master = master_domain->master; struct arm_smmu_cd *cdptr; /* S1 domains only support RID attachment right now */ @@ -289,7 +290,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); struct arm_smmu_domain *smmu_domain = smmu_mn->domain; - struct arm_smmu_master *master; + struct arm_smmu_master_domain *master_domain; unsigned long flags; mutex_lock(&sva_lock); @@ -303,7 +304,9 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) * but disable translation. */ spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master, &smmu_domain->devices, domain_head) { + list_for_each_entry(master_domain, &smmu_domain->devices, + devices_elm) { + struct arm_smmu_master *master = master_domain->master; struct arm_smmu_cd target; struct arm_smmu_cd *cdptr; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 38e83386f631f1..6a3a5df836b9c7 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1968,10 +1968,10 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master) int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, unsigned long iova, size_t size) { + struct arm_smmu_master_domain *master_domain; int i; unsigned long flags; struct arm_smmu_cmdq_ent cmd; - struct arm_smmu_master *master; struct arm_smmu_cmdq_batch cmds; if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_ATS)) @@ -1999,7 +1999,10 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, cmds.num = 0; spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master, &smmu_domain->devices, domain_head) { + list_for_each_entry(master_domain, &smmu_domain->devices, + devices_elm) { + struct arm_smmu_master *master = master_domain->master; + if (!master->ats_enabled) continue; @@ -2489,10 +2492,27 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master) pci_disable_pasid(pdev); } +static struct arm_smmu_master_domain * +arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain, + struct arm_smmu_master *master) +{ + struct arm_smmu_master_domain *master_domain; + + lockdep_assert_held(&smmu_domain->devices_lock); + + list_for_each_entry(master_domain, &smmu_domain->devices, + devices_elm) { + if (master_domain->master == master) + return master_domain; + } + return NULL; +} + static void arm_smmu_detach_dev(struct arm_smmu_master *master) { struct arm_smmu_domain *smmu_domain = to_smmu_domain_safe(iommu_get_domain_for_dev(master->dev)); + struct arm_smmu_master_domain *master_domain; unsigned long flags; if (!smmu_domain) @@ -2501,7 +2521,11 @@ static void arm_smmu_detach_dev(struct arm_smmu_master *master) arm_smmu_disable_ats(master, smmu_domain); spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_del_init(&master->domain_head); + master_domain = arm_smmu_find_master_domain(smmu_domain, master); + if (master_domain) { + list_del(&master_domain->devices_elm); + kfree(master_domain); + } spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); master->ats_enabled = false; @@ -2515,6 +2539,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); struct arm_smmu_device *smmu; struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + struct arm_smmu_master_domain *master_domain; struct arm_smmu_master *master; struct arm_smmu_cd *cdptr; @@ -2551,6 +2576,11 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) return -ENOMEM; } + master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL); + if (!master_domain) + return -ENOMEM; + master_domain->master = master; + /* * Prevent arm_smmu_share_asid() from trying to change the ASID * of either the old or new domain while we are working on it. @@ -2564,7 +2594,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) master->ats_enabled = arm_smmu_ats_supported(master); spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_add(&master->domain_head, &smmu_domain->devices); + list_add(&master_domain->devices_elm, &smmu_domain->devices); spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); switch (smmu_domain->stage) { @@ -2881,7 +2911,6 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) master->dev = dev; master->smmu = smmu; INIT_LIST_HEAD(&master->bonds); - INIT_LIST_HEAD(&master->domain_head); dev_iommu_priv_set(dev, master); ret = arm_smmu_insert_master(smmu, master); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index b7a65e37c51e9e..59bb420af52e28 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -695,7 +695,6 @@ struct arm_smmu_stream { struct arm_smmu_master { struct arm_smmu_device *smmu; struct device *dev; - struct list_head domain_head; struct arm_smmu_stream *streams; /* Locked by the iommu core using the group mutex */ struct arm_smmu_ctx_desc_cfg cd_table; @@ -729,12 +728,18 @@ struct arm_smmu_domain { struct iommu_domain domain; + /* List of struct arm_smmu_master_domain */ struct list_head devices; spinlock_t devices_lock; struct list_head mmu_notifiers; }; +struct arm_smmu_master_domain { + struct list_head devices_elm; + struct arm_smmu_master *master; +}; + static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) { return container_of(dom, struct arm_smmu_domain, domain); From patchwork Wed Dec 6 17:28:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482114 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8F745C10F04 for ; Wed, 6 Dec 2023 17:30:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UunW5Q+s9ujda16y96LE0i9jrllHlKLPbFgCLxF0jFg=; b=ur5yOtn1beojsO Uh3sbjBEoroGhTn2XTIH3baJslPlPjyn7+7G9Wk6hngAsbV9Pz/M+p+XkSMU8v2h0NjpeAzw+rBWN ZWDtbexP5WJhgp+625UpZ/Vnrecpw+1khOtsp5K9tOOOTHCkDZ3LAjYLNyqhDuwKqNHjBuWN+Q3tj Ow+IYwmy4OJAvyA9qfuCBW2IGRy9Y0AH8Kj6yDt8BlMOVLSK6ZUDkDiH8wXlKLmc7tPMGuZG+LQ2p HRHZ5iu+x9YQz4q2UUg8tjwZASbrvqADIOXxTNgwWxwZB5q9/qeuEqXZfvfnp4EzINM5GvctKoUxn qwlLu3PDLV5oadW0T7DA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAviZ-00AuI3-0Z; Wed, 06 Dec 2023 17:29:51 +0000 Received: from mail-dm6nam11on20615.outbound.protection.outlook.com ([2a01:111:f400:7eaa::615] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvho-00AtKY-0F for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:29:13 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=W4Z8TUtvP1gs1bQX5suMfzQVhWw8DmZ6SV7v5OkWqdxpansH/+ZRuEiimjoxIFiYNfNRaeBnTTUdgXvCglwMTeKy4nqKyDiMMxYwHglmh0VPJbnFe0dbMf6Cc37GsUvDa2cecxlONgBg634o2HWj1zu+w6QwY4mobUV9W3Fd+HsAN/KHz+HZ9/otwtN7zrZvyg41SZPt69EbmU+5JfTwo/KlAyzOsLUDVc1d+xkDpQcVCcdJTMYrx/EZB/KLKKNO0oDpT5ET01AiHcXwCDm6bltTQi/4s3VP70ZhOSsmWqyvUshHEW2i3MWnfilQfb2VkTwdzoGE0dE8q8cKadrCNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9GHsobYzYGanyus65XCeCWIwam5qEzamwpJPi4GaJ7s=; b=jplefmNOJG7sZ/cz49ERPal70tOCT6BOJzYV/SCZKPDhGf3NeYlvmR8914oh9yaahrtkr+z0VqtXJnyv/vbJHTsPmPySJFCO7Nnqyl7JYXU2TuUUacWDAoscO62QA+vqMf/7vr7/sbJikxa2UMSiA8RoLSTyWuMrv8FtOE9ktqT7W3Vbhy2TztxeZhMxdDdnVvgvOsvF2Fee84R+NHobUR8b9QPZFBF8j/PbAkjLuhfF7I03zPnFXR/mB+QCr0KSaJLA8u++P0d2//domiNlONtNHAcuSHsHtcEX2+f8A8RSW6dVXHN7+ZFe7BL2MqsNj2m0hPiq9zLR9aRHQIg3iw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9GHsobYzYGanyus65XCeCWIwam5qEzamwpJPi4GaJ7s=; b=oWQlN1w/biwQ2tGba9yhsQgAmMNxBOTwT86HspPB0qURhxchGiNNOEkGSreHHS4mzTGVdlHh9x21MIMcQgflmiXfA1Iql5MVnBjTycGiMWS/TTmSC20d+VK1LX4ZDXby07CI8ipYR34HDTwisIpzg/BgewsGI8QjDDJIGooI/4QcO565gkj9gC7aiWiTEY4T676944JPgc9IHgyyf/AGn3yF6TLCztTyzMFVsvy5fiRCZZt4TLZwPJoF6lAvWRcjidLu7wo3D1KR/ws7n4DPvmTGY8nhw0QLCNpBCkZUSQDOfFXZupwZYVi3bAd5ItTszB3jXARl5Bm73iaygY+W1g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:44 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:44 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS Date: Wed, 6 Dec 2023 13:28:20 -0400 Message-ID: <14-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR22CA0006.namprd22.prod.outlook.com (2603:10b6:208:238::11) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: 4d8e4d25-2d92-4e98-3cd0-08dbf680c724 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ZewSW9huSYh30xfiAXU1KxTcO/NDl/UnmKUPhh2WLQQgCZlD1ntJ42IjaCMTXcmKg0JlPzHNxOs1ukAWzlNjtnr8ZS0VJCvUGLW7e7A99T3VU10TXmU/42Ws6SslbQkF++m+j2uwnw3nFl5LTs9eAAQVDRCS7Mhquye/xZMYiA1ccnIz/ZuWVOS411emB3RDZRvuqOP6vApL+Q/R2t9Wmj3J8seGdIn+0fvc0drB8APJeJyvUxduWXpctV4UeZ6MdMRMQsvTeuQ4y1M2rozDA/x6VRsHe9d4Bl4p0AgRC6I+9EJFNfAU1RN1rJ4mim7Txd2vE2qLPQV9xnq7SUp5YdleE2Crj3xERrXVrfbmnFy3eyjh5GvO1Xe5oiPzDN8JO1I0/BcIojiXO9N72wnpOQ3gmcEP+ouEyF1Q2FgI+lpZkIhNSI86nFWOy+vkfhBb5Fb5880tYjAUUdVe7UjEgXaUUJcGNW2ccWNiVtrThSJnk+Huzh3oKkvoM+Cqb2bq6ZzFRZ8wNDUghZ3pJcYc3QLu3kSDd/4YKdmzkUf3hk3aKxNaCSgbHjIQBx0rzYMw X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(30864003)(4326008)(54906003)(7416002)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Rm7gLsr2zpohxbOzSqNCUOc7Jll67yZ2N0B3RVypkH70+3xO7hARXY5b1ozR+1CDPGYMUoFMv5ftDFafJBVQ95kl8Lkmf29RQ6qqo1VBSqKOrFci8EvJfA5uVmIQI3jbdNI76ZAbONrLZGSk4OPxqYIfJvysX/rucvua+pRPx2jfxwnUh2az03ELBfY0K5/IsyHtcHxYfnsh69XNG7snXEKJXuD7SIcCAWr/lyqbmITjmltwtp5eL0xMIruyvx5+Nt55AzOkr08D8/GnVvEPEhuVComMOXYO5+bmZiM8N/t74ynA+UIFZR3e0yQCeRCnr45WYsEYTytNxkcUZ5vVVqeDlWR99XNGJfvuDCpPnU0ZqW+veNaZMSt5nu/hGAXUUVZ8BQQinXh1DKdqm7pO8xR8zebcxieebeeB5I0MVe8ac1FE4p7PMnfjoTIUKX1zQQmhxfenSjm40Uy3tXj89ADqALcER2tHEZLVUNyRVZtg9yGkpuC9em8LtBPvyfrlRsmpmuRErF+6GQpHCC98o8g8kRJuKdRU2zDvc2KH3Elp+SC+fQ3V1z+bKVXhbbQ22DoAAHsOc36FCJAYZYbFZwi//TAlrh9Ed3oOzGcFQ0SwCFTvm7jV8weOBtFKeBVAziv5hm34O8R8H7Yk5JUA5yIgoN40Kj4RbH+WknkzCn/EiutIjnXTiU1Kj83DhlwjTgMLd+ctZc+4QfHubGkS/82kV8VQBKzNzNPHaHU3xl2loknGy2DFRpGmVL0DLCsYUA7X9P5Fo5KI659AaUWvWAVVWvsOWQ8nW/FvbGCjqK84c5wXUCgQ1kcyvlWVgRhDm/n/BiKxB4spaLXL3pK3xRXxVIZVT+AGzcJ1YLLjtiwuOag4oldztTuyLCutfjDQA7T0Y+G1fAkD2A4qXcxoyK46VZUr0WBlpgcZQbvbYlTRNV7THUXb9vo64i9XlQZGyrwNPPswaExPi3x3lioMAGlmPpVcAiWKijsbf6v+VhqjrikazDuf5Au95qo4riPsNjdkUF1YhciYB0c+u4VLBZ0mnJx4yCKGC9qRACJTVkKNNCk4EdPW4bRaLjaSz04ZCg59SfMzsqhsLSldbZwzt+AnqF2bxOG+PJC2Xj3kIRIlBiBdVWmEQFbWx0f/Q04qrjeTnEGlq2f4Ip6Tuvfb7W/BRGyfqqke4on9p9zouP0yG5/821jXpAkiONbTp8dfFJNSKldo3ZiBO0ZG4FoYjXtSDC5VmFrB4cEIwT+kc7ndH+EkCSJGQ6QXRpFkCjlMTRrbCxXpqAVHjMNqmyFC8jInt2XBW+VKlEP4AR7VyKo1Mg9zauzEqflYLstVFuiDjUOSHS39LjDIkvTkWZj2vVrHbG0HvPd7QqvVRkGdM2e/KuhyQ2qb7Eq6c1Zq0bw3fqhnV6JQgSi9bzaGIab6CPkVWYi/MqSnbz09zmUk56Ns6qk9Zm+lAWd1sHODDj+L3wmqoBuQgKOD/1Cb4SoxcpE/d8OoH8OOE9XpacJUlHx5X8xaXzy5p1h+jNkjmuVomxG15So8/KzuQ2BxXWAmJbH4iip9BzOaO0emS/tDGuAtB5VcwVK5BIu+xZr2fhJm X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4d8e4d25-2d92-4e98-3cd0-08dbf680c724 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:36.2214 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6/oYkqy0XzxGwGXnGcBwIF20ondRnPUTk/46NXVp4P7xZdLrFJhQ3e5D7gEjW67/ X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092904_156947_F77C89F0 X-CRM114-Status: GOOD ( 30.86 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The core code allows the domain to be changed on the fly without a forced stop in BLOCKED/IDENTITY. In this flow the driver should just continually maintain the ATS with no change while the STE is updated. ATS relies on a linked list smmu_domain->devices to keep track of which masters have the domain programmed, but this list is also used by arm_smmu_share_asid(), unrelated to ats. Create two new functions to encapsulate this combined logic: arm_smmu_attach_prepare() arm_smmu_attach_commit() Put the ATS disable flow into arm_smmu_attach_dev_ste() - we only disable ATS when going to IDENTITY or BLOCKED domains. Installing a S1/S2 domain always enables the ATS if the PCIe device supports it. The enable flow is now ordered differently to allow it to be hitless: 1) Add the master to the new smmu_domain->devices list 2) Program the STE 3) Enable ATS at PCIe 4) Remove the master from the old smmu_domain This flow ensures that invalidations to either domain will generate an ATC invalidation to the device while the STE is being switched. Thus we don't need to turn off the ATS anymore for correctness. The disable flow is the reverse: 1) Disable ATS at PCIe 2) Program the STE 3) Invalidate the ATC 4) Remove the master from the old smmu_domain Move the nr_ats_masters adjustments to be close to the list manipulations. It is a count of the number of ATS enabled masters currently in the list. This is stricly before and after the STE/CD are revised, and done under a spin_lock which more clearly pairs with the smp_mb() on the read side. Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 171 ++++++++++++++------ 1 file changed, 120 insertions(+), 51 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 6a3a5df836b9c7..b50d5cfbef8c71 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1490,7 +1490,8 @@ static void arm_smmu_make_bypass_ste(struct arm_smmu_ste *target) static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, struct arm_smmu_master *master, - struct arm_smmu_ctx_desc_cfg *cd_table) + struct arm_smmu_ctx_desc_cfg *cd_table, + bool ats_enabled) { struct arm_smmu_device *smmu = master->smmu; @@ -1512,7 +1513,7 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, STRTAB_STE_1_S1STALLD : 0) | FIELD_PREP(STRTAB_STE_1_EATS, - master->ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0) | + ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0) | FIELD_PREP(STRTAB_STE_1_STRW, (smmu->features & ARM_SMMU_FEAT_E2H) ? STRTAB_STE_1_STRW_EL2 : @@ -1521,7 +1522,8 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, static void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target, struct arm_smmu_master *master, - struct arm_smmu_domain *smmu_domain) + struct arm_smmu_domain *smmu_domain, + bool ats_enabled) { struct arm_smmu_s2_cfg *s2_cfg = &smmu_domain->s2_cfg; const struct io_pgtable_cfg *pgtbl_cfg = @@ -1537,7 +1539,7 @@ static void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target, target->data[1] |= cpu_to_le64( FIELD_PREP(STRTAB_STE_1_EATS, - master->ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0)); + ats_enabled ? STRTAB_STE_1_EATS_TRANS : 0)); vtcr_val = FIELD_PREP(STRTAB_STE_2_VTCR_S2T0SZ, vtcr->tsz) | FIELD_PREP(STRTAB_STE_2_VTCR_S2SL0, vtcr->sl) | @@ -2405,22 +2407,16 @@ static bool arm_smmu_ats_supported(struct arm_smmu_master *master) return dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)); } -static void arm_smmu_enable_ats(struct arm_smmu_master *master, - struct arm_smmu_domain *smmu_domain) +static void arm_smmu_enable_ats(struct arm_smmu_master *master) { size_t stu; struct pci_dev *pdev; struct arm_smmu_device *smmu = master->smmu; - /* Don't enable ATS at the endpoint if it's not enabled in the STE */ - if (!master->ats_enabled) - return; - /* Smallest Translation Unit: log2 of the smallest supported granule */ stu = __ffs(smmu->pgsize_bitmap); pdev = to_pci_dev(master->dev); - atomic_inc(&smmu_domain->nr_ats_masters); /* * ATC invalidation of PASID 0 causes the entire ATC to be flushed. */ @@ -2429,22 +2425,6 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master, dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu); } -static void arm_smmu_disable_ats(struct arm_smmu_master *master, - struct arm_smmu_domain *smmu_domain) -{ - if (!master->ats_enabled) - return; - - pci_disable_ats(to_pci_dev(master->dev)); - /* - * Ensure ATS is disabled at the endpoint before we issue the - * ATC invalidation via the SMMU. - */ - wmb(); - arm_smmu_atc_inv_master(master); - atomic_dec(&smmu_domain->nr_ats_masters); -} - static int arm_smmu_enable_pasid(struct arm_smmu_master *master) { int ret; @@ -2508,40 +2488,116 @@ arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain, return NULL; } -static void arm_smmu_detach_dev(struct arm_smmu_master *master) +static void arm_smmu_remove_master_domain(struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain) { - struct arm_smmu_domain *smmu_domain = - to_smmu_domain_safe(iommu_get_domain_for_dev(master->dev)); struct arm_smmu_master_domain *master_domain; unsigned long flags; + /* NULL means the old domain is IDENTITY/BLOCKED which we don't track */ if (!smmu_domain) return; - arm_smmu_disable_ats(master, smmu_domain); - spin_lock_irqsave(&smmu_domain->devices_lock, flags); master_domain = arm_smmu_find_master_domain(smmu_domain, master); if (master_domain) { list_del(&master_domain->devices_elm); kfree(master_domain); + if (master->ats_enabled) + atomic_dec(&smmu_domain->nr_ats_masters); } spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); +} - master->ats_enabled = false; +struct attach_state { + bool want_ats; +}; + +/* + * Prepare to attach a domain to a master. This always goes in the direction of + * enabling the ATS. + */ +static int arm_smmu_attach_prepare(struct arm_smmu_master *master, + struct arm_smmu_domain *smmu_domain, + struct attach_state *state) +{ + struct arm_smmu_master_domain *master_domain; + unsigned long flags; + + /* + * arm_smmu_share_asid() must not see two domains pointing to the same + * arm_smmu_master_domain contents otherwise it could randomly write one + * or the other to the CD. + */ + lockdep_assert_held(&arm_smmu_asid_lock); + + master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL); + if (!master_domain) + return -ENOMEM; + master_domain->master = master; + + state->want_ats = arm_smmu_ats_supported(master); + + /* + * During prepare we want the current smmu_domain and new smmu_domain to + * be in the devices list before we change any HW. This ensures that + * both domains will send ATS invalidations to the master until we are + * done. + * + * It is tempting to make this list only track masters that are using + * ATS, but arm_smmu_share_asid() also uses this to change the ASID of a + * domain, unrelated to ATS. + * + * Notice if we are re-attaching the same domain then the list will have + * two identical entries and commit will remove only one of them. + */ + spin_lock_irqsave(&smmu_domain->devices_lock, flags); + if (state->want_ats) + atomic_inc(&smmu_domain->nr_ats_masters); + list_add(&master_domain->devices_elm, &smmu_domain->devices); + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + return 0; +} + +/* + * Commit is done after the STE/CD are configured to respond to ATS requests. It + * enables and synchronizes the PCI device's ATC and finishes manipulating the + * smmu_domain->devices list. + */ +static void arm_smmu_attach_commit(struct arm_smmu_master *master, + struct attach_state *state) +{ + lockdep_assert_held(&arm_smmu_asid_lock); + + if (!state->want_ats) { + WARN_ON(master->ats_enabled); + } else if (!master->ats_enabled) { + master->ats_enabled = true; + arm_smmu_enable_ats(master); + } else { + /* + * The translation has changed, flush the ATC. At this point the + * SMMU is translating for the new domain and both the old&new + * domain will issue invalidations. + */ + arm_smmu_atc_inv_master(master); + } + + arm_smmu_remove_master_domain( + master, + to_smmu_domain_safe(iommu_get_domain_for_dev(master->dev))); } static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) { int ret = 0; - unsigned long flags; struct arm_smmu_ste target; struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); struct arm_smmu_device *smmu; struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); - struct arm_smmu_master_domain *master_domain; struct arm_smmu_master *master; struct arm_smmu_cd *cdptr; + struct attach_state state; if (!fwspec) return -ENOENT; @@ -2576,11 +2632,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) return -ENOMEM; } - master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL); - if (!master_domain) - return -ENOMEM; - master_domain->master = master; - /* * Prevent arm_smmu_share_asid() from trying to change the ASID * of either the old or new domain while we are working on it. @@ -2589,13 +2640,11 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) */ mutex_lock(&arm_smmu_asid_lock); - arm_smmu_detach_dev(master); - - master->ats_enabled = arm_smmu_ats_supported(master); - - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_add(&master_domain->devices_elm, &smmu_domain->devices); - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + ret = arm_smmu_attach_prepare(master, smmu_domain, &state); + if (ret) { + mutex_unlock(&arm_smmu_asid_lock); + return ret; + } switch (smmu_domain->stage) { case ARM_SMMU_DOMAIN_S1: { @@ -2604,18 +2653,20 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr, &target_cd); - arm_smmu_make_cdtable_ste(&target, master, &master->cd_table); + arm_smmu_make_cdtable_ste(&target, master, &master->cd_table, + state.want_ats); arm_smmu_install_ste_for_dev(master, &target); break; } case ARM_SMMU_DOMAIN_S2: - arm_smmu_make_s2_domain_ste(&target, master, smmu_domain); + arm_smmu_make_s2_domain_ste(&target, master, smmu_domain, + state.want_ats); arm_smmu_install_ste_for_dev(master, &target); arm_smmu_clear_cd(master, IOMMU_NO_PASID); break; } - arm_smmu_enable_ats(master, smmu_domain); + arm_smmu_attach_commit(master, &state); mutex_unlock(&arm_smmu_asid_lock); return 0; } @@ -2648,6 +2699,8 @@ static int arm_smmu_attach_dev_ste(struct device *dev, struct arm_smmu_ste *ste) { struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct arm_smmu_domain *old_domain = + to_smmu_domain_safe(iommu_get_domain_for_dev(master->dev)); if (arm_smmu_master_sva_enabled(master)) return -EBUSY; @@ -2665,9 +2718,25 @@ static int arm_smmu_attach_dev_ste(struct device *dev, * the stream (STE.EATS == 0b00), causing F_BAD_ATS_TREQ and * F_TRANSL_FORBIDDEN events (IHI0070Ea 5.2 Stream Table Entry). */ - arm_smmu_detach_dev(master); + if (master->ats_enabled) { + pci_disable_ats(to_pci_dev(master->dev)); + /* + * Ensure ATS is disabled at the endpoint before we issue the + * ATC invalidation via the SMMU. + */ + wmb(); + } arm_smmu_install_ste_for_dev(master, ste); + + if (old_domain) { + if (master->ats_enabled) + arm_smmu_atc_inv_master(master); + arm_smmu_remove_master_domain(master, old_domain); + } + + master->ats_enabled = false; + mutex_unlock(&arm_smmu_asid_lock); /* From patchwork Wed Dec 6 17:28:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482096 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F21B9C4167B for ; Wed, 6 Dec 2023 17:29:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=C8NxlXtcEqwUEq8G50vNTq3Q4ZCn3y9RNTG841ZsG4c=; b=fqGVxvra1mY4Ih n2Bber26qVYKlCnFQWwh88/gyWW/t8u8k3KbmMsvgZfM8+JyG34+7jyGQppYhbNev9n4eWIcj66kU nBFTEUzMf/YnrjzWR62CPwnT+lIbNN7wJffRqlglh6U1Inwq/mEkI2hf3TmpTro0hZ+3g1KD4/c3W 0mUW/+Ozv5IzwOil/6cq1gnmjpYydp9U1eSA/EZ3IHTA6Ra3Uv107BuZFtuvNmaw1Wj1LNrzgVIKp cotj04pK2xZ7L9/Ukd2oDnpEFQGVnf7wQfUXFVT7nytj6bdO9fyIIhxpeW3JIa0Dmg5jhhJP+MrRv 83y7YSzqH91o9jZwBgwA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhb-00AtNz-1A; Wed, 06 Dec 2023 17:28:51 +0000 Received: from mail-mw2nam12on20609.outbound.protection.outlook.com ([2a01:111:f400:fe5a::609] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhV-00AtIF-2x for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:47 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Zfa4Cnaicxhl1IfHdJP9wLRKbvHj4fqqhtHq34QcIa5RNXTcsI7ObK9BOcdPa9yEpOSdWNi3cZKmBxe/2v1y8U8Siv+ffHzyGrN90lRpEVseBTmhppmcGqICjatzvy+1GsrYYPVdibhDgwyYTk0wH0AfsCx2/kGgJpX3DPjB1BEMTgduLVP1Yc45loScLHRenZ99lSdRY72paWmRaD44UubM7WMRGuC4YBj0HiKsWMI8wmKLE8PJNdg4FDw18QxRE2bRfAu/ascu5JjEcOxlRoLfKKNpLX12RfjCRTIdUC04SGL+LV5tEz8PpRf8psQk4xiXWJLXB4YL5/dbO7Tdmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3szW7cIHmYMg7ErJje6VuauWlAflJUMe2HWJ/uj75ho=; b=m1ncxH3xKI0aFJhDJ5DWw8/Or6cyRD6TVCn/p8ztI6GEuYJGgTq+Wi9IVi/6L2eWtwFDfYOeyAJackAFgJgOf7nDpRPkmGERQyDg46KF7hk5WPbbzlSsoU/It6dLqLDRtBuiZHD2XXG6OjF59RcPlB2jXiDP51lN1hVJxYv6Q1rZ7Kz88o6rBgG8G1v4Y191OywpeFFuU155BBm3uOhPuWtePN/OuTCuXUt+cXyomeKDZphV81+MLvaV4PoUUfI8XPOPNd+bHwme+F25apD52OHNAcCRhdYfS6tf+WkoFfcUnU5uMadeSBMIYJLE8E7Zaj8jal7MK2Jm+vDir53zFA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3szW7cIHmYMg7ErJje6VuauWlAflJUMe2HWJ/uj75ho=; b=qzX1GE3fJRzXoJXK2/0tblnj6NEcbzfMaejxiRPDLSVzCP54uDVXOdUVRV1olX7AkzfpqkY3b3M02bJ9cB6nGTExG7BSlCbbWNUzurh2t3tdv+bvarTZCjxIht/OAPFAm/pkhB2ju5Y5SmpUH9/qFXEyo+6mO74jdYidhj1o41dYFC7mIiCHfP30oTUyHByLbFC2kx1hikDr1VxrvC/j9KzDftbuO0DFNnEODhJMrAS9rkjDTZzbq6oyw1bs8oVN2wCQlQ873HtewYubZAQz5sGKIzblrZtLp8/MSzrg4TyV7iE1YZOfzljsqYXW87k7yTvVfMcGaHs29SOSTrYkhQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:39 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:39 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 15/27] iommu/arm-smmu-v3: Add ssid to struct arm_smmu_master_domain Date: Wed, 6 Dec 2023 13:28:21 -0400 Message-ID: <15-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1P223CA0002.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:2c4::7) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: 1a868ba4-6a70-4a5c-4806-08dbf680c62c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Tzt/hnsxo64LxYmTYhb6UrWRV6ZdrJIidvqAu8Fd3oOAUQtuDAB4zm0ndhVyLb0MAYKE8sk2trQIYYeL/Mi+8ozhpG/T6soFosK6EFaIFb2bUi2xkAjK1fXTYFr8TIb/ijBuxM/q8VPjT2EQsxpYvzFyYKFVjXRVacdyNZ3FrC8rOlT2lfDBTzaARhxQJAGbhmlVfaRF7cf3CQmgG0RaLOubBJtwlCJQFn6Al4/t6MHDpL818MzGuSWl43mXo6thO4KwiIPNSlve3dnC/mFqQr70iERppEU5ot2CyZQBwQuz277g8F2YK5cf7Wd+Q4vittoZUlTyRuTInlpcJiNFozncZAyXd5yb7LyiIg8VSyjirLVoJvQtvN337zvNTKF5WiE3YwqOcyrnR81U4Cu/d2TN4HHe8PN0sn3CQuIuI7dtAriThAm+Aaoh0LSU2dI5tZZxggWwv7zPd/TTcfrD/tysP3fSGC+KUA1VESzCmePDX4c65RTNGWcx1IyUJ5O+jE5TdWKM6iNVSMoi+70WjrOmYrAb/BVC+ldccacDeM9A8oCFI9a2PQCFe+vDNJoI X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(4326008)(54906003)(7416002)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 9yY7cbdfm8ICLzVmrDPh/8uO4weZBOVuLWrwdpQXMrkWMabLZ34QgyylxyhoD5s/IYqBOvxnvq2llTUoUPODpm3LmbhkEMztlWDyEX2n0bR1X25Fyqa34neUCpFfJ9P5rE63pRUeulvf56yjMrL5Btyb/jZUnL7RPiYoM0tw2tfwbvI3Pf0GTkilzGtnzEd0gMd2ad/21grwUaICAdJ/loIT0dyX5i4jG7GUTtIMeBs8bBQCt448/ZsQsLGpQ2XNj+uvrMP+Zp7D/ZS3kwMqkJysTlBXoeox5P5v1x/OPMMbd46N0QJPDAhgcJNwSAj5bOXN4w1fV6ZBF89qm704qOPDGLZU1sHYBNaQS3VFmvI1bnE2WAjQAFrrilmlqa5DSo0w9eM69hZ4CTF+ZSDhQkJdq32EJJLXRqfJk1TIHX/Hqz3cilNDKBfiFhr1gPhpynXJjc2Y+ljzX9y4TCtBTGAzIt38CtSJDF9J2d4XF0KM8PahB8SpT+JnFS5Tiq/FA4xUZvbTrL6s0qfl8syvrKWOJFl0Z0TDFeJtFpvqd00YZ/x6GFlL69MLHTI9G7Wg/5Kq5h80EceD+oqpG/RmCb0aCxbgsus0XhmDW4AuK9t8m2i5qMdEE5hbwdGdRnEIUDlWjgQYY6iv6f6NPj6IZZCyfe3s/lsLDXQOP4bKIsbXmyjxqqP1mWEVstYCM82SFAa9+5g8dFvIAUOr3zJBequ36/Kd9LX4Y7gbVIhxKbp47ZpvppDj8osigXcsIMTOYryDE/KIqG1PW1WQVtx1r6Br7b92QQRm6VkzCYhqeuEUK9hcA0ruYfYrHG9PjAcDqJOjrgtcGQBmGDGRJOZar49hK/iE57cLv1jnG2OGGWK+18bdVb/Qrn3juwbC1XkFJs7hJRWbuMtno5QxiJnL04v8SVJO28C02XZx+kKgX2qTXbKAX/SIHlfK3TM5ihP4KBFugkMLyrFMITZGPu5K7tc0yg48fep6ljg+Eji3MMZC8H7uYzb/CEZGyPDBNx7WT8FidD1cveyrIQoza1SCMJaxSHFX+DT9YwRlZAlBcxTC8uouWIUbCsSuxtrc3BiXScIitgNcnt/wijJtMibba7iDXepC05RG1UKJCM7kI3QuzfiZN9v7KiPU+WJDFz2iPPkqgpnC7ZpKc+GFAoyUdUct7KpI3shtEDkiA/6eswJAk6bHNVXqLOJF1Gwg3TBGtKlZBHWL0n30XEO4GEvhcsJdUrb0G7kiX82AHSTwcxNnQhGBNI3rle3UQFudvcVeHYXUrEdCkTcr7gdI/w1Z9kCLZtizULetCW6dFq+tj1V+Nk86szZiFCNR56SQEK7jsiYZMy06OneT6ik/Tzqffc8lVgUUhhwTLRu2UWitDR+buOS91MjjSNsYySOBl4CXUDHljpfG7AiavPoocoXI5h5ARUoiITn314YzTc8yrr6ZLD9jiXTa8xbuIOEmuEAYKUEwD57Hz197MUvgdeo8RX5z9gxhPG2ry5TgRjvZwpq71cGQbU7BdDMCoVIscynzT3+c2SKOzB0GyfBm1l3zbdL6F1mGJIJqd1jls4ks78eBhclzhEGpPct7NPEpqUN8 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1a868ba4-6a70-4a5c-4806-08dbf680c62c X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:34.5861 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: riR23mn0eryMqWTC/vVhjWnmKfrd08s714VI06TepkI4tln+5vhQ+AlbDwONe3DY X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092846_037729_03051AB2 X-CRM114-Status: GOOD ( 16.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Prepare to allow a S1 domain to be attached to a PASID as well. Keep track of the SSID the domain is using on each master in the arm_smmu_master_domain. Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 11 +++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 42 +++++++++++++++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 ++- 3 files changed, 41 insertions(+), 17 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index c541b94917d036..4644d4601c0830 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -47,13 +47,12 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) struct arm_smmu_master *master = master_domain->master; struct arm_smmu_cd *cdptr; - /* S1 domains only support RID attachment right now */ - cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID); + cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid); if (WARN_ON(!cdptr)) continue; arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); - arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr, + arm_smmu_write_cd_entry(master, master_domain->ssid, cdptr, &target_cd); } spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); @@ -283,7 +282,7 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, smmu_domain); } - arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, start, size); + arm_smmu_atc_inv_domain_sva(smmu_domain, mm->pasid, start, size); } static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) @@ -319,7 +318,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); - arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, 0, 0); + arm_smmu_atc_inv_domain_sva(smmu_domain, mm->pasid, 0, 0); smmu_mn->cleared = true; mutex_unlock(&sva_lock); @@ -398,7 +397,7 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) */ if (!smmu_mn->cleared) { arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid); - arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, 0, 0); + arm_smmu_atc_inv_domain_sva(smmu_domain, mm->pasid, 0, 0); } /* Frees smmu_mn */ diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index b50d5cfbef8c71..a2263d9c3658a3 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1967,8 +1967,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master) return arm_smmu_cmdq_batch_submit(master->smmu, &cmds); } -int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, - unsigned long iova, size_t size) +static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, + ioasid_t ssid, unsigned long iova, size_t size) { struct arm_smmu_master_domain *master_domain; int i; @@ -1996,8 +1996,6 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, if (!atomic_read(&smmu_domain->nr_ats_masters)) return 0; - arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd); - cmds.num = 0; spin_lock_irqsave(&smmu_domain->devices_lock, flags); @@ -2008,6 +2006,16 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, if (!master->ats_enabled) continue; + /* + * Non-zero ssid means SVA is co-opting the S1 domain to issue + * invalidations for SVA PASIDs. + */ + if (ssid != IOMMU_NO_PASID) + arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd); + else + arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size, + &cmd); + for (i = 0; i < master->num_streams; i++) { cmd.atc.sid = master->streams[i].id; arm_smmu_cmdq_batch_add(smmu_domain->smmu, &cmds, &cmd); @@ -2018,6 +2026,19 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, return arm_smmu_cmdq_batch_submit(smmu_domain->smmu, &cmds); } +static int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, + unsigned long iova, size_t size) +{ + return __arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova, + size); +} + +int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain, + ioasid_t ssid, unsigned long iova, size_t size) +{ + return __arm_smmu_atc_inv_domain(smmu_domain, ssid, iova, size); +} + /* IO_PGTABLE API */ static void arm_smmu_tlb_inv_context(void *cookie) { @@ -2039,7 +2060,7 @@ static void arm_smmu_tlb_inv_context(void *cookie) cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd); } - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, 0, 0); + arm_smmu_atc_inv_domain(smmu_domain, 0, 0); } static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd, @@ -2137,7 +2158,7 @@ static void arm_smmu_tlb_inv_range_domain(unsigned long iova, size_t size, * Unfortunately, this can't be leaf-only since we may have * zapped an entire table. */ - arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova, size); + arm_smmu_atc_inv_domain(smmu_domain, iova, size); } void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, @@ -2474,7 +2495,8 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master) static struct arm_smmu_master_domain * arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain, - struct arm_smmu_master *master) + struct arm_smmu_master *master, + ioasid_t ssid) { struct arm_smmu_master_domain *master_domain; @@ -2482,7 +2504,8 @@ arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain, list_for_each_entry(master_domain, &smmu_domain->devices, devices_elm) { - if (master_domain->master == master) + if (master_domain->master == master && + master_domain->ssid == ssid) return master_domain; } return NULL; @@ -2499,7 +2522,8 @@ static void arm_smmu_remove_master_domain(struct arm_smmu_master *master, return; spin_lock_irqsave(&smmu_domain->devices_lock, flags); - master_domain = arm_smmu_find_master_domain(smmu_domain, master); + master_domain = arm_smmu_find_master_domain(smmu_domain, master, + IOMMU_NO_PASID); if (master_domain) { list_del(&master_domain->devices_elm); kfree(master_domain); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 59bb420af52e28..887669f04b55c1 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -738,6 +738,7 @@ struct arm_smmu_domain { struct arm_smmu_master_domain { struct list_head devices_elm; struct arm_smmu_master *master; + u16 ssid; }; static inline struct arm_smmu_domain *to_smmu_domain(struct iommu_domain *dom) @@ -783,8 +784,8 @@ void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, size_t granule, bool leaf, struct arm_smmu_domain *smmu_domain); bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd); -int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, - unsigned long iova, size_t size); +int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain, + ioasid_t ssid, unsigned long iova, size_t size); #ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported(struct arm_smmu_device *smmu); From patchwork Wed Dec 6 17:28:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482102 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4FB04C4167B for ; Wed, 6 Dec 2023 17:29:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Y2qUsz9OnwqjtAjrRlUDDwD6/r6CkpERugUtaFT4wwM=; b=SUfWob4OlHDw5U VdhTzoW1hDp1Fy/gNAB5QxCdlpez1p7NcCuImOXRJj8ILyiomSbSKAhEpMpsOsjxL1g9kl32TuWy5 hat67oGRM5L8AoaXkXCdyDvHRPBgajvOkEiiv5S4wZB9ZtRK+sLBj2K9phWX+H9+btSDUbHdmxWg3 43BffnMMGWc6EFE6ZAhKGb1jSPkwVk8MOp7IHPlUkONDsSqv83skFAyxOH4WpnMEFIW5OisNriKG8 O1t9/l2yejnYc1uUO1F2B/kmVCDjPdmE7NsHniiiE25LyQ3rt3xL1m/1CNzuPegH9MsoYMb9JX6wE rqnD0t0UZQWH0AZ5vB4w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhn-00Atd6-2Z; Wed, 06 Dec 2023 17:29:03 +0000 Received: from mail-dm6nam11on20609.outbound.protection.outlook.com ([2a01:111:f400:7eaa::609] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhZ-00AtIR-1J for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:51 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kPbgvcgRNRx3R9ruVurVtipZ/9EBvysdmwyhA5QhPBjQA5cDHlVeDSZ9XKHxiXUS34tqKXdSOSC61Z/9jNcw8/6AFwYgl/8FtrK7RF1MeHzlnVutVWybNT2hWxHTnAFjubCituLI64XW5jQAtBc/c8c9vtGkT3VB26M4pA9JGnDYI4PCXVGyE31uQYK1UiCN2FPZCrpp7mtx2SsI6bmieCwhd3skiBy/qdQdX39MvgSxT6FIEaQ7kRUUirPPKXlzZ1NP8grwPymsYp6151C97RBVI17Pz7pS3Zt/h5LKvlX0dP2LhsT9bpUQYMyT8ZEIreidxZ1Gn67mwStd7GiT4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vGUYkm9BPytjmNaW08camwInHZW5EVUnExPqcs81rk0=; b=OKmVRT3QoE5Y4GlSc4MoPCypHe9J0cunFR31rzem9KttmTJwD2ZGI6jROzKW613SUzFK+YgvjQCuWPT4fEgL+8266KMPIljqnIHHFi3BXcNoWUd5i0r/5kQqSzGuY/JOMpik2WY3OjdG5rklTw0BbAmnpUIMSDDrVi63HP1rO08jNF6zzT6xHkI6hW7tH0OB9knf5UGnj16h8egsnUAhqlSpR44XTiI36uuxnh3V4M4eL1lNZR37rPxjD3ZnU6Csf6owaXJU0jFnZkNRi+w6FH/xjvvBPD1s/9lh13VMGM2c6a7w7g+BXSCRMz8SlXayli0wDs9AO53QZMErFzXnLA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vGUYkm9BPytjmNaW08camwInHZW5EVUnExPqcs81rk0=; b=EsQ9ZyUaxKrBAQE+iMZIIdQiXb5ugEhVXthESR6tRTtf0GS8aCi5wn9C+T7Lf8uLMAjBhSO3nqKusR7vRgTxo/mv+67MPatVfanPwZXTBth+ASmvr4KtDCuVKdCjnLWyh1zGpbmqElc3mosaFZfF3/DXRT3HTDLC5EJqb1fxGttbO50mnXo41CRUQn5pvz2/i+MEUHTmT8QYrWpkmznlLQXagI91HXBmhMDtNKAhSu0/KEftrVST2Uoh6ftkp9ViMG4NMIyaKe6mwjOcYsX01izsOUXBtS8ysbn7cw1xaX5uev6OF5Z2cr8LrAHcxlLeST7PUKYzn09J51Mt7HP6KQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by CY8PR12MB7364.namprd12.prod.outlook.com (2603:10b6:930:50::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.25; Wed, 6 Dec 2023 17:28:38 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:38 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 16/27] iommu/arm-smmu-v3: Keep track of valid CD entries in the cd_table Date: Wed, 6 Dec 2023 13:28:22 -0400 Message-ID: <16-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR22CA0022.namprd22.prod.outlook.com (2603:10b6:208:238::27) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|CY8PR12MB7364:EE_ X-MS-Office365-Filtering-Correlation-Id: f0488ae6-3c3b-496c-ab85-08dbf680c60f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: t7u8XT+o44nNZ8rC4S3jzQD3ElhBLs5GezdLNkcUNJnZ/VqdPZyvSJB5SiG81y6mTlxmhRFNNqSvwjlHP1VwdIJFuiPnSAwKA83+RHwqBXJ6mZcjhcCkM0LlmF7MGbaSL1Xs4aMicA2tx4yE5FHUTS/DlS3QLuo9u4VfqA9Xn/zwWUFr5pP4fTC2QSKSCcGSm8EDBMx/NnXlVJDjeMf4PwzdGZOM8YiQN1totpmvPBO0B+zI/H4mYGjyvZ0TtRgKkKyYmyjxZ22b/bbXSGufwTliEPRI3/YOmEZviDzUDw+Udkrkm7111zYVddKmH/OLQNondY91Qz+ub13MDDCqI7Yicv8VLdWvJD4h7AJ9O5XfF8pJqgYwWjZAhB/MVATjd1sYm69JV8fRZmjM95TIfLAz9rxGAveGo3L8JBN20c4R/AnKlMBYkUKYmCYUFdW7LIk9nT6RWUQMLIR2n7uddHa40Jctpq/Cc0zqIZZLZYFVWqd3YwjtMX7fkNmvznylDKNxy2IORMMteBox7SmIyQIZTlgu2UPBYBEVlJE3bNzBXRBsI4ny9THjN5bV/J2IvCST+0r5QqYgRMgaHag2CbOEAWtZ/O4SG+9AYTGbyVp9SiOle32ETrit6RowyfiRVbngbVQICX5O4vGU8M4gjA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(346002)(396003)(39860400002)(376002)(136003)(230922051799003)(64100799003)(1800799012)(451199024)(186009)(8936002)(8676002)(4326008)(2906002)(38100700002)(83380400001)(7416002)(41300700001)(36756003)(86362001)(5660300002)(6666004)(26005)(6512007)(66556008)(6486002)(478600001)(2616005)(66946007)(110136005)(54906003)(66476007)(6506007)(316002)(473944003)(414714003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: jsJ6TMSjgzZZ+A971QBOcOBM/8eLei7UrH59A33R9zxjy+9PEaqYkRbQfdGXVako9Y8sRRX2ncoBtKXfPJTuNaCichHbEL5wpTTEwjX0Ab6vKZ9nE44FT8u/qkr5d5nzWdrGQo650nDZsM8PW79HYjGwW6nDzk7srdzmPVr2CtDDYeNLOr5DUfvA4WYdhEEJHriMv985dxhZQfwdK/2DyMpQEAsaPgCgjgR/E8Xxrd1UrYjtKbO1nbukBRZULRf0g1VeQMN2bMIVCmtJSsuvlbgFqbFBh041Vd+ALBHTm5HyOXgoJW01UdTeuCJd4AzdJ9GSh5oZC76dbfNY/pCZSBAPF6tJoHJo6Yki244DsSV0xiyuHF6NqQP6bbuqqUJb4Z39aFvl29Ri/FiHFEU5SqI2BmMUQ1vuCpfFWvdssxewn3Lk9rQaUslJWUFuOhWgis6dHXEr/8vpzj7QTBwwH901yFfVGyupj0ly0xlhImva2mdreupaQ9QaplimNRy4boblGewx03UE5imiKXlFU/Eil19WXeLpzriF7L7+ctS9P6M0+vWbmGgzRslM4jmXnhjYH350wDbsryLj5Y4cveHfKY7InItGPgXWZLj80CzkQiaefMZJxO9tqYqicpuVbe0FjOuvmtf6vG/8bmsobIe6xj0uwIdP8ONEKJsyXtnrgJiFySnsGnuPojj7Cdi5FtktTeEBhM9BxdjxcUXnqmVVvOnWZMGjW+PsngktZASUup3uTPYjye90KF3qbnEX3lBueqHE40FDsV9T7RGH37/MM41246frCq/6qSba+5UpdgEUW8VmkOUqAj6ytmaHlGGRNGHmU6KOHkOyB1BGg2y/g0OZZpDrrac8k2AZ2xUsyfA2mcxj3uBl2NsSwJK8nRP4pvsrjlhsGC9RmvB5+rmNOzM8KIxMmc0+42eZKd51zYowNBY6wyAJ69HLmn9a6GhNnuCWP8JFqUmDk5nW+tnJK8gZ8ogTJ+H4S+dApclUFeMdqS3DKk7qVSlhdqgpoA5+8QXTgl2szmlpvCf8wIhEE95YhBBxs/zfSRB4UR9c+mnV6zlESSg1gDAdev6AbRORF04kJBAQ9RaLTnuUFHr2nC2keFIaHIGGM++66FFD3Iq2W8IR3D/xESPklWlS5NnjRoERNync5t8iyie/mjr7nG4Dhgg1AL9Minem/rxphPfNmjaoFHMzE/k+bhBlqAYZ0HuuwQGtctRSKLYfdPu2tjRT+nvgkEWVM0qbSaIcx3Al4KxtBPvhEztZoA8PXY5WAssnhDbAlTDyql0mw9NrTB4FlyxK1W3BWiG/vlVx+X2A/Lb7vqdWqqeH++XccEQ25rKqBsfr6eB9WOxuT3WTAKv6tx8czlAtoud9OBDAowdmMraCcOPDZtc/uq39SjB7ZUW2rsK56dqbfdmjAuQUwqvz+6X1WQ97YEzgOYTqieUmoy7jiPX/aOj168iP3b4xBZk8QY8YzFRT6AWyumrej+ejoGpCmKMQeMkkef+HpsW7fU28IlQjGKJKgh+cBx75PCfxXg1UwROdJeVNarrX3LrbCnJCwa/C3QnPK1bp+3PZVGr6DBQCgcAzrZWD X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: f0488ae6-3c3b-496c-ab85-08dbf680c60f X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:34.3490 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: etDutqek+h9DiF9ElzJGY2mCmZAwsJE39K9US2/mfkyivenT4wrzhx3I/Q5wLpCW X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7364 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092849_447067_8F403295 X-CRM114-Status: GOOD ( 18.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We no longer need a master->sva_enable to control what attaches are allowed. Instead keep track inside the cd_table how many valid CD entries exist, and if the RID has a valid entry. Replace all the attach focused master->sva_enabled tests with a check if the CD has valid entries (or not). If there are any valid entries then the CD table must be currently programmed to the STE. Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 5 +--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 26 ++++++++++--------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 10 +++++++ 3 files changed, 25 insertions(+), 16 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 4644d4601c0830..2f818b24d931c2 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -417,9 +417,6 @@ static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, if (!smmu_domain || smmu_domain->stage != ARM_SMMU_DOMAIN_S1) return -ENODEV; - if (!master || !master->sva_enabled) - return -ENODEV; - bond = kzalloc(sizeof(*bond), GFP_KERNEL); if (!bond) return -ENOMEM; @@ -622,7 +619,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain, struct mm_struct *mm = domain->mm; struct arm_smmu_cd target; - if (mm->pasid != id) + if (mm->pasid != id || !master->cd_table.used_sid) return -EINVAL; if (!arm_smmu_get_cd_ptr(master, id)) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index a2263d9c3658a3..5f6ffd816aaee6 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1186,8 +1186,19 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, const struct arm_smmu_cd *target) { struct arm_smmu_cd target_used; + bool cur_valid = cdptr->data[0] & cpu_to_le64(CTXDESC_CD_0_V); + bool target_valid = target->data[0] & cpu_to_le64(CTXDESC_CD_0_V); int i; + if (cur_valid != target_valid) { + if (cur_valid) + master->cd_table.used_ssids--; + else + master->cd_table.used_ssids++; + } + if (ssid == IOMMU_NO_PASID) + master->cd_table.used_sid = target_valid; + arm_smmu_get_cd_used(target, &target_used); /* Masks in arm_smmu_get_cd_used() are up to date */ for (i = 0; i != ARRAY_SIZE(target->data); i++) @@ -2629,16 +2640,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) master = dev_iommu_priv_get(dev); smmu = master->smmu; - /* - * Checking that SVA is disabled ensures that this device isn't bound to - * any mm, and can be safely detached from its old domain. Bonds cannot - * be removed concurrently since we're holding the group mutex. - */ - if (arm_smmu_master_sva_enabled(master)) { - dev_err(dev, "cannot attach - SVA enabled\n"); - return -EBUSY; - } - mutex_lock(&smmu_domain->init_mutex); if (!smmu_domain->smmu) { @@ -2654,7 +2655,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) cdptr = arm_smmu_get_cd_ptr(master, IOMMU_NO_PASID); if (!cdptr) return -ENOMEM; - } + } else if (arm_smmu_ssids_in_use(&master->cd_table)) + return -EBUSY; /* * Prevent arm_smmu_share_asid() from trying to change the ASID @@ -2726,7 +2728,7 @@ static int arm_smmu_attach_dev_ste(struct device *dev, struct arm_smmu_domain *old_domain = to_smmu_domain_safe(iommu_get_domain_for_dev(master->dev)); - if (arm_smmu_master_sva_enabled(master)) + if (arm_smmu_ssids_in_use(&master->cd_table)) return -EBUSY; /* diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 887669f04b55c1..f72aebaf95f981 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -602,11 +602,21 @@ struct arm_smmu_ctx_desc_cfg { dma_addr_t cdtab_dma; struct arm_smmu_l1_ctx_desc *l1_desc; unsigned int num_l1_ents; + unsigned int used_ssids; + bool used_sid; u8 s1fmt; /* log2 of the maximum number of CDs supported by this table */ u8 s1cdmax; }; +/* True if the cd table has SSIDS > 0 in use. */ +static inline bool arm_smmu_ssids_in_use(struct arm_smmu_ctx_desc_cfg *cd_table) +{ + if (cd_table->used_sid) + return cd_table->used_ssids > 1; + return cd_table->used_ssids; +} + struct arm_smmu_s2_cfg { u16 vmid; }; From patchwork Wed Dec 6 17:28:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482107 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 63B6CC4167B for ; Wed, 6 Dec 2023 17:29:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xAnuPweyUXR4T862u9iQk3nXpQNwFYfCKAVh2ux5ZB4=; b=Y3LPrTIPrtRq4m ICO9pfg4l0/kQhAWvVp5IvZ7eoU8xUloKTZDzonONUyXqRkxoPUQEIQKwjf/2uodtvpQJWnQylsjr gm2ill98IwrZfkEeiTMyek5JzRGnBAWyUgR27tO/GUO5NtlZXZ4vsnrLPBYZhHne3VTIv+9q0YD// B+HzSz/Jt6nLmf0rERbEkfkPdL16kv84mNNNh3LMbLkie/oEHiKatDaAxiVHRLOEXRRC3Rr3RAg+n /7DsX4t0Nc9Jltsq+bwIL0pfSbntIcKd2vxJOmssOLqgeJY+9qNsMSvQEFvvzVa+0eKMtefSC72tk vwBzpx/OIYacqa5RHgXw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvi3-00AtqJ-0A; Wed, 06 Dec 2023 17:29:19 +0000 Received: from mail-dm6nam11on2062e.outbound.protection.outlook.com ([2a01:111:f400:7eaa::62e] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhc-00AtFn-2F for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:54 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QozAT6tOlwNm57TTQXEGaMTPN/b1uyj1Bn4nZxWyceSNZspXLJY3D257bDcOFaUWXKfmj4JyyqzieijaGb/OxkS+7JVHMsAxt5ClCBdssZmovg7ovkuz3aax8LjrY8t9C9dPbeMidwFnb83kNskGoRHKIGoSuO0OMgalZbJr8cdKgsO3nPWHrSct/A8eOIGGKmVUTTRCquWSC3rPzmSxls7xmku+lPKwDKETYqrCsqLl7cSiwm7pl2bdHBGdqrOA5q7jcgo5KSEfgCQ2YZrdGwOLWma6pNIe2ID8hwFqe7tF+LLC96r2Jp7o3LMWD8cGZnlMRbZmYSrBWHxN3VpKKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=B5kMJb17X3q2OlpYDtvCNFMDA85eXN62Bg/rYC505wY=; b=SmRb/crVntAPpFbSNntnV9BNfxFm1EacuNJNTRquoLFNwQ2XB7YJFbuDK82M5BjpQ6br1LRqg7sTTSXw0X7P3cdXPPomiCP592JXXM/9UbtyozU7bvyQH6Gl+yjPh5Za3WJjh92mkRSfOcV8IFZPmvoNgdD4fDfbJNmn9gXSJ81XxwFlJN194fjzXN9kzZ7HjKF7WTTxRpxLcV6FMr2n1bNu4wRCHaxf/bONMnEVG6XaO6YUOQ7qANG4kqPiJhhauBaWnPfjIL9PHHvoE4Wo48XG7APaL8AqXIZhcmKzFD2K9HZ0Zomic1myr2JsJ0QT9HLyrhnW7b9cAp+2JzWYdA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=B5kMJb17X3q2OlpYDtvCNFMDA85eXN62Bg/rYC505wY=; b=mhq0M7n7cvL7NDIhO17WOuOz1GBnF7rto5qb8LNleJ7+Hl4J2T6ZqfM55pr66Ufym/DEI2PZag5Q5NcMnLijSE+uQxBFZMaLP2JcpPHzmRm/MV1lBJJTCYOctw/2rSyr5lAv0OLrVk02h34mYEnL+8mKUrTI0pHIZLHOe0o43bss2TzrXvq2UgnxvCdU+cp5SGkYFvgblPoyPpNag9H03hJXvkvzx0MXb9Eub9xrNPZfcQI6f4Dl0hHuf4R0Qpv0mfbaszUoae4IbcLJC02/yFM1BJ1Ru/Gm8TO2V9D6zq7UAfVtXBv8s4DBDWu3uoXX55/sXxo+79sysP8oN9Ymzw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by CY8PR12MB7364.namprd12.prod.outlook.com (2603:10b6:930:50::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.25; Wed, 6 Dec 2023 17:28:39 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:39 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 17/27] iommu/arm-smmu-v3: Thread SSID through the arm_smmu_attach_*() interface Date: Wed, 6 Dec 2023 13:28:23 -0400 Message-ID: <17-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR22CA0011.namprd22.prod.outlook.com (2603:10b6:208:238::16) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|CY8PR12MB7364:EE_ X-MS-Office365-Filtering-Correlation-Id: fb920c62-906a-4634-830d-08dbf680c627 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: eflRoCAD1J/+uif1so+VSqd/sq6S+au2Dct0xCkr49IejP2iDG8li9Yzv3TL0y4lLXETbyog4PXeTuIcJVKFTm16v1NtT+u/KEjnfXh1liHwwBGFFyWwlY3lvyHXDwkIA7X/07NI5hoetyD4Zt94YK6jL57NFyt9VlGn9QBr17Tgh31mRkRqOxmNi5gjKKZYjNQdKQyaJYFfuW0liWKvgk61XtFe4D7K8+eo4FUfW5rLWYUOiwQQrmOea2VdrS2DCzAJT1UUgYdBw55EmoYWaQ/YLQB6H7VHMFQpxJm+8UBwdl10pzVGYLoX9ZUAkRnheh97HMXQxKXo+mcyvgQ4q8KmWLvX9Gn0HObLKf4Oj7vlj+1em00xLIH2a8yd8j9uOn/CkiMp5zhjSYM8QDYPNrr/3xl3/l9iFr4tvkqwjchFPgJ8dLs4IyzDzac25Dn06ANxw/49/DuZnH/DL8PJWbDzo2Czy9Im6uIKm2NWkcaljr8kQqFSGZblylBjmNMMJRq8/vkcZNj6s5wMW9cyWkysebM/DROCpZM7FgX+Cj7at+elz023LGbJCs9qsERn X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(346002)(396003)(39860400002)(376002)(136003)(230922051799003)(64100799003)(1800799012)(451199024)(186009)(8936002)(8676002)(4326008)(2906002)(38100700002)(83380400001)(7416002)(41300700001)(36756003)(86362001)(5660300002)(6666004)(26005)(6512007)(66556008)(6486002)(478600001)(2616005)(66946007)(110136005)(54906003)(66476007)(6506007)(316002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: oAivjkaDZmZMG6LEOBH/dtV6ZrFcnxvnRrUv3qWqfC2IGIl7aQ+WdC49DNjWQunekGPpSBwImok9N+S2qze+yfWAVRqSdl1DypoE3qE0YJ/318+0hjZ2dRFncKDCwCkuyfVJI9reybPa7Myen4mR+21OMeUszF1KEsqmtHvCPhe0fovVpKc4kbKZfPJZ6rWbbLt96CemeXcqZDX9CaJBPsNXDirXUesYZ9S1vWEcUwH8eGTiyQOb+jJM1Dw+HmKPxlv/xjm/MMc4KclOML5yqAkD3JoUSOsi3PeBsmx0dyoQuMfrLYHgHWbfChi+7+M41MZ3eRu3fGNuBD49SjLJn1qIGGFGhOYrHndwBrFb1P5tE4UAdCcnQi65IZzwm0Z1nFW1bJNPTzhklJwejAvV0+0uFeAuiIM1ZoxwC6n11bkure3ItbdYg25b7te0reAe/2dXX5rT3Gaw9YS/cjhK6urdmAXHedP+5OHOXGXIq0FbBDMWJQaAYI1/DyWkuJUUPr4HVNiV7CG6p2HwGbr5e1JQywKO2C5H7hSXGWVlTzgFu41wIziryM1XA0Y0lnwf1TiU5Zh9wHBg+Aoc04y5DgslD+mFCC6WEfWK1g0Ky+SxazuzdRX/nWfQYY/JlUdi3W0gJvLPHO5WYkHpCq7rs1yk6TQBr/L1VoXI6J4QqM5HJQW9pwtaBuFxBA8XUHbLyd35iMZEqkm6a8jcHfk9rCAlDPrYbbRGUx2zVGjHDmE4Xa2cGF5kgn4sM9Fa8h2tp6a2zFHpkFY5cR1Rrntf9fvXVBzwIrkNkxDFkZJ23inP8vxI+7n4lj01+6T3KAiju7y3nBV8/guBvNIJ8cvDm73QnNG9voqIE44CaNyEzit8RF9dnZjytwxi2FipDCdaLeLnAU9a919kL4VkzyvNCrXyl0jJzkH7+TZPaKP88RBjlCjdEiHV2bapPmSw8svXcRQbZ9vJL5ZtNn3yh2N4tK4s2CaVJ341nCAvpcKLrpCi68P1/Y+BiwNkcDRuakjb/+anV8bQOoBmYi0cqpR/QZ4usyue2R5/zAPXUuf6/OkRwlxpqPHW9jfGeXttUAFfazQKgHYWGLqdefmrk89efp3qLUthS96MKwloyUeyHM1kg6jEIB9mc0DAey97v7Y4mdTsH71DZfa8+O+HLGsDcv0L3QeKUfv3O4TNHepJOKmBG4V04hbjcdQvRQP0xQ6r+Fgr5v6gkTJpC+QcE9xVXiH/I11uOP8H/ibqbD5liuN3xuhZgMqTURQVFGKJ3GU7N6nyA+KVLQdhzuz0soG9l8TADBxUGqkkeH1Y1yLd8MLEL/1fRhDsk78pJCZ/RbUMuC/mJe1T+sYT5l0w/mFXVcAPbJ1xr51UJyRRiSiuMIJWij2sEiegPYXgydTnkqz5un1k5IH5c/ukQqsPb8UsHeZ5ji9v0GCJHZmOGiGynCDG6//0EZmAJt58PwnKKEK7Sw0BPy2TNYyHBW3Be7583BMdItkjUa8S/QZHey4xiRcKT572UmPn8xnf2ZubbNj3K4EQY1SElaQS08QtUnFK8lWxCDeA79/15N4H6MAuvdmCX0JRrk6P1A8ubYPDdrdW X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: fb920c62-906a-4634-830d-08dbf680c627 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:34.5015 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wevykx50v299wZUQmzOP36QcWrzyChrN010IBVOA6wsZzQ2QIuP/yTkecUhYooyU X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7364 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092852_762444_C3BE17C3 X-CRM114-Status: GOOD ( 13.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Allow creating and managing arm_smmu_mater_domain's with a non-zero SSID through the arm_smmu_attach_*() family of functions. This triggers ATC invalidation for the correct SSID in PASID cases and tracks the per-attachment SSID in the struct arm_smmu_master_domain. Generalize arm_smmu_attach_remove() to be able to remove SSID's as well by ensuring the ATC for the PASID is flushed properly. Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 33 ++++++++++++--------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 5f6ffd816aaee6..a6cd4699ddcc76 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1961,13 +1961,14 @@ arm_smmu_atc_inv_to_cmd(int ssid, unsigned long iova, size_t size, cmd->atc.size = log2_span; } -static int arm_smmu_atc_inv_master(struct arm_smmu_master *master) +static int arm_smmu_atc_inv_master(struct arm_smmu_master *master, + ioasid_t ssid) { int i; struct arm_smmu_cmdq_ent cmd; struct arm_smmu_cmdq_batch cmds; - arm_smmu_atc_inv_to_cmd(IOMMU_NO_PASID, 0, 0, &cmd); + arm_smmu_atc_inv_to_cmd(ssid, 0, 0, &cmd); cmds.num = 0; for (i = 0; i < master->num_streams; i++) { @@ -2452,7 +2453,7 @@ static void arm_smmu_enable_ats(struct arm_smmu_master *master) /* * ATC invalidation of PASID 0 causes the entire ATC to be flushed. */ - arm_smmu_atc_inv_master(master); + arm_smmu_atc_inv_master(master, IOMMU_NO_PASID); if (pci_enable_ats(pdev, stu)) dev_err(master->dev, "Failed to enable ATS (STU %zu)\n", stu); } @@ -2523,7 +2524,8 @@ arm_smmu_find_master_domain(struct arm_smmu_domain *smmu_domain, } static void arm_smmu_remove_master_domain(struct arm_smmu_master *master, - struct arm_smmu_domain *smmu_domain) + struct arm_smmu_domain *smmu_domain, + ioasid_t ssid) { struct arm_smmu_master_domain *master_domain; unsigned long flags; @@ -2533,8 +2535,7 @@ static void arm_smmu_remove_master_domain(struct arm_smmu_master *master, return; spin_lock_irqsave(&smmu_domain->devices_lock, flags); - master_domain = arm_smmu_find_master_domain(smmu_domain, master, - IOMMU_NO_PASID); + master_domain = arm_smmu_find_master_domain(smmu_domain, master, ssid); if (master_domain) { list_del(&master_domain->devices_elm); kfree(master_domain); @@ -2554,7 +2555,7 @@ struct attach_state { */ static int arm_smmu_attach_prepare(struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain, - struct attach_state *state) + ioasid_t ssid, struct attach_state *state) { struct arm_smmu_master_domain *master_domain; unsigned long flags; @@ -2570,6 +2571,7 @@ static int arm_smmu_attach_prepare(struct arm_smmu_master *master, if (!master_domain) return -ENOMEM; master_domain->master = master; + master_domain->ssid = ssid; state->want_ats = arm_smmu_ats_supported(master); @@ -2600,7 +2602,7 @@ static int arm_smmu_attach_prepare(struct arm_smmu_master *master, * smmu_domain->devices list. */ static void arm_smmu_attach_commit(struct arm_smmu_master *master, - struct attach_state *state) + ioasid_t ssid, struct attach_state *state) { lockdep_assert_held(&arm_smmu_asid_lock); @@ -2615,12 +2617,13 @@ static void arm_smmu_attach_commit(struct arm_smmu_master *master, * SMMU is translating for the new domain and both the old&new * domain will issue invalidations. */ - arm_smmu_atc_inv_master(master); + arm_smmu_atc_inv_master(master, ssid); } arm_smmu_remove_master_domain( master, - to_smmu_domain_safe(iommu_get_domain_for_dev(master->dev))); + to_smmu_domain_safe(iommu_get_domain_for_dev(master->dev)), + ssid); } static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) @@ -2666,7 +2669,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) */ mutex_lock(&arm_smmu_asid_lock); - ret = arm_smmu_attach_prepare(master, smmu_domain, &state); + ret = arm_smmu_attach_prepare(master, smmu_domain, IOMMU_NO_PASID, + &state); if (ret) { mutex_unlock(&arm_smmu_asid_lock); return ret; @@ -2692,7 +2696,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) break; } - arm_smmu_attach_commit(master, &state); + arm_smmu_attach_commit(master, IOMMU_NO_PASID, &state); mutex_unlock(&arm_smmu_asid_lock); return 0; } @@ -2757,8 +2761,9 @@ static int arm_smmu_attach_dev_ste(struct device *dev, if (old_domain) { if (master->ats_enabled) - arm_smmu_atc_inv_master(master); - arm_smmu_remove_master_domain(master, old_domain); + arm_smmu_atc_inv_master(master, IOMMU_NO_PASID); + arm_smmu_remove_master_domain(master, old_domain, + IOMMU_NO_PASID); } master->ats_enabled = false; From patchwork Wed Dec 6 17:28:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482094 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D4DD1C4167B for ; Wed, 6 Dec 2023 17:29:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7HIjLpotvaQMZN+iTC/Sd9FY4Vg8l+fV/E2GM5FS4vQ=; b=Wd1z1v8kdwSmuW vCqygVLVbRzS1NTbgRs+xg4aiaiH0LPDRJvBlpR/apmBvZP0BacNmbID3+fqgqYPdvU+JBoFR8hzf 3PwPiDglhUECEwS01iBfRXg/7FdjXwGlIxbkI7rGVPgGv8nhfUWVFXN0KoKihIlEsesQRcZ9DcowO gSuLc7WH5qDaFoq+A69vCHxRMFF60nkEPk/biUdDflWgZ+0vXuouwFtBQurgFxeRqfjP8aZXioE3i Dvkf2/9ei5o3/bFaY7GlGapiJLogf5/Sl0KS/KuBb/KyClTjgZJttvTLUAMnp9KAsYcVLkPvcea+t 7HrY3hQ2fF0IR1C0bvUQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhU-00AtIM-37; Wed, 06 Dec 2023 17:28:44 +0000 Received: from mail-dm6nam11on2062e.outbound.protection.outlook.com ([2a01:111:f400:7eaa::62e] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhQ-00AtFn-2V for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KMcinf0hBL+oARYEB9tW5oYXUez1u3oeaUo85XR900npKvvFep2kblaXEqXq62vyKroENHCA4oOKuRNNXdwP2XI73CtdTSke+XJnmQCcQbPJwCe0uNvYLw9s+hxZUVbQ2zCnqRN/nVLrsbFLgLUKFtJwyB11uukSGidekEJyRzB0dWmb4EpiLLGNqZrfdsH1AFrvQN7/C2Ug8EfFQoLe4bk3ItnHRchEmLmF+cP18/d0dxUH8FrPE517Ouuc4z06EieQz4JYxwLUqn+IP4c6hLgNzGRoDPIhqI1pPcAbbqgt9ctabgZ5JUtv/J5nGi51Wty+7EyHErsi13xIv+mv4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=p3rxQR5ReyI12IY96LBqICVfizbG777B0eb+yQcJlP0=; b=dPxhDysNxPK0BsLzE/HXnIMjTuKjvi2yXo48UfTa49u1UtP8/hYHJsPwDBNuOkUOjzP4ZBDQBMHbnQO1PQCBfkDajGZUGWJCgQf/mp+aBOI+GtrYl8XIxGfgrEiwGddqS72uAJoeyLGXvC7yTIVaz6BS3xfD8tBHaoHWHAWG3KFQh/oDSgcrYA+7CVDTE37uIEjlXrjR+sHfvOFLV2a1+OqxEAA2rWxMFLpkHLV/Y6+YMrHNVk6yMl0fYmkyWwK3NKOWajiJEqjCh7z9BxdHkD7R1A5nJs8l4ccv7mQWvAEs2CfbpCkmJvR1DEcL7iVysUKcaWJTXEG1S1tL2PaIgg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=p3rxQR5ReyI12IY96LBqICVfizbG777B0eb+yQcJlP0=; b=cTjR8zb0F2q0q7HHNQs74SBqPHb5W5Ewj2k2MGpaVVa0Pccnnr1p8Wo2OYXfON967gPCDMcI1Q9eOawj5nOCdqMI60MrbiHqVPtfbzPmJk193Pl0kpmJcz6kXVeNac+4NW2+3B4lVBsLaoNQF4t5lvtPZa0jVKn494indQu8oY1ooLVdcImX6320KMSvbR6TicWvoj1PsWgD5HKjMY8MJgGnVSHlPJgo866cfShyNDRvxboPq/x7SdicIwOZMy9UOKGeLxwV3IENfzWD5CDJY3a4oc92hkPh1bomTFnC9FJShJNRc6QcHfr8IQwcQ7tJxviTjN7Vs4z3AvLVq/B7BQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by CY8PR12MB7364.namprd12.prod.outlook.com (2603:10b6:930:50::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.25; Wed, 6 Dec 2023 17:28:34 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:34 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 18/27] iommu/arm-smmu-v3: Make SVA allocate a normal arm_smmu_domain Date: Wed, 6 Dec 2023 13:28:24 -0400 Message-ID: <18-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1P223CA0022.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:2c4::27) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|CY8PR12MB7364:EE_ X-MS-Office365-Filtering-Correlation-Id: 82d38760-fff1-459c-6071-08dbf680c5c3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: aV1Wwx3fCnC6BG/fWF9yZ45Jl9L3mQnPeIUp8ylCd+PzyKP7MJAGdHTb6SS/nyQ5iaQsxc4dKBqKzr0Q6T+n5gXrYuDsf6+PuFpBvcxRBHxIUQq5ItCx9zUd0iGnd3dXYgb0EoZNmaTWO1xmwFO1X2Tr8DyGvjnWLMldIlc4lIc01IC5NJY5D5vsJQ1a/iUWAaniiy+omjrFW9GIsKuJLUToTMlIFZ50rSZ911nZhmnpQO9y5UsadUXjcCbPkEkw5NQaN4nZUQ/ay3npJ9yACGKK6k2J12vM4RuJo9e2rp60/NTQHOx1J68mvmmAQkneCJDPZY+nio3DuAoZf3AA93ltJFzO5oamuPPXT0AWo0j72UKfr61w5x15xVwdq7ekV1FZef7Ujr1Ww2zTwLrCTN2j7FE0ZeJU6OQaBA+e6+926CXodESkj5yRB9TGLBoPN70wzObNeQ+uVlEnMkAQ+ywRCNFDwSlf8v0VvRu6B8fjdQXj1TeVmL459oPh/C7rOJI+QnuTERZrE3Ycv+Evso8EPYI7ipqpdJ4Kdl1NLGAXmmTP5iZVH1c2etHCmrtX X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(346002)(396003)(39860400002)(376002)(136003)(230922051799003)(64100799003)(1800799012)(451199024)(186009)(8936002)(8676002)(4326008)(2906002)(38100700002)(83380400001)(7416002)(41300700001)(36756003)(86362001)(5660300002)(6666004)(26005)(6512007)(66556008)(6486002)(478600001)(2616005)(66946007)(110136005)(54906003)(66476007)(6506007)(316002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Kt1uN62LTTkaawu/fsm9lTFaQkZekTL5SvuNJo0yoJ0K/fp4XUrkZIjFv1z3C75XS7DJfNrhWBanqWIuIH8UzLgME9Vtbj6sZM+Kb2ORRk26UL+WeIsMqMFSJnG+pdq+tc8uta52YmGVBCoJDAjaZQs5UbzyEjOLnteYbNaLcir2v1bf9LjpZKUj7nsNwLELV9aVQiobap8s4E4k+ITWi4alMchGTbPFNiHoz85iSjfNM8grOh2egnm6HomLXUOFy5TxFifsDndG/VHf+OYJ4X+jytpWB360kJR8Nfx0tNuBoPgn+s+bXDG6NYlE4ozXvHo9bmW7Nc1I9Et0xf0Gb5ZCyTGOFuZ/QzQKnNi6Zic2ixpI4dxFeJOX5u/8X0KsE3vRMRcsilErJnjhjyFz0oKJYwH8k1cjbxoxFvKd0Q1YYXWKKxrPUY3aCQ1kEXXrU2Cg8u5tpVStKnKmRbQ03gN5MumRvtzZq/5036bgmmeSyPVdI4AeWjo5711EnpYS9UpeBmW8ATJfqYPzvLJZfNplhAPlFfcp+aU48VXE5T6pHPhgBTaoH++yiFm0N//PBKl/rwKMVLK7+dZTpHMelod8GYvB6zLz3u+eVMK1PIFzHoFMmrOU+fU9hpPDcFtu81x5rDyGQp7hMoUD4bRK19CQru5orMRdU+HucC4f8fFtburE0iuGbIGwyYTu0fQxNd8LIVz/MpX59c+26FugWMZDSEULf6xcRlLKazrMi44+T96hOyAvBzXoHH+aGwMXnRRQI/jxtojIjEk1i+wTOZ4hdrLbQL+eMT7qh9o9Im5uKUxU7h9Vdv7VAia7DgHVIvXkdCx1JKEJ5SJ9kbLLEpboYUc/Q1xXHsF0auJ9dd+lF0wbRlHe7BXTXR5J2EGVAdb08WZYfGMCgxzJOIWP44Mnmu9AFwpcRh6AUc6zMRn+dn6RzDHgVDP5yLbs08LS50yBKmkwPNsHxG7NsPPlx/jR1xmr2JlV/FRh7nhxBkhb7WQ9Fo/LSNJOyjAS/2ps1kQ+81WkGGI4BJqFugfZOz1BW6Vu9IwHZ/oWs3d3YotAt7q3VNhhMFDNp2H0Vz1HFepgWay0uPH2h3d9uq7oPDHTCQvqHUruE1iLgIdiVUScs02Pg2SPXLZoBLBwPVWKsPW6O89mTvk02vY3SM3767ijKFS2bpp5WQgRSd71E/lZgBIPvZxJ51dw5rNt+3zKRg26+eOtbnMfTP6BkZzFughv9+aDUXEUQd9TFl3mm+UsOufGpD4W3Iiq7Szk7+TPxg/Awy8ZsGeS5bzeKv8+o7B1iaVYR62SsyDSImYDGsHL8u6AqfqDWFC1/29k+TBCLK4naFMuPV+QLShHVSdPSMLtOWly1A3j0jZ/bcTxDJenPAO4Ddb07xIQ74uSXysQ6+2Nj1z/S24qEgwK+C0JuT7dXG/+UWQBkorIVfUDhtvxvjEaurdzWMn1dhPxIxxGmuIyeMCAyjEJ3DBi8oliJ1jUJdHqCP9lA6HXZWIHmyojpwUwJE6TINSSkg/WYUuUgta5nKKOz7j2fNvwjuDOhnfq/RgdIpF8Q8C7yaNB/W4JxXmMwba3lWH9iLWbLwvs X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 82d38760-fff1-459c-6071-08dbf680c5c3 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:33.8370 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Ag8pXhTYKR/j66B7wmhQDEFa1mH4sVsXPIBLut5VoncaMjxFecHpM1DQ4AnQSsSK X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7364 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092840_837298_67AE07CF X-CRM114-Status: GOOD ( 16.28 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently the SVA domain is a naked struct iommu_domain, allocate a struct arm_smmu_domain instead. This is necessary to be able to use the struct arm_master_domain mechanism. Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 12 +++---- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 34 +++++++++++-------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 ++-- 3 files changed, 30 insertions(+), 23 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 2f818b24d931c2..5894785aa901e8 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -646,14 +646,14 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = { .free = arm_smmu_sva_domain_free }; -struct iommu_domain *arm_smmu_sva_domain_alloc(void) +struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned type) { - struct iommu_domain *domain; + struct arm_smmu_domain *smmu_domain; - domain = kzalloc(sizeof(*domain), GFP_KERNEL); - if (!domain) + smmu_domain = arm_smmu_domain_alloc(); + if (!smmu_domain) return NULL; - domain->ops = &arm_smmu_sva_domain_ops; + smmu_domain->domain.ops = &arm_smmu_sva_domain_ops; - return domain; + return &smmu_domain->domain; } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index a6cd4699ddcc76..5db78e6e03dcfd 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2228,23 +2228,10 @@ static bool arm_smmu_capable(struct device *dev, enum iommu_cap cap) } } -static struct iommu_domain *arm_smmu_domain_alloc(unsigned type) -{ - - if (type == IOMMU_DOMAIN_SVA) - return arm_smmu_sva_domain_alloc(); - return NULL; -} - -static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev) +struct arm_smmu_domain *arm_smmu_domain_alloc(void) { struct arm_smmu_domain *smmu_domain; - /* - * Allocate the domain and initialise some of its data structures. - * We can't really do anything meaningful until we've added a - * master. - */ smmu_domain = kzalloc(sizeof(*smmu_domain), GFP_KERNEL); if (!smmu_domain) return NULL; @@ -2254,6 +2241,23 @@ static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev) spin_lock_init(&smmu_domain->devices_lock); INIT_LIST_HEAD(&smmu_domain->mmu_notifiers); + return smmu_domain; +} + +static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev) +{ + struct arm_smmu_domain *smmu_domain; + + smmu_domain = arm_smmu_domain_alloc(); + if (!smmu_domain) + return NULL; + + /* + * Allocate the domain and initialise some of its data structures. + * We can't really do anything meaningful until we've added a + * master. + */ + if (dev) { struct arm_smmu_master *master = dev_iommu_priv_get(dev); @@ -3207,7 +3211,7 @@ static struct iommu_ops arm_smmu_ops = { .identity_domain = &arm_smmu_identity_domain, .blocked_domain = &arm_smmu_blocked_domain, .capable = arm_smmu_capable, - .domain_alloc = arm_smmu_domain_alloc, + .domain_alloc = arm_smmu_sva_domain_alloc, .domain_alloc_paging = arm_smmu_domain_alloc_paging, .probe_device = arm_smmu_probe_device, .release_device = arm_smmu_release_device, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index f72aebaf95f981..c1181e456b7d5f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -765,7 +765,8 @@ to_smmu_domain_safe(struct iommu_domain *domain) { if (!domain) return NULL; - if (domain->type & __IOMMU_DOMAIN_PAGING) + if (domain->type & __IOMMU_DOMAIN_PAGING || + domain->type == IOMMU_DOMAIN_SVA) return to_smmu_domain(domain); return NULL; } @@ -773,6 +774,8 @@ to_smmu_domain_safe(struct iommu_domain *domain) extern struct xarray arm_smmu_asid_xa; extern struct mutex arm_smmu_asid_lock; +struct arm_smmu_domain *arm_smmu_domain_alloc(void); + void arm_smmu_clear_cd(struct arm_smmu_master *master, int ssid); struct arm_smmu_cd *arm_smmu_get_cd_ptr(struct arm_smmu_master *master, u32 ssid); @@ -805,7 +808,7 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master); int arm_smmu_master_disable_sva(struct arm_smmu_master *master); bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master); void arm_smmu_sva_notifier_synchronize(void); -struct iommu_domain *arm_smmu_sva_domain_alloc(void); +struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned int type); void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t id); #else /* CONFIG_ARM_SMMU_V3_SVA */ From patchwork Wed Dec 6 17:28:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E93EFC4167B for ; Wed, 6 Dec 2023 17:29:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+orfktqBiFiBFMVqFJXw7hY481A+SIwVL4qd5bGGFA4=; b=nMybQhr7lBU7Mu 5tqKXgOnFASpPuwxnoCO+b0FmZIyxFGdEJVOR7SUyMwlDV405sVoL82sDhRuwj29cqpGI3njlt2Vh AWSUG9v68UpOyIkLQiWwRCbgrYB892DBCx0V6Obh2Yjb5K5zvIwZENnohKFvnYF+XLO3WMFmXaeZ4 Vt0NkS5oiEfR6E07hliW34Q19BnHTWdoOUGfRPoBOv/UNdOdUae28azay9Y8RjnmsV3PZzpgVuGsY wWqgf001wXx5N3nnEQs+u+AukQWgWTJwy2BmO+/jO4usDH30H+9hoZKsD5KS3KU9FT3yTcxzIz5fz JxrGkOA429EB0hh84TIQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvi6-00Atta-2C; Wed, 06 Dec 2023 17:29:22 +0000 Received: from mail-dm6nam11on20615.outbound.protection.outlook.com ([2a01:111:f400:7eaa::615] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhd-00AtKY-2D for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:55 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=H2dNAWpUrM5T0J49BYecQIk5134YDBogTUXrqBN1UadjaY0FKiol9+MZh0NMCxSJibPpzGDu/dmgOE65zYRrPtq94AICSor6+AErYwFPJplNxhBZkEsrI3ioXUXEx0rKRIKaur5UzHYRKYNzhBdHFg0sitpQ4Dlm6f6DbyndIQF7GczpvEv1dR5ygNx/wNRvHRI4h+3W+GNsfDRsAryNVeTdCbBkew6bAxs/UBfTib3thZbzKN1hGUneQis5HLzRIwJPlTmD5Bw7L1cbmX6zrrXjSShHxStgGYTeHotBmoXFthrwBr+t8hQ4eRaAXbims1mG+66lS1M6W5YZGMaXSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9sjhrtrMLRm8dbYksvZfGbyaQVx7b30tXACD03u1ajc=; b=hnQWhNZJpW8rJqOaFgPscP7Rh4fAQI78gNt5j1DOC10Q8825dPLnvzwlwn55Hm+MRMqmc09VJoMZKXOOrMfjyl3fp31EzeFtl1Vr82JekUHDAnFsbJRUGjCj0ihfcKcRv31QQXAMD4qjeVcJNMPXNgidtIp5UrYSQM57kBnVj3eqZtS7suvnHZWTpxEKxqgiM2/iEmWOx4diZ7MJ1waYAEtYzF7IWJmgAMTBaV+ZBY4esy8MSt7FoerfL/BWS92zPH8qATGF6KObJjSA9MjhBtc8UhLkAYtkBKkzsdO/y77+XFLEflC8jUtJvlpL0vPP/oAZETKovRZQ/o9w6UO/pA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9sjhrtrMLRm8dbYksvZfGbyaQVx7b30tXACD03u1ajc=; b=CZMHOntumtRcrYWQV20libssDW3QtO5+VXNOxktxajfqRq3Y0mnc65RZ8hRFW+f3eESUe+jN5FpiPn7KTgTzYdDK60TE+uH2ivznLvbXv6VG/7JuQZiNPlrvG2W67xmGbrSBiYsW9Grb8gkTj0uRGxG2CtRN3l96ZVP/THMuzlMPBIdKvuIhpHvtcGspfhaMM+M8I6x4LvYZnZ8nI/lPbNxtQbNFhKT9ht1Dpxx3oNO3KLjRqoWMvwQcLNplVMcKtxlDBuakBGmjfg7J96dmBSwr1mhFCda4B5+6NasXiXTcS3epKqjme3Acr+75lo5Urou3skYtkphYc5ok6ijVXA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:42 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:42 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 19/27] iommu/arm-smmu-v3: Keep track of arm_smmu_master_domain for SVA Date: Wed, 6 Dec 2023 13:28:25 -0400 Message-ID: <19-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR1501CA0026.namprd15.prod.outlook.com (2603:10b6:207:17::39) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: 688ca31d-d696-4777-cd16-08dbf680c692 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4Y1xJIXs4nY7rMnDz+4DeLCaw9LrsmCjMHZCPiOXqrBS+4djbwfH48dccPwGVdJnW+f3K4+09K3KQJwNx4dPbjWoLa7dfZAGPBZKKvD85saQc+Xm+I1WSZvS3YSEC4Xn/rZzZggthYTiIj95nmn4UlRooBRSyXRj3eK5kxD84IyOXIbv5v2mwPX2KgtlHh0kx4dlDbvWFNqqESRcsVeO4Ief9DmHodlVRlJNWNCYwXvLAzwUdnhC/4d34rBx/5KtBm8zGElHrJZ65TExFHeaUdCIsPgcKP1BdAFvwS9IqDekFhdWnHFgjHtD6dINfabZrjC1tOUG8c+6dllXb+hLm+3IQp5y28BKo0PnGYVi+Aq9lDmgxPahu8POT0PKoAYo7vuRObseF4OhRnYPv5gM9Gf8JjpfNBODwWWvgUa5RLa9lEJ5QpCZ23eGrhMNcR57OOwWjoU2t129IUDHT5K9/EsZX2YEvDqBA4dgYWTKVr1wvmR5jAaQuEre4iDSJJhXBwXkDjI/QnN7rItivfyrrhMoUH54ZaITyUf4CIPk3g+JRCPhkwY1sEd9HRBrjm50 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(4326008)(54906003)(7416002)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: /1TSTdvP01GRDCWUYfogrcfxL0T6A0s8EOvJ0MdqL6P+RTz4ThgwiRd+R53PuGc8tGAvF4AevjiZ+MvuM8/dQJdV3IXeDLN8Qlxt47smxZsDpEOABF8vAkGJ2g2mVOsGJPgwOHBdQkJyC5012XbX9XdzN0Wxf1nK98eRu5gjkL6A9xUT21ybISiJM0UD/1NsUXTbt8Cwf1zi4X4MQbfaLb4gHkGIyhMnHpBRpIRxm9CPmeRnrPsw+WqN+/H7/a0WTSoU0CjeoT+wN8kG9cv4GgC7groIedcQ2U3p7FDCimb4yzfjwqKGnChMrRZq0yRvWqNZVrNAEKRLMOQqW6Vs5LdNEE24ayuWyqpWkaIFmtVHWFwHFGUmkwa48TeE39gnHgPp9g8W2VbM1GdTtXHu7FZjkESi0xF4LN5MQJvivmxbqfg6oaSOyBYzOuNzJ5TCkh9lhBvxfpgz2aiNMpk43ZI5LPUsweYLwQYkKEBF0vJWgxoRsMzJUKED93K4O8wIHsGj7Em6tBZweuLyqU2/wj39VhXKmm4VV0sN3nVLdDj7qsb7IZdTWHy/xb5oSG2p1MXWS4IR5Y15p8UQCa/Kpgf4QuFMyXDFeHpdnZFqswEQ0Hrl6US3CUpaLD8zj7sm/AaprByn+dsYIQp4GjTegTuT5e3PN+PHggVhNYyVkKJ668yWx0m3tfQtOT1C72JbJHQE2Pe3pVAjOMzKMinrLhUQUfNYedY2cs+aUpPK6nWgbd/uayWr33avCfT3DtIQ3rG6PsePMX14k9oHmx+CiR/cmLJ/BmQNiracT8QVrKpPiVKqFqaiMHjd/Mg4kuraM5RHCX9Ztnkkzp/a6X4DHIeK41Q0dVyzyc9bILc8XXxqhbDv4Q7eH4TUuuiqDC3sqvw1nxuo/TAoh4NMt+a9P1iQkBGqr6jC2UzWoOA5h4ipHPHgR2ikG2Al4KwtA3k1T4YMCMGVGn3pjsE+SIaFP313SCG5mfEapjeVGiNqJg1sWKCMASQbkWN2YcYr9v+we0Fh5sEbUknlNCFRvFXgaR/0cBHQ0DsQajxg5G4SsdabaFhla2ffa93eXxBOava/yopYMa7tSS/EWb9GzSUZBUc4af6TKimlS5AJPDdejakGPf+7nicY0YZEnChi1aFLF8acyL3/LZZfKSsuj1O/gvB/3YavCiQPWmP9NXy742ZvMXsNgL+s0AwNkqAQRFMkO6p+T4s2hknu7iLwoHWZdYjXg2A48mG7drAyDXj8woQkvkehDSlUT7juBeUVQkBfmqHB41ZBQiyiaZidurwVcdC5TH3K8KBZvYfEtd5S22cHtRFrJJPrk2+fnOlC8Dp0eWV3g/KkMgL3BudB64n9y1TXW3kGopblujxdGn5DE6GyE6ODMmueWkjYoXV9E7EEmC4GZeA0OlKWsmAgn7Imfzhrkn+WnZ4QJ3poA1r074ySPIznF52+IQPBWx9DL8oZEY6Und3ekY9pvnxBVdeyoAOxKRwAd7MVWTJzjY/+I/bmSqPrrZx6MoKovl3JN5LeUVIyQhDG9ASqD83OMmSm5GZNHcKrnQ1GxCN1BgM4Pw5gyo2ze3ZPYj2ywwvKW473 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 688ca31d-d696-4777-cd16-08dbf680c692 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:35.2055 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Djhak9lsDLU+h9aKhRqOJxctTrhYSHaJPa4xqQE+CWGiMKul4web2Vpo2FPxHAeb X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092853_728855_C4A3F551 X-CRM114-Status: GOOD ( 10.51 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently the smmu_domain->devices list is unused for SVA domains. Fill it in with the SSID and master of every arm_smmu_set_pasid() using the same logic as the RID attach. Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 5db78e6e03dcfd..10dc7edd40e1b1 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2712,6 +2712,8 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, struct arm_smmu_domain *sid_smmu_domain = to_smmu_domain_safe(iommu_get_domain_for_dev(master->dev)); struct arm_smmu_cd *cdptr; + struct attach_state state; + int ret; if (!sid_smmu_domain || sid_smmu_domain->stage != ARM_SMMU_DOMAIN_S1) return -ENODEV; @@ -2719,14 +2721,30 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, cdptr = arm_smmu_get_cd_ptr(master, pasid); if (!cdptr) return -ENOMEM; + + mutex_lock(&arm_smmu_asid_lock); + ret = arm_smmu_attach_prepare(master, smmu_domain, pasid, &state); + if (ret) + goto out_unlock; + arm_smmu_write_cd_entry(master, pasid, cdptr, cd); - return 0; + + arm_smmu_attach_commit(master, pasid, &state); + +out_unlock: + mutex_unlock(&arm_smmu_asid_lock); + return ret; } void arm_smmu_remove_pasid(struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain, ioasid_t pasid) { + mutex_lock(&arm_smmu_asid_lock); arm_smmu_clear_cd(master, pasid); + if (master->ats_enabled) + arm_smmu_atc_inv_master(master, pasid); + arm_smmu_remove_master_domain(master, smmu_domain, pasid); + mutex_unlock(&arm_smmu_asid_lock); } static int arm_smmu_attach_dev_ste(struct device *dev, From patchwork Wed Dec 6 17:28:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482146 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DA068C4167B for ; Wed, 6 Dec 2023 18:31:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7BpwkH2HReUWpX19lTtSy2eIBJ9f8ocJetkOMaPpcqY=; b=dt9dl9BLGvCAo+ 6sBKd8RpDZYeWaTGROIUABNRV3uOcZGjP1XG1+v1V3W3O8IPDmFgOicNCJntywWO2A0s3eVZSPz4j JsB3NX5ZTHlqYUleebKoq+OyRpyPu6w/hCioTWBOUdws43Mfp7bYsbHGIaSsEmWpczwJptGilDkMS /eYglMf7VUrVa4vFJ9dECCUqOonqXfZKRJm5I7U3SbulSkuj2ChBuy42Ke/o5m7tGAzMT95gQWqe4 +dhBLZQk28cUlq4TVO1xQlmHTgD9qkXDfs/wVgT4rExXf2EXaGhm687wh5l41yy3dUisXoj4xl4ke CV7+TTzqUeNTap11Cxhg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAwfP-00B2B7-2m; Wed, 06 Dec 2023 18:30:39 +0000 Received: from mail-mw2nam12on20609.outbound.protection.outlook.com ([2a01:111:f400:fe5a::609] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhh-00AtIF-3B for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:29:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TcuKrJ+eHCsnatuGwtUBt9ag7Nz9I5A0lUJvWSrzT5oe8cb5VV42nCJYggzD1x0sN7K5tQPd7ncI71OUPrMKVWv8axGV79pGbCJ3APcb03zipd5RVGtzZN55Xs4EhTBK91QrKgcKiekaps95s7F2Deb7u4rt1jW4HX94/RuqqQTe+m0u5uyjtyNOolmR9Y/QSfAT+bwl39yz5PQVyuMt2OX1emb6mcppQCLGPS3kclVMZoodmES7TNTMflL3wWHBLQrCDhETpzNKdFum8bgHaaO8xRyqtv0X4v5r3vIPe4Npx3YvP3S2S6MA4koKS2qs2p2JL8gcm/xeAa3J6jzybg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1OTGz3qIaECpLc7wVaNfV9wng7AosVZuvH9ccjI3ex4=; b=DLitG49i0X+9/3tCiJCnjNIWJ/ASzxjytoPMx3GRu00uykRTohVvnJPxi68QaXrWaaijYSw11vYfAh90nTKz3IHWEFU/4clK7jiQeYQHTVlpuagHpIJmaV08ywiUozsma9kiQavNyfxJ+KlW8f3sphzTT8d1YNwvu8ia93Q9YYGxnN2TDPbj4dMIaJRLal8P8eNcoBqldLy4n5DJ6ZoK3V8BuId3IkPeFNOqDZx08r7kxPXFAAwa4JuRvt5qtovz/vbjPmc768BNbBbsmp0BxQCB1jCAt1Q9B89Sz4H6pYkCk0VAOOD5q7urTKU2A/T8x0oEJi8qBh6TnQj+0SRcVg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1OTGz3qIaECpLc7wVaNfV9wng7AosVZuvH9ccjI3ex4=; b=U4FtBIR0y7nAQmixxNthsTtbsru7L97GUD7Qqy0GAX8vd4QYAI+76HVGy6syOpLcOOJnHEFTAQKtXSmhuRkmm1awHOvZGm5C5TbDOAbVJT838brWSb91YI7BFgRfCzz6C+q9DnEBRn6Q9XggZomfazXz7KCQX/HvlPdbV6BG9MYv+z1ONznHrMX9IuuEOp30QCjKyCR+Tp4gtftIXTPeHWhQsTXYQU3voVhXgdpWliq6EwQ69maq/tdGY0UW2+j2SNlfFMgmD3llunEPULvRaD7bXEV8HImXlB9LmHwBjgVHxz5lAoa1lH4+aXd7cRpj2dmBAb599vXu2jJzgYbcDA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:43 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:43 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 20/27] iommu: Add ops->domain_alloc_sva() Date: Wed, 6 Dec 2023 13:28:26 -0400 Message-ID: <20-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1P223CA0017.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:2c4::22) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: 7bc88469-475e-4831-66b5-08dbf680c6ed X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: N+K/KGbpmAfU+DZA4YRcMmHkSSrZvtqy/SCM2GnPnv/hzsKkVL0ioqaVCNMP6a7O/Xoo6TckxA0Zz6A5vgJr4laT+ktyp7ZOZj8qvOPBAJ8CSro8eS0WnM5oVEJOJJ14ewTAzimRdoaUg+YNG2kvls2zVc25gUrHHCI9JB10p3zMSJxMYexxZeg9hr93KCAOC06ntVGRu8su7g9HYmgeCsP4XAya/2CuIkNbZAZq/3MR5MIsYkRC1QqXC1RSYDBkEYp72qnp5IXkHDJRfYA4Pos0cXnwAQCoOQTfciKeIEYVLtO42fT+IdB6z2rSkdDhsGY7Tu4xe63VY6iGnkNY+ZeEgag2iUqyS75ZOEAidIOXEOXpf5qCXA9H5Y98bguJRIZeryLxt2l1V/85avFoIc+S/XvfyMqYado3+IEJS8PSw8XA8zbkNfzv0qnNRATKr7nZkSooE/3EbdbShOnAEzdmcy7lYcdbscw2BulgbTQOKQspI5ekmrCxszgxyvmf/bRlLR85ptSkXynZ22SclJnWlR9sI1ZdavE4v+AEODRZQlLqC0stzbCCTILfUrA8 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(4326008)(54906003)(7416002)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: dYOSobRA6pt+aSr70d9GYOXQ0RHiBV6vv4xF7uQjdCRENODCtr+HDnLzs0uzPcF1Kko6+TMa8+ibSO2o8f3082CCx2qdpU9xHWnKN2NoDUTIWptmtDh7Gyd4a0bOmDd+Ob7gmZnBitkR+h3RCNgxLdvMFidBzVfDdsJ1ct096uRvn2hnUOTcYU7yRNCmzYneyluAfUe05WqDjko38vx+mVQG2DGifM40bpkCgGPSvG91bXME2L+5RqTRe5AO0sLjhObVNjj3d9fm4OxtMXY8BUB0BWNxYbUVvE8aONd4L4p/rMTZP5aC3VPOTxDlaj3d2lcXVjaD4WcqbqHsOFox0PuIHXfW1wjkAR6aHoK6a5l6qh8o5y5retmkW4puGYxZi022fIPkQi03gDck71VPxI4cQfUhfYA3udHE2QgDxvXv9dlob7CkRshcFLFMBZXzEzuCTeSRYYicv0SgPaPKW+l/zK7LCxvbXHFHXk0tKBlwQB9AlyxnLLuESQQefv8fVJ1BZdmagtOFsRO+9UFg5p0hGBl5mYUoVW1JE8507KRojj8rgx9BHVUhVztWc76rwIQS0tF/kGigT1FDHDKnbc8XU/7PKf6yss/UIrWcaAmH/5aCDaIJ1XVFubbR2lipTfwUeDMYniYQuKpcY0liKq0f1j5obUFzP6a+WdjhnP6lRyBEBTDmN2wETOdgLWzNHYWU2OF2JLqKIO/Io/ipmiMamxSNf5uGvpKsxU+s0lNXhLkZjv32n3qm3C8LPOcsgNCfs5j9Q4c29Ejsn5wtmyuR7Uw243YXbDKpL+NpjOFVPhhdHnAsCHHaDw/HAPLjIfEbB4qYflCa8kgF/OrDNX7NAYBkEh5DdA2Pg5T8xhWt08ZmjZq8x/4wJFxvgilVtF5/lLmG3lhKzJCj4evEREl1bVROLZ6NFsinspIlHUlOd38wMj2MN6n9EvfNdXZJjvx5r4sdirTyhnzI8PotJkQkJEkMzo+VJcPNb9460EPJlUmAxWGv/uXJAzlnanZLRBAx/gDsSsOsu3qlQhYk9E+rdauSGjJDnnTE4BSmsuQ6y241R+iQN52WpMhjdw9Dq5xY8p1vvMqT0tVyy1am7yX4xRc0+MyTlKUIB5nSS6IRhrFebAmHJXUbnubfdg2tPmLPJuMAnmn6/kkpOzVvcy5z0VwMJAm7cTllhYtk7+5XGmBfOtr9Q/TIJ82BXHMsa6CG4dUfwV5RmQgzg9yCQgYv6sg2J0S8Y4IdnaNCX+1iK56UL6mfxxMnmjotWy87F6qT7HFHxpbxpu14HDdRIROGmaI9GWtiV7AX2YooMgMmXa8wpwpAW3l5PPHCeSQxp9Exsn+WNsAj2CW90ukoK8aATnFD0nJnlmSaqJ85yMTvhyOL7YAbz+J/k75hK78dgHQVQiwC5srZduDEi2lW0AUH90YZp7HXGIdSns10a6g/R1Vlu6H4jDrXTchMUN2EpvWZG1Z6pn9BeOBInl5sAr7Y+dWC2xEt7praXyOJu0nDfjM6qmH2N+G5WGoe3ocrZpLtd44OT9rHWMADr9W1TaVn5Ge1Ea4k8jBudJniTQmUtas/Ruz6Wa6wu7B+0VWG X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7bc88469-475e-4831-66b5-08dbf680c6ed X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:35.7954 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PMMZ4Fvyt2vet3wcb2SuB8ImOt96ZdYAt0uTb7juI6t8gTcZSL8hElHXkNJQytcL X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092858_028089_FEA85012 X-CRM114-Status: GOOD ( 20.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Make a new op that receives the device and the mm_struct that the SVA domain should be created for. Unlike domain_alloc_paging() the dev argument is never NULL here. This allows drivers to fully initialize the SVA domain and allocate the mmu_notifier during allocation. It allows the notifier lifetime to follow the lifetime of the iommu_domain. Since we have only one call site, upgrade the new op to return ERR_PTR instead of NULL. Change SMMUv3 to use the new op. Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 12 +++++++++--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 2 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 +++++- drivers/iommu/iommu-sva.c | 4 ++-- drivers/iommu/iommu.c | 12 +++++++++--- include/linux/iommu.h | 3 +++ 6 files changed, 29 insertions(+), 10 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 5894785aa901e8..991daffbee31aa 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -638,7 +638,7 @@ static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain, static void arm_smmu_sva_domain_free(struct iommu_domain *domain) { - kfree(domain); + kfree(to_smmu_domain(domain)); } static const struct iommu_domain_ops arm_smmu_sva_domain_ops = { @@ -646,14 +646,20 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = { .free = arm_smmu_sva_domain_free }; -struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned type) +struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, + struct mm_struct *mm) { + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct arm_smmu_device *smmu = master->smmu; struct arm_smmu_domain *smmu_domain; smmu_domain = arm_smmu_domain_alloc(); if (!smmu_domain) - return NULL; + return ERR_PTR(-ENOMEM); + + smmu_domain->domain.type = IOMMU_DOMAIN_SVA; smmu_domain->domain.ops = &arm_smmu_sva_domain_ops; + smmu_domain->smmu = smmu; return &smmu_domain->domain; } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 10dc7edd40e1b1..d09505acd4f8ce 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3229,8 +3229,8 @@ static struct iommu_ops arm_smmu_ops = { .identity_domain = &arm_smmu_identity_domain, .blocked_domain = &arm_smmu_blocked_domain, .capable = arm_smmu_capable, - .domain_alloc = arm_smmu_sva_domain_alloc, .domain_alloc_paging = arm_smmu_domain_alloc_paging, + .domain_alloc_sva = arm_smmu_sva_domain_alloc, .probe_device = arm_smmu_probe_device, .release_device = arm_smmu_release_device, .device_group = arm_smmu_device_group, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index c1181e456b7d5f..48871c8ee8c88c 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -808,7 +808,8 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master); int arm_smmu_master_disable_sva(struct arm_smmu_master *master); bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master); void arm_smmu_sva_notifier_synchronize(void); -struct iommu_domain *arm_smmu_sva_domain_alloc(unsigned int type); +struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, + struct mm_struct *mm); void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t id); #else /* CONFIG_ARM_SMMU_V3_SVA */ @@ -854,5 +855,8 @@ static inline void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, ioasid_t id) { } + +#define arm_smmu_sva_domain_alloc NULL + #endif /* CONFIG_ARM_SMMU_V3_SVA */ #endif /* _ARM_SMMU_V3_H */ diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index b78671a8a9143f..a52b206793b420 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -87,8 +87,8 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm /* Allocate a new domain and set it on device pasid. */ domain = iommu_sva_domain_alloc(dev, mm); - if (!domain) { - ret = -ENOMEM; + if (IS_ERR(domain)) { + ret = PTR_ERR(domain); goto out_unlock; } diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index f17a1113f3d6a3..899d73062f6e67 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -3543,9 +3543,15 @@ struct iommu_domain *iommu_sva_domain_alloc(struct device *dev, const struct iommu_ops *ops = dev_iommu_ops(dev); struct iommu_domain *domain; - domain = ops->domain_alloc(IOMMU_DOMAIN_SVA); - if (!domain) - return NULL; + if (ops->domain_alloc_sva) { + domain = ops->domain_alloc_sva(dev, mm); + if (IS_ERR(domain)) + return domain; + } else { + domain = ops->domain_alloc(IOMMU_DOMAIN_SVA); + if (!domain) + return ERR_PTR(-ENOMEM); + } domain->type = IOMMU_DOMAIN_SVA; mmgrab(mm); diff --git a/include/linux/iommu.h b/include/linux/iommu.h index ec289c1016f5f2..7bd8c898e67559 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -346,6 +346,7 @@ static inline int __iommu_copy_struct_from_user( * Upon failure, ERR_PTR must be returned. * @domain_alloc_paging: Allocate an iommu_domain that can be used for * UNMANAGED, DMA, and DMA_FQ domain types. + * @domain_alloc_sva: Allocate an iommu_domain for Shared Virtual Addressing. * @probe_device: Add device to iommu driver handling * @release_device: Remove device from iommu driver handling * @probe_finalize: Do final setup work after the device is added to an IOMMU @@ -386,6 +387,8 @@ struct iommu_ops { struct device *dev, u32 flags, struct iommu_domain *parent, const struct iommu_user_data *user_data); struct iommu_domain *(*domain_alloc_paging)(struct device *dev); + struct iommu_domain *(*domain_alloc_sva)(struct device *dev, + struct mm_struct *mm); struct iommu_device *(*probe_device)(struct device *dev); void (*release_device)(struct device *dev); From patchwork Wed Dec 6 17:28:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482112 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D2140C10DC1 for ; Wed, 6 Dec 2023 17:30:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sx57tBeyR/wN+0TYpgVscPfrx4NcjjFk56/A2V1Osnk=; b=sVkCWMkNIQ4gAk FAR2PHEDKUOzwb3/gT+2JsVPM2lbQgbREQttVJV9qfRi/8O4/eS26yYTZmgdy5eLEUzas0v1jc9VF +6BHoGCHJ3Zx5dysqgb86FL9kXBTJLNmJkw+dmWlD6sfGU3+XwVeRzEUVEN/EPcRqLGMel7oNDDg2 EbV47cOJG+xaDTss7ZbfAOW5SHGGyY+31qkVpsqTKoe8wX4bTHCQcRNv2LE5FZiHwoAqjMqbaHuWY ij4AvOmPXOz8Fzx70FChWFrPoo8UHTUXNv0ebvFJ+W5OfEkboAZbd7pdfFgphLsaZz16i2ELbCpP6 UHX/s8LgssvdOmtsesZw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAviK-00Au6J-2Z; Wed, 06 Dec 2023 17:29:36 +0000 Received: from mail-mw2nam12on2060f.outbound.protection.outlook.com ([2a01:111:f400:fe5a::60f] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhd-00AtP3-2K for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:58 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Idt9oMficM0By2tv3UOafquVsjlcvQRYgOKes3XAzNGgg17SRjLErkVk3Y5iSIrNhyQapARcqnA+fjHrMh9iub5ui3FjkSFNK+m5rwAnhcOZBA4EdUPfd6Qu00aUWMYknAuPrdZa2sMl3tP7t6AmbhM0HFh2Ic3B4d3i9wMfw0EnZ8YQWG35uXifspX3elh7Nb+zUb6PlO6TANOPl+gjyZdTDNiQ+FX/HtJuXI6BwhOzWRMpXVusH4tYviOcV9xJGTNcR2Czg3c/zAXKFh9DkE5KJoHz+jNTjXt0BhOJ7taB2IZKKMJaeEN45XL7c1c6sRurwRDgzpgVQ/JSgaX15g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xR4Bi7xyXsv46xWFMxe0290eqp9g8/gMLabDGK8gjOU=; b=cMld95tiQ8NpZXkxv3bXaEu5F9iDDNFHMoBFNIZ9TPXoix1W6ku/um91DAyNGqaD5YQN6spaZUp+KDwoZOPDe6O4MNviyuAIvuWwhgPC0d2vm/OD+xfcEiYpeSha/cahU/QKP1dIG5rSfqw3wRcG/eyTnPknMyJTR0FGTgjbb/Gs+F7sPLLjdNE3X2Mo+TDSnZ8m1pCeCKWlrAApDlz/YDReWpDpb8YQFFlDox3NS4ReSks476meM0If9g0WQ8/gUi28Ev/n3j4ne1giYwEdEt+L3R2zSHEeC1HAA4v1juMIUjgacs2jFZL4HNjA4+ZMqxnpbgrBuPYKiwgwlgOmJg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xR4Bi7xyXsv46xWFMxe0290eqp9g8/gMLabDGK8gjOU=; b=C/xZV51npscDUmpTnUlOpYGWNCNQM0dkEtmE41k8g44XomdjYUzN+uwDHUmgms3XFkdWBDZkMz12/nP/36jfc+Ckmt+XVDKAhc6TBSzX9jEwNJ+7uU0x3gZbUnLc8guk/s3CFc2bl8kFJCUxy7p8l3dcdcTQNg2X15c1VRXSeNQKWWzKQCIEqeHqvez54nDGJs4grr3yPRTYJvtufwvCEsbnBMU8DDrrt4RzvVqLcVG1A8yPIvrfll5GWSiUSd6ZYbPURQau33Nj/V9fgpfZvgro9Js2JsdrS0lW5hIgs1lRbTJm6OMVp3ouhrDKgAbgN18UTyY1kGsB8SSO1hYBaQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:40 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:40 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 21/27] iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain Date: Wed, 6 Dec 2023 13:28:27 -0400 Message-ID: <21-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL1P223CA0016.NAMP223.PROD.OUTLOOK.COM (2603:10b6:208:2c4::21) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: f2eed925-e01c-4acc-321b-08dbf680c657 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: eMgjPDHH+cVvmtKcRBpHBmBmCergCS2+urZc5/5DSGIHgpS2decrojd12jXcmVKeG3jPgLs6T7acy4CZVlaLr9fYmbx2Gj8iu1f6N8x9JwKIYBNvnAlhxL6k8v/X4ZH75BsRX3iS1AeDXrSG2SVNOCcqt3ycL65bSe1T0BMKUu7pvtSA84VLYEKzywNkG8Wak0sI6W7e07+b7kJFDvZoK8bGHtj8uvSo3Ugpax2QGQb6ggs4135HPf+BEeb9vRNVaIWAxtfR7i5tBHX+V6/pfzGSb+bDjw+usVAqi9KKn3NFxpHTh3yhi6NSPq9QWwpTzrilS36l+yhV3Ic1SRadwXgl55SDWUpsKpbKtDKxW7xzrjzDwePHUqERjiCND60X58a++Ud9GeNEItmdNzIyaQ19wu3ptp5lxxKmyd1ItgVDMzHiBbJYd68HWEaf4TT8lEEPSlJ7TQeMu5A3G3yGfGk0+geZ9UDVmN7L7qPPxs6dPXXZ4gMu6RHMRhvYTN3rRfaLVdmw8qPUKh/cDR4w+zz5hVHKbhbbEhW/keXI2V/ZHZXJpBL2OZ2mGkUjVE5q X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(30864003)(4326008)(54906003)(7416002)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: J2KqrD58vroOMV3fDgDHXi8PMcLqcIk/zcHjWK7AmTCfyMt2EhUPhDYeM8MtLqxciM7E/vvBJ7LWmQo++zg8Buy/ROIOZatXjmCIXHAEFvmrajbffBJD2aG4U6HAmwh/M1Fnueyr+Kw024OxamqX8YDhmPMFalc0vH92A/IS4rW6OBFouARYGDE3NrP6B8khKbS3HBfXJZQeJkONKQHlxnzfyiCpF7TijZjZKqqYAbeKyH+IHdK1XwpI5PfQ1a9yiOJJ4F3W5MqZ4VvtA7OHpxpJiUYx6sFCy/uYGHLyIXxYjqRHpLS8CzzStKfdQn5b/C8/kq9nT6N034h+0NimVspOOgvCrgvoRgu4BwAc2uffK2LkxC82BGb9Z+TB0y45ZNwVOyTo5xRess9l1T3glx07WyE1M7Y7ERBtQdo0LgXRrCyv21KYndN13I/PwbpAqOcOEhgrwxAEs1oeadKAecMkFnqGq79959ahrHrLRXfTtkTTf1dKazVUVqCLQU5rZUAhf6SXfYlT0Ychz8qM3unseExRmrM690Jl5DVu/G1RZ8xL3Gw++RZhfYsI3XSO/tHuYCxoonS84xDSMdNJ40sTZyRK2prpFsMxVfJu1FY6RyQYxiZ8RHQpu7R6IR5hTer0cPH/iBcC239He4ItkZ3U5QfngUY6ueuApEc6TV0Pv2PpoUIob6EGyjDiB657ppG6Hytbf4Dk4wHsEZNAMWokL9mCmmMqw5GZ3e0sAZLsgwVlgdoAmn726t4sbtzeOs4twxA3f2FaGGhZlgnIxtsxOrbOJLzmehHiTgjMiyYFyRU+vPDJd7M8/Z9IpEHrx0Zaktr0GEe0B+O58lStfYuRqo098IccZIa38H3I8R+M4SuTKQ+tkoBhCHaWVpfxPE3aNj6tU/hKHFE6ddwExKOX0n9To/eJ+ekyTWqPQemWbAwtD0MenJWKg3oKZ+ItUEP82mDLBJ2SZmT/zf8hxJwKt+F4hwaoNWPu+f+uQj/4j3llCTXNI2blan53a8oHA6rw2/Pq45FqOXZ5FXZdaV3Y+jpqVYhU0h/1D1M0tnEg6puEBd798MOyXMgDyjkr4uIzctauE73hvK7jU8YvjvFpWYukLeI290I7isU7Vw73ehmJyNOglrp6bQmdOggSG8I9zz3bcKWh8TvEN8Pwnq3NY5Ve8m47jKI4hPMX52JBga+B6WUnnGgo7hfkz3z+g53Km3KTcmpoLBn7rOsV+Wuo8bX64akcRwTtL9DN/KhrIMr3Apb1WCFQTioJqp5nQA64mkFTVNnCVBIWu4hsVWZ5Z2HslxkJf8+U+pC/IaFGYJOuKtlajc39XC9YyQ0yH0o7AJBlxSVYdx/8gqrj3OBliS6w35c1G1JYKkDhRjFooG+2omknXam4RN9Slo6Jrc30a58Ve9PnYyCxek6RcpxofiF3e2HDM5ZR7Tf4IPbNr/Ia5XA11JL7avpVd2F+UFRQedMLQEfdguIY3WqMmQ5nEFGNqdf53Aw3Xip+84IZVvmuKxJHeUgyiTvvMAuN7bxg8NMK5rihzIBo/u8N1oz3GVhxzaF3YtTLEMfLRJEXjFWUhtO04706kwy+DmcS X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: f2eed925-e01c-4acc-321b-08dbf680c657 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:34.9283 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: irsCvGpH08+cNMvhsKv7oJZv2/jZOh3MlqcRvHL4WlCnPdLHed2thSrznuzaOMJo X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092853_848464_C496CA36 X-CRM114-Status: GOOD ( 21.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This removes all the notifier de-duplication logic in the driver and relies on the core code to de-duplicate and allocate only one SVA domain per mm per smmu instance. This naturally gives a 1:1 relationship between SVA domain and mmu notifier. Remove all of the previous mmu_notifier, bond, shared cd, and cd refcount logic entirely. For the purpose of organizing patches lightly remove BTM support. The next patches will add it back in. BTM is a performance optimization so this is bisection friendly functionally invisible change. Note that the current code never even enables ARM_SMMU_FEAT_BTM so none of this code is ever even used. The bond/shared_cd/btm/asid allocator are tightly wound together and changing them all at once would make this patch too big. The core issue is that having a single SVA domain per-smmu instance conflicts with the design of having a global ASID table that BTM currently needs, as we would end up having to assign multiple SVA domains to the same ASID. Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 385 ++++-------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 80 +--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 14 +- 3 files changed, 101 insertions(+), 378 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 991daffbee31aa..30cdd0661c00da 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -13,29 +13,9 @@ #include "../../iommu-sva.h" #include "../../io-pgtable-arm.h" -struct arm_smmu_mmu_notifier { - struct mmu_notifier mn; - struct arm_smmu_ctx_desc *cd; - bool cleared; - refcount_t refs; - struct list_head list; - struct arm_smmu_domain *domain; -}; - -#define mn_to_smmu(mn) container_of(mn, struct arm_smmu_mmu_notifier, mn) - -struct arm_smmu_bond { - struct mm_struct *mm; - struct arm_smmu_mmu_notifier *smmu_mn; - struct list_head list; -}; - -#define sva_to_bond(handle) \ - container_of(handle, struct arm_smmu_bond, sva) - static DEFINE_MUTEX(sva_lock); -static void +static void __maybe_unused arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) { struct arm_smmu_master_domain *master_domain; @@ -58,58 +38,6 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); } -/* - * Check if the CPU ASID is available on the SMMU side. If a private context - * descriptor is using it, try to replace it. - */ -static struct arm_smmu_ctx_desc * -arm_smmu_share_asid(struct mm_struct *mm, u16 asid) -{ - int ret; - u32 new_asid; - struct arm_smmu_ctx_desc *cd; - struct arm_smmu_device *smmu; - struct arm_smmu_domain *smmu_domain; - - cd = xa_load(&arm_smmu_asid_xa, asid); - if (!cd) - return NULL; - - if (cd->mm) { - if (WARN_ON(cd->mm != mm)) - return ERR_PTR(-EINVAL); - /* All devices bound to this mm use the same cd struct. */ - refcount_inc(&cd->refs); - return cd; - } - - smmu_domain = container_of(cd, struct arm_smmu_domain, cd); - smmu = smmu_domain->smmu; - - ret = xa_alloc(&arm_smmu_asid_xa, &new_asid, cd, - XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); - if (ret) - return ERR_PTR(-ENOSPC); - /* - * Race with unmap: TLB invalidations will start targeting the new ASID, - * which isn't assigned yet. We'll do an invalidate-all on the old ASID - * later, so it doesn't matter. - */ - cd->asid = new_asid; - /* - * Update ASID and invalidate CD in all associated masters. There will - * be some overlap between use of both ASIDs, until we invalidate the - * TLB. - */ - arm_smmu_update_s1_domain_cd_entry(smmu_domain); - - /* Invalidate TLB entries previously associated with that context */ - arm_smmu_tlb_inv_asid(smmu, asid); - - xa_erase(&arm_smmu_asid_xa, asid); - return NULL; -} - static u64 page_size_to_cd(void) { static_assert(PAGE_SIZE == SZ_4K || PAGE_SIZE == SZ_16K || @@ -123,7 +51,8 @@ static u64 page_size_to_cd(void) static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, struct arm_smmu_master *master, - struct mm_struct *mm, u16 asid) + struct mm_struct *mm, u16 asid, + bool btm_invalidation) { u64 par; @@ -144,7 +73,7 @@ static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, (master->stall_enabled ? CTXDESC_CD_0_S : 0) | CTXDESC_CD_0_R | CTXDESC_CD_0_A | - CTXDESC_CD_0_ASET | + (btm_invalidation ? 0 : CTXDESC_CD_0_ASET) | FIELD_PREP(CTXDESC_CD_0_ASID, asid)); /* @@ -176,69 +105,6 @@ static void arm_smmu_make_sva_cd(struct arm_smmu_cd *target, target->data[3] = cpu_to_le64(read_sysreg(mair_el1)); } -static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) -{ - u16 asid; - int err = 0; - struct arm_smmu_ctx_desc *cd; - struct arm_smmu_ctx_desc *ret = NULL; - - /* Don't free the mm until we release the ASID */ - mmgrab(mm); - - asid = arm64_mm_context_get(mm); - if (!asid) { - err = -ESRCH; - goto out_drop_mm; - } - - cd = kzalloc(sizeof(*cd), GFP_KERNEL); - if (!cd) { - err = -ENOMEM; - goto out_put_context; - } - - refcount_set(&cd->refs, 1); - - mutex_lock(&arm_smmu_asid_lock); - ret = arm_smmu_share_asid(mm, asid); - if (ret) { - mutex_unlock(&arm_smmu_asid_lock); - goto out_free_cd; - } - - err = xa_insert(&arm_smmu_asid_xa, asid, cd, GFP_KERNEL); - mutex_unlock(&arm_smmu_asid_lock); - - if (err) - goto out_free_asid; - - cd->asid = asid; - cd->mm = mm; - - return cd; - -out_free_asid: - arm_smmu_free_asid(cd); -out_free_cd: - kfree(cd); -out_put_context: - arm64_mm_context_put(mm); -out_drop_mm: - mmdrop(mm); - return err < 0 ? ERR_PTR(err) : ret; -} - -static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd) -{ - if (arm_smmu_free_asid(cd)) { - /* Unpin ASID */ - arm64_mm_context_put(cd->mm); - mmdrop(cd->mm); - kfree(cd); - } -} - /* * Cloned from the MAX_TLBI_OPS in arch/arm64/include/asm/tlbflush.h, this * is used as a threshold to replace per-page TLBI commands to issue in the @@ -253,8 +119,8 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, unsigned long start, unsigned long end) { - struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); - struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + struct arm_smmu_domain *smmu_domain = + container_of(mn, struct arm_smmu_domain, mmu_notifier); size_t size; /* @@ -271,33 +137,27 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, size = 0; } - if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM)) { + if (!smmu_domain->btm_invalidation) { if (!size) arm_smmu_tlb_inv_asid(smmu_domain->smmu, - smmu_mn->cd->asid); + smmu_domain->cd.asid); else arm_smmu_tlb_inv_range_asid(start, size, - smmu_mn->cd->asid, + smmu_domain->cd.asid, PAGE_SIZE, false, smmu_domain); } - arm_smmu_atc_inv_domain_sva(smmu_domain, mm->pasid, start, size); + arm_smmu_atc_inv_domain(smmu_domain, start, size); } static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { - struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); - struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + struct arm_smmu_domain *smmu_domain = + container_of(mn, struct arm_smmu_domain, mmu_notifier); struct arm_smmu_master_domain *master_domain; unsigned long flags; - mutex_lock(&sva_lock); - if (smmu_mn->cleared) { - mutex_unlock(&sva_lock); - return; - } - /* * DMA may still be running. Keep the cd valid to avoid C_BAD_CD events, * but disable translation. @@ -309,24 +169,27 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) struct arm_smmu_cd target; struct arm_smmu_cd *cdptr; - cdptr = arm_smmu_get_cd_ptr(master, mm->pasid); + cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid); if (WARN_ON(!cdptr)) continue; - arm_smmu_make_sva_cd(&target, master, NULL, smmu_mn->cd->asid); - arm_smmu_write_cd_entry(master, mm->pasid, cdptr, &target); + arm_smmu_make_sva_cd(&target, master, NULL, + smmu_domain->cd.asid, + smmu_domain->btm_invalidation); + arm_smmu_write_cd_entry(master, master_domain->ssid, cdptr, + &target); } spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); - arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); - arm_smmu_atc_inv_domain_sva(smmu_domain, mm->pasid, 0, 0); - - smmu_mn->cleared = true; - mutex_unlock(&sva_lock); + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid); + arm_smmu_atc_inv_domain(smmu_domain, 0, 0); } static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn) { - kfree(mn_to_smmu(mn)); + struct arm_smmu_domain *smmu_domain = + container_of(mn, struct arm_smmu_domain, mmu_notifier); + + kfree(smmu_domain); } static const struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = { @@ -335,109 +198,6 @@ static const struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = { .free_notifier = arm_smmu_mmu_notifier_free, }; -/* Allocate or get existing MMU notifier for this {domain, mm} pair */ -static struct arm_smmu_mmu_notifier * -arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain, - struct mm_struct *mm) -{ - int ret; - struct arm_smmu_ctx_desc *cd; - struct arm_smmu_mmu_notifier *smmu_mn; - - list_for_each_entry(smmu_mn, &smmu_domain->mmu_notifiers, list) { - if (smmu_mn->mn.mm == mm) { - refcount_inc(&smmu_mn->refs); - return smmu_mn; - } - } - - cd = arm_smmu_alloc_shared_cd(mm); - if (IS_ERR(cd)) - return ERR_CAST(cd); - - smmu_mn = kzalloc(sizeof(*smmu_mn), GFP_KERNEL); - if (!smmu_mn) { - ret = -ENOMEM; - goto err_free_cd; - } - - refcount_set(&smmu_mn->refs, 1); - smmu_mn->cd = cd; - smmu_mn->domain = smmu_domain; - smmu_mn->mn.ops = &arm_smmu_mmu_notifier_ops; - - ret = mmu_notifier_register(&smmu_mn->mn, mm); - if (ret) { - kfree(smmu_mn); - goto err_free_cd; - } - - list_add(&smmu_mn->list, &smmu_domain->mmu_notifiers); - return smmu_mn; - -err_free_cd: - arm_smmu_free_shared_cd(cd); - return ERR_PTR(ret); -} - -static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) -{ - struct mm_struct *mm = smmu_mn->mn.mm; - struct arm_smmu_ctx_desc *cd = smmu_mn->cd; - struct arm_smmu_domain *smmu_domain = smmu_mn->domain; - - if (!refcount_dec_and_test(&smmu_mn->refs)) - return; - - list_del(&smmu_mn->list); - - /* - * If we went through clear(), we've already invalidated, and no - * new TLB entry can have been formed. - */ - if (!smmu_mn->cleared) { - arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid); - arm_smmu_atc_inv_domain_sva(smmu_domain, mm->pasid, 0, 0); - } - - /* Frees smmu_mn */ - mmu_notifier_put(&smmu_mn->mn); - arm_smmu_free_shared_cd(cd); -} - -static int __arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, - struct arm_smmu_cd *target) -{ - int ret; - struct arm_smmu_bond *bond; - struct arm_smmu_master *master = dev_iommu_priv_get(dev); - struct arm_smmu_domain *smmu_domain = - to_smmu_domain_safe(iommu_get_domain_for_dev(dev)); - - if (!smmu_domain || smmu_domain->stage != ARM_SMMU_DOMAIN_S1) - return -ENODEV; - - bond = kzalloc(sizeof(*bond), GFP_KERNEL); - if (!bond) - return -ENOMEM; - - bond->mm = mm; - - bond->smmu_mn = arm_smmu_mmu_notifier_get(smmu_domain, mm); - if (IS_ERR(bond->smmu_mn)) { - ret = PTR_ERR(bond->smmu_mn); - goto err_free_bond; - } - - list_add(&bond->list, &master->bonds); - arm_smmu_make_sva_cd(target, master, mm, bond->smmu_mn->cd->asid); - return 0; - -err_free_bond: - kfree(bond); - return ret; -} - bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) { unsigned long reg, fld; @@ -565,11 +325,6 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master) int arm_smmu_master_disable_sva(struct arm_smmu_master *master) { mutex_lock(&sva_lock); - if (!list_empty(&master->bonds)) { - dev_err(master->dev, "cannot disable SVA, device is bound\n"); - mutex_unlock(&sva_lock); - return -EBUSY; - } arm_smmu_master_sva_disable_iopf(master); master->sva_enabled = false; mutex_unlock(&sva_lock); @@ -586,59 +341,54 @@ void arm_smmu_sva_notifier_synchronize(void) mmu_notifier_synchronize(); } -void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, - struct device *dev, ioasid_t id) -{ - struct mm_struct *mm = domain->mm; - struct arm_smmu_bond *bond = NULL, *t; - struct arm_smmu_master *master = dev_iommu_priv_get(dev); - - arm_smmu_remove_pasid(master, to_smmu_domain(domain), id); - - mutex_lock(&sva_lock); - list_for_each_entry(t, &master->bonds, list) { - if (t->mm == mm) { - bond = t; - break; - } - } - - if (!WARN_ON(!bond)) { - list_del(&bond->list); - arm_smmu_mmu_notifier_put(bond->smmu_mn); - kfree(bond); - } - mutex_unlock(&sva_lock); -} - static int arm_smmu_sva_set_dev_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t id) { + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); struct arm_smmu_master *master = dev_iommu_priv_get(dev); - int ret = 0; - struct mm_struct *mm = domain->mm; struct arm_smmu_cd target; + int ret; - if (mm->pasid != id || !master->cd_table.used_sid) + /* Prevent arm_smmu_mm_release from being called while we are attaching */ + if (!mmget_not_zero(domain->mm)) return -EINVAL; - if (!arm_smmu_get_cd_ptr(master, id)) - return -ENOMEM; + /* + * This does not need the arm_smmu_asid_lock because SVA domains never + * get reassigned + */ + arm_smmu_make_sva_cd(&target, master, smmu_domain->domain.mm, + smmu_domain->cd.asid, + smmu_domain->btm_invalidation); - mutex_lock(&sva_lock); - ret = __arm_smmu_sva_bind(dev, mm, &target); - mutex_unlock(&sva_lock); - if (ret) - return ret; + ret = arm_smmu_set_pasid(master, to_smmu_domain(domain), id, &target); - /* This cannot fail since we preallocated the cdptr */ - arm_smmu_set_pasid(master, to_smmu_domain(domain), id, &target); - return 0; + mmput(domain->mm); + return ret; } static void arm_smmu_sva_domain_free(struct iommu_domain *domain) { - kfree(to_smmu_domain(domain)); + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + /* + * Ensure the ASID is empty in the iommu cache before allowing reuse. + */ + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid); + + /* + * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can + * still be called/running at this point. We allow the ASID to be + * reused, and if there is a race then it just suffers harmless + * unnecessary invalidation. + */ + xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); + + /* + * Actual free is defered to the SRCU callback + * arm_smmu_mmu_notifier_free() + */ + mmu_notifier_put(&smmu_domain->mmu_notifier); } static const struct iommu_domain_ops arm_smmu_sva_domain_ops = { @@ -652,6 +402,8 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, struct arm_smmu_master *master = dev_iommu_priv_get(dev); struct arm_smmu_device *smmu = master->smmu; struct arm_smmu_domain *smmu_domain; + u32 asid; + int ret; smmu_domain = arm_smmu_domain_alloc(); if (!smmu_domain) @@ -661,5 +413,22 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, smmu_domain->domain.ops = &arm_smmu_sva_domain_ops; smmu_domain->smmu = smmu; + ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain, + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); + if (ret) + goto err_free; + + smmu_domain->cd.asid = asid; + smmu_domain->mmu_notifier.ops = &arm_smmu_mmu_notifier_ops; + ret = mmu_notifier_register(&smmu_domain->mmu_notifier, mm); + if (ret) + goto err_asid; + return &smmu_domain->domain; + +err_asid: + xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); +err_free: + kfree(smmu_domain); + return ERR_PTR(ret); } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index d09505acd4f8ce..c87e3f197e0fda 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1339,22 +1339,6 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_master *master) cd_table->cdtab = NULL; } -bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd) -{ - bool free; - struct arm_smmu_ctx_desc *old_cd; - - if (!cd->asid) - return false; - - free = refcount_dec_and_test(&cd->refs); - if (free) { - old_cd = xa_erase(&arm_smmu_asid_xa, cd->asid); - WARN_ON(old_cd != cd); - } - return free; -} - /* Stream table manipulation functions */ static void arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc) @@ -1979,8 +1963,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master, return arm_smmu_cmdq_batch_submit(master->smmu, &cmds); } -static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, - ioasid_t ssid, unsigned long iova, size_t size) +int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, + unsigned long iova, size_t size) { struct arm_smmu_master_domain *master_domain; int i; @@ -2018,15 +2002,7 @@ static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, if (!master->ats_enabled) continue; - /* - * Non-zero ssid means SVA is co-opting the S1 domain to issue - * invalidations for SVA PASIDs. - */ - if (ssid != IOMMU_NO_PASID) - arm_smmu_atc_inv_to_cmd(ssid, iova, size, &cmd); - else - arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size, - &cmd); + arm_smmu_atc_inv_to_cmd(master_domain->ssid, iova, size, &cmd); for (i = 0; i < master->num_streams; i++) { cmd.atc.sid = master->streams[i].id; @@ -2038,19 +2014,6 @@ static int __arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, return arm_smmu_cmdq_batch_submit(smmu_domain->smmu, &cmds); } -static int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, - unsigned long iova, size_t size) -{ - return __arm_smmu_atc_inv_domain(smmu_domain, IOMMU_NO_PASID, iova, - size); -} - -int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain, - ioasid_t ssid, unsigned long iova, size_t size) -{ - return __arm_smmu_atc_inv_domain(smmu_domain, ssid, iova, size); -} - /* IO_PGTABLE API */ static void arm_smmu_tlb_inv_context(void *cookie) { @@ -2239,7 +2202,6 @@ struct arm_smmu_domain *arm_smmu_domain_alloc(void) mutex_init(&smmu_domain->init_mutex); INIT_LIST_HEAD(&smmu_domain->devices); spin_lock_init(&smmu_domain->devices_lock); - INIT_LIST_HEAD(&smmu_domain->mmu_notifiers); return smmu_domain; } @@ -2280,7 +2242,7 @@ static void arm_smmu_domain_free(struct iommu_domain *domain) if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { /* Prevent SVA from touching the CD while we're freeing it */ mutex_lock(&arm_smmu_asid_lock); - arm_smmu_free_asid(&smmu_domain->cd); + xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); mutex_unlock(&arm_smmu_asid_lock); } else { struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg; @@ -2298,11 +2260,9 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu, u32 asid; struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; - refcount_set(&cd->refs, 1); - /* Prevent SVA from modifying the ASID until it is written to the CD */ mutex_lock(&arm_smmu_asid_lock); - ret = xa_alloc(&arm_smmu_asid_xa, &asid, cd, + ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain, XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); cd->asid = (u16)asid; mutex_unlock(&arm_smmu_asid_lock); @@ -2715,7 +2675,10 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, struct attach_state state; int ret; - if (!sid_smmu_domain || sid_smmu_domain->stage != ARM_SMMU_DOMAIN_S1) + if (smmu_domain->smmu != master->smmu) + return -EINVAL; + + if (!sid_smmu_domain || !master->cd_table.used_sid) return -ENODEV; cdptr = arm_smmu_get_cd_ptr(master, pasid); @@ -2736,9 +2699,18 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, return ret; } -void arm_smmu_remove_pasid(struct arm_smmu_master *master, - struct arm_smmu_domain *smmu_domain, ioasid_t pasid) +static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) { + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct arm_smmu_domain *smmu_domain; + struct iommu_domain *domain; + + domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA); + if (WARN_ON(IS_ERR(domain)) || !domain) + return; + + smmu_domain = to_smmu_domain(domain); + mutex_lock(&arm_smmu_asid_lock); arm_smmu_clear_cd(master, pasid); if (master->ats_enabled) @@ -3032,7 +3004,6 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) master->dev = dev; master->smmu = smmu; - INIT_LIST_HEAD(&master->bonds); dev_iommu_priv_set(dev, master); ret = arm_smmu_insert_master(smmu, master); @@ -3214,17 +3185,6 @@ static int arm_smmu_def_domain_type(struct device *dev) return 0; } -static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) -{ - struct iommu_domain *domain; - - domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA); - if (WARN_ON(IS_ERR(domain)) || !domain) - return; - - arm_smmu_sva_remove_dev_pasid(domain, dev, pasid); -} - static struct iommu_ops arm_smmu_ops = { .identity_domain = &arm_smmu_identity_domain, .blocked_domain = &arm_smmu_blocked_domain, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 48871c8ee8c88c..a229ad0adf6a49 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -587,9 +587,6 @@ struct arm_smmu_strtab_l1_desc { struct arm_smmu_ctx_desc { u16 asid; - - refcount_t refs; - struct mm_struct *mm; }; struct arm_smmu_l1_ctx_desc { @@ -713,7 +710,6 @@ struct arm_smmu_master { bool stall_enabled; bool sva_enabled; bool iopf_enabled; - struct list_head bonds; unsigned int ssid_bits; }; @@ -742,7 +738,8 @@ struct arm_smmu_domain { struct list_head devices; spinlock_t devices_lock; - struct list_head mmu_notifiers; + struct mmu_notifier mmu_notifier; + bool btm_invalidation; }; struct arm_smmu_master_domain { @@ -796,9 +793,8 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, size_t granule, bool leaf, struct arm_smmu_domain *smmu_domain); -bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd); -int arm_smmu_atc_inv_domain_sva(struct arm_smmu_domain *smmu_domain, - ioasid_t ssid, unsigned long iova, size_t size); +int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, + unsigned long iova, size_t size); #ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported(struct arm_smmu_device *smmu); @@ -810,8 +806,6 @@ bool arm_smmu_master_iopf_supported(struct arm_smmu_master *master); void arm_smmu_sva_notifier_synchronize(void); struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, struct mm_struct *mm); -void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, - struct device *dev, ioasid_t id); #else /* CONFIG_ARM_SMMU_V3_SVA */ static inline bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) { From patchwork Wed Dec 6 17:28:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482149 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 50B9CC10DC1 for ; Wed, 6 Dec 2023 18:31:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GSERgaeOEy9JDSvB/zZUEU11hT1n5LecjXZiwA7BzP0=; b=1e7sUrOyLnt6E8 RreTIb/VzC1N43qEi+UhMv7n/1nA2Ocny/rNANU33qiqKlbGTjWBbUQr9wZGIFBJKCT2uvVepsauI rsXz/AI0ivlZK6x9aDNFMm4u23TEfypRj8PV/0cEjd7wZYNb4tzbDsoxjOl48doznwWAA9nWoFooY rkMlMatfJpqETEzEbKbfvIoQKe7KX3yrbQuuCkMbObSiJidctR/Ace0kJ21hzE9DLJw95EcZufuUa hBGrCD06mrT46s5DVjlJmt2IE0Sd1zbMNPFCSUHLR1KoZnOwtLn0+KOiODloHguhyHqdMzUNTfXTz gblW5j9onhJ087s75hXw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAwfQ-00B2Bm-2A; Wed, 06 Dec 2023 18:30:40 +0000 Received: from mail-mw2nam12on2060f.outbound.protection.outlook.com ([2a01:111:f400:fe5a::60f] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvho-00AtP3-1D for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:29:15 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=l7aa7YlpkD/tieJGt+nEXAcFys8Zs/VwKTvnKt3OdarSk7+l/8cA0WGchYOb+ugTbM4R0Yt5WIep6saLa0tKhW2+FSKH8Ypsy6/jmMOMRq9UlFFk7kDW6P4RfPCsbvHSjlV1pjZ5sT5PZvspX2FImVEbRMKsQ1LoCzZvIGLriWGn1oY5S0wMc821djy7YNfQX8TdnRzuPtTChae3D9ABvW5Xv6R1Wr6RiU1VLTIYr62UPdT683kRxLzp6hi77+k+2imj+AKE4RoqPPS3CEzNCB6pKBstpnFYId+Niv2HIijlsS8f+KqbsIaGA8MTGGX+Xn8YLGYL1wUMA33fXb/Xdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BFa0jDCQ7EnqwSERTTcTJj48bgdBwAAyV8J/yQgvDL4=; b=ZBRyspauDb3iRQBcI6zmvK4LMbO0TvH3Yd6XDG3fdIlHVXVO3gdlOj539vDlqOdZZmT7wXRfsm9q5/TVaA3ODp/SCYG/Fc5YI9SWwJ0pcn5KV2kf6UurTzhv3rxLC53QG8cru/r2P7eyM2G8WwT01xG7Rusac2qaXNq3cFUa43ZN1kXEUHJ9zdjTyTi+r3SFlXh7T5YTeoI+WBJnivqjWMyhC+fl/IpDrVF0eQMXJOooFggGuU76xWVM9Rg15aRV756oZ0LhGc3C5AbIz6eWHxLA0GRYIx+MMCiNreBxtCna1svivu3Qf2ZjZQ0M+FJCGvLed8MWUdz8Azyqo7aVjg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BFa0jDCQ7EnqwSERTTcTJj48bgdBwAAyV8J/yQgvDL4=; b=JUCoZWgQwoeDUzk9XuP+xFg0g9cBCHoFgCBOTrcty4qX/wiMrZVQcc1zdu7dzkmvxrDCUoNGTLwv6wZBt3AuuvXLvk4oArQ/CkJ13DVoFaflCImhL6gxHqobHNSVCOaILMuVLOk7//zX0FmCQces4HYoMfpNx5Msg4FlR3qMFWpFq8Zt0b6DDygp3Zl+GxpVddZbuzXPSXHYolSAldUSYnfTakGj91WZoWOxqKhv0WCFhMElFDYhpEPPKYXKWg8A/6squjRZCIzLJXnP78Eh1zB0Zx/cN+373bVfqD8tj3PPbU1q1IBeHU/td0ehk7qG7ghmHaPBeqcizoT3iJMAmg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:44 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:44 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 22/27] iommu/arm-smmu-v3: Consolidate freeing the ASID/VMID Date: Wed, 6 Dec 2023 13:28:28 -0400 Message-ID: <22-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR0102CA0039.prod.exchangelabs.com (2603:10b6:208:25::16) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: 29108c28-f026-45d0-4fe0-08dbf680c737 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: I+87ny2UYmga1s2SRcm5MqYg+SMOXgkjD91FGDUZzUv9F4ownjTweVo6o56AZxRtJnIfKRamMltfqc0mpetMm6MQ/KMubTppxDEjHeLP7ckpykVNihw4gtnRCp/PgpVpb2M0tFN5UWWEMM5UvdGSxCOKKg0pTWyf1Dt4OwLilKxTGd3QFun9X7QxVWKkSNImWeJnBooYXvWUSyhNTQ3VkehAUZQ8YNHpeMH0X/5ZVSeER+Ne8aOKgE6z6wvu5uen8mCX7Tl+fYOf+wTG1ybHadcox3XuW3GHT9BtN/7+8beGyAuE6pR5POjl1DE2N1Rmr5NEXloizhaV9Bcs4M3m0Uh7pbND+idf2lXKTXrjDZd939O4S8rJzadkIQt20mWteLgS3zE8e2oZQxGPSs512f1p5O3757yEGMsfqIYi//LgjS2qpyGzL1zJaogIx7hbtJ2O+++DGNnABOPVy8pvK+y5P50yglGIxNkiTmZEbkXxAmSmhZ0ugA9PRtEO2+CWl5IyYMb34AzqU5Fupn5Vands0bTe7jp13C3eR7wMmqIhTlrQ5l/HNgRN1o9jGqL4 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(4326008)(54906003)(7416002)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: +SH81RBpk8UUEXpz2eqggNDvPnB28cPX5jpYBcttWB+UeLVuAszmIATG0igI0j9PBNXmGd3vo82EbN+8T4lJobhrEWlZ07Fdt1dN8Dl/VREpUgNzl8PBb5rrReuiQyy3rnnTtu+dV3fYn5aPIvhLnaGn5vEZcEvmYrVWYoOBB8hmHIfBziAfE/Mq61D1oGvkQVR0AiZ+EKf/e1vX+rFQ9RraMXSQS6C67hnaq/Onspjo3naShPjhyqy0vTmvB1JMWmqwHX+hY8+ZbzA7WYXaBUeYiXVX214XIp3duLK9Rtwc/vJJrfoylmsc2sxFG/sJQAZ/EaVf1pXNJThlbxLgNi7EP9lzSoYtCggEe402Q2C4Vilsj013siWBYcfgMUYgkHJWZWMmdzyvTCfDeobVDv5MQ/ESfo6Y0kTUpXLjcFmz8ANBRL0ti0hA32dSFl/guTGujwH0xfvTS8cjYJFwbrD4ET5E4FNBkjkYS/gnL2NuWBd5T1FbDxyAeLn6/5hRu395b2SgAy110foaumXmIZ+geluFT8jKdKn8+npkHOpbPfOKTVRd+IMe2tSPUWoQCH8KQTEPH1HYeiTiFQqOUlWZZ9Jce8Av/i6868+jKP7wKDlZ1zZDUolIcGuCViS3KoSkel4A2slNPzwD3bWtKxCzrp8U6L0unfp7jhYJRJrrk1Xuh3o/CozU8tA5NsiMHI/ti9Op028KLRyRU7Y8gsl8CsboUFhxJ6g/eyxH/1RoLKXjRA+2uU07sVe8soVwA8CgpT5W6POLBK6rKlNxp16/VYTLL24qAkPi6zMYULbeBF+xbAG/bryOkkaoubHdfsBPg6OiqhIzVPWVNsVNoXac1o9QJlDwixN4k++HIzxjL3sAULlWcaqsZ2x/e4Z5Cq0t0CY5g8WK9vKqvA+gkmr8wX7Q4BNzBR+ZibkISdKyrHc2CQhA0ToiBKG5pg54qjgyiNpP5iBPAnjS6HQvlzu6qoLAli8ej8W77oYd7vF90SyQehH9lf7hEjakQ504+FJWeQsJcH/YMbpsV9leDrG1cl5BV3sCsbfj8sAnI9w8dznWpSh+NNIipGPYjhyNHcxkeqOfXO7z5ioXPxf8yi2nZSGiL8hB5bj71AjWALdAff3vlLxiAddZLQIjPmkfbiMqnD39CIxsXcHV4foBrmXEfEpnBuYufc2UYix+NUXjgtnQ28GM2SdMX5oOHvVLu3AUNiO+nNzZKdkWz+/Nic4lPdSMqIO/6U6AtaAkYPru/Wyf/HOoMxgjiPC9RiUY5yG5ZqPEDMMoS14vUq6TPacybhdSMDbXkoL3qmk/GIh+txX7DIKnVLniTFtNY5NDp69yC8C5pZz195VoR5CAbcbHU27btQDZ8nbISTxgQr9fSvlY45HXZyD093wXl9au22h8mbTzctMogQPFYnYthiuayzwbn/eSAdNTLlbfSaB+luAPvjrkvsJX6Zjb91rdraJIqixBekfhuwW43SL5wBHW0etYsHn6032W+Y3HGkC45EFy3RtPOH4DQewhMJIGDy1+sc9eglCXxoTDNEBEtCKPiigGDKKlf7t6RAJB+nOZxduQ+HMgFOfnkt4gzPxM X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 29108c28-f026-45d0-4fe0-08dbf680c737 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:36.2999 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jrgF4tO/WbnxTwjlCUGvszwU/hIsXYS4/pduGvtXt1HVy5IynGd3KWjluPLx19Sn X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092904_454197_184DDD67 X-CRM114-Status: GOOD ( 20.21 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The SMMUv3 IOTLB is tagged with a VMID/ASID cache tag. Any time the underlying translation is changed these need to be invalidated. At boot time the IOTLB starts out empty and all cache tags are available for allocation. When a tag is taken out of the allocator the code assumes the IOTLB doesn't reference it, and immediately programs it into a STE/CD. If the cache is referencing the tag then it will have stale data and IOMMU will become incoherent. Thus, whenever an ASID/VMID is freed back to the allocator we need to know that the IOTLB doesn't have any references to it. The SVA code correctly had an invalidation here, but the paging code does not. Consolidate freeing the VMID/ASID to one place and consistently flush both ID types before returning to their allocators. Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 9 ++--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 36 +++++++++++++------ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + 3 files changed, 29 insertions(+), 17 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 30cdd0661c00da..19b4d52be7ab66 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -371,18 +371,13 @@ static void arm_smmu_sva_domain_free(struct iommu_domain *domain) { struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); - /* - * Ensure the ASID is empty in the iommu cache before allowing reuse. - */ - arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid); - /* * Notice that the arm_smmu_mm_arch_invalidate_secondary_tlbs op can * still be called/running at this point. We allow the ASID to be * reused, and if there is a race then it just suffers harmless * unnecessary invalidation. */ - xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); + arm_smmu_domain_free_id(smmu_domain); /* * Actual free is defered to the SRCU callback @@ -427,7 +422,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, return &smmu_domain->domain; err_asid: - xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); + arm_smmu_domain_free_id(smmu_domain); err_free: kfree(smmu_domain); return ERR_PTR(ret); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index c87e3f197e0fda..83e31177215533 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2231,25 +2231,41 @@ static struct iommu_domain *arm_smmu_domain_alloc_paging(struct device *dev) return &smmu_domain->domain; } -static void arm_smmu_domain_free(struct iommu_domain *domain) +/* + * Return the domain's ASID or VMID back to the allocator. All IDs in the + * allocator do not have an IOTLB entries referencing them. + */ +void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain) { - struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); struct arm_smmu_device *smmu = smmu_domain->smmu; - free_io_pgtable_ops(smmu_domain->pgtbl_ops); + if ((smmu_domain->stage == ARM_SMMU_DOMAIN_S1 || + smmu_domain->domain.type == IOMMU_DOMAIN_SVA) && + smmu_domain->cd.asid) { + arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid); - /* Free the ASID or VMID */ - if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { /* Prevent SVA from touching the CD while we're freeing it */ mutex_lock(&arm_smmu_asid_lock); xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); mutex_unlock(&arm_smmu_asid_lock); - } else { - struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg; - if (cfg->vmid) - ida_free(&smmu->vmid_map, cfg->vmid); - } + } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2 && + smmu_domain->s2_cfg.vmid) { + struct arm_smmu_cmdq_ent cmd = { + .opcode = CMDQ_OP_TLBI_S12_VMALL, + .tlbi.vmid = smmu_domain->s2_cfg.vmid + }; + arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd); + ida_free(&smmu->vmid_map, smmu_domain->s2_cfg.vmid); + } +} + +static void arm_smmu_domain_free(struct iommu_domain *domain) +{ + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + free_io_pgtable_ops(smmu_domain->pgtbl_ops); + arm_smmu_domain_free_id(smmu_domain); kfree(smmu_domain); } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index a229ad0adf6a49..c7c4f4fda31297 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -789,6 +789,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, void arm_smmu_remove_pasid(struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain, ioasid_t pasid); +void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain); void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid, size_t granule, bool leaf, From patchwork Wed Dec 6 17:28:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482115 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4135AC10DC1 for ; Wed, 6 Dec 2023 17:30:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/ThV+AsLayneKOetUROlbK13v61cJvNpAINOW2ECFRA=; b=bnzLmAE7auD4d4 BB0Px+KeivtYt/IyQQu3Gh91uFz52g1lOVdNQz+EPgyKzDrNDaMKlbT500ZlryCpKEfAWGRkcM7VV nkTZtvyuo1zVCixCFVlJFcO92Y2Qo2TasxMJnpdKEZL38GwLji5d0TV+pX4xWWu0VzRXOIpx5lpT4 B0vTWD0CjPGrEBLK22ijt85/mpzm3oMewhCoMKB6qMvnnJMdo8z41KdThR8n7kyQQz3emK/RkNJ0Z gWUjAjDgUxZ/kzb6JMV5Q7WpWfX5YZ1M33iiLZmoLSMBc4tL4YngmlCE9i3CrQvQEuW4AHL53bWrV 4dZo7wIjzZN4DjtB8ipA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAviU-00AuDt-0Q; Wed, 06 Dec 2023 17:29:46 +0000 Received: from mail-mw2nam12on2060f.outbound.protection.outlook.com ([2a01:111:f400:fe5a::60f] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhi-00AtP3-1H for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:29:03 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cwqRD0sL8BTSLzlwCQmxMchOnp7D7/atN5PJxF+2QZojOuY0w5phdzvfypCpUeI2Ml3TrDsm91eebEDcvytXjL9WAk3nyHe6LhxTIFXM/RlhKRHaPHgFv0zfxeMlyY1UyQ8VP0g6qlBq2e12Th9pJV2wwmTlMMTwSYg3KmPVq5Rrp1KOYjd5Rpgs9MOnFKs4wtHrz/l+4ab0OVNUOVWNbJMIL5lF01h5va6vPoiMx1fIRP/xZgg03EfQxiRLwYf7syKMFYZUpSuOdG674vG7PjwDaDG9DuRRXxkNQeHnm5arqwov86y1xGTzGqjvT705CIUHDau6IcXTkve9Ige3Jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=deJ8narlj3Slb7yQBSdPfAhHExA6UttlyYQOvNEG6Ec=; b=NoSoPCePKPn93MFc36sTnUpsRgiVmBZSrCg6DwZJiCnIIvoO2mFMhfqp5lo5LHDD4ADiBE4qMQ03SU1uJib85QhkHXcWrepQm/ZZu6lF3SznuV9LqZBfcHxeuftuQjJB/86Fclt0VllRvbVVGSuhLAT2YnCxcnO6F4slHroN2tph4D6rs4caOA6WhHVv8XkLBtQ446+k4SLIf0GAinnzAq++g5WI0FKy24NkVWBD3eDHMLL5NV4Zxxz/CFh4eImAAQOhbg3d17HWZA25YCo+rjnVqyTTfHcvBOz6zObCsOWC0WwQUsQ4P7NmlJ5Qiy5OXuWLTTZBqTBuYg9xqa8rRw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=deJ8narlj3Slb7yQBSdPfAhHExA6UttlyYQOvNEG6Ec=; b=mXa9pKndArj0AGDSrbN2vpSpLwXFgNkcxHxiJKvjLMtBR5zMquc1EOVuQzazZ6Xd01m0AAFpJPmhH1LKRch+o0M7ygcdu/WCsHwHBXYM0wL67AGtA+xZlZrAD+nuiXKIBPmrPpfrDiBhiqHgsfh943DFst03SEGWrbbTnMXhV5tuP4g746dAjJJHJYlt9g5KcUj2RJNTJEUwROE5GtmqOksxKuLwQuXdFaOcliv4Qs2kiWTGkQ3XWebUD8oExM7TjF+kkepHqEd8g82wMw2QOmTW370eGGJByFeukg2iMR+L/7HwgJCVPTgN/IxpOwQYV8GJQL4OAStEHq713b7MVA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:43 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:43 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 23/27] iommu/arm-smmu-v3: Move the arm_smmu_asid_xa to per-smmu like vmid Date: Wed, 6 Dec 2023 13:28:29 -0400 Message-ID: <23-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR0102CA0057.prod.exchangelabs.com (2603:10b6:208:25::34) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: 112b197b-e773-47e9-6801-08dbf680c6ce X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1gOeiETMfDg3YCgkTy2dddeSboN1ymOGJ2xOLwcgsyCzWYYfAPRY7C49suvXVbo14Q4a6fLb6UpwaYO+OBW2UkIJjLWSGrJxoNF2jS9ajetVr4xu5weH9w+93SCcisUT8VFilCmbNRvlRYVTQVlH8LFKIjD+i7vYuc8SRg6fYnWYSKCQ8flH8fpxNhyO9XMEzDRD2kDZTu1wqXPyUrAAkEmud6M8MkmMDniieSC+zvlPqOiiAfNGrg4xmplwPOm+2Fg4hBWccrWTPiY7TNEtsX14bm+9IrURa7NloCTXlOQMqIWSlj6o5v2ny95hRMB9cFfgwksnmNDCSD5A4gr2Q1EDnXIKl4HxZ2h0H5I/j9K77pr01vUIvxuY/bdotJk3SPXQ+sg6JScgTgohwes5afx7R3CyRz33ck7x/1crbSDfwACPWihUm3tClnvrvyCLZHZBClvTNO2E1jVSq68amT9opPeoZR6JKkgBZdyf/VldatLhgCFL2F4U1GtafUe8Zk/hZgdk6yEo/eKmJBfL8meeAA3iAH1xN1K5hpC66cx15gGavMe7uwC5k6AgKWUa X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(4326008)(54906003)(7416002)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: hVK69gVt60dPH0h8tHEJNPwyHPmCJQIN7jeYcC9KphBB0RQC86rrkX7N3E8TWu4Bq/Vh0VMjKYycKuT7z3S6XNlasJnqf+RKw1ieodB4Lq5AmrKPRVs1rxmUQ6I4UtRoKtrBlsRx4udz2przn4RuzbCl9tzEfJW+nLvgxfsak0z8Loqz5HHHpnXmsm8QqRX7fw31cCQDFooIPAFsACwSonkcFa47gQ42Co9Wf70G4P2uaBbCqzR4BjiVRDxWJFcqOIkh+EEtyZfCoNB3KC0PfgrIUM8cmT3ebjx8avopQng10aNjE+pRJpbwgpDBsokbcov+vb5ogZNYOaKHONL8peKP43NxEjijRwY+k9+YGNpbu3IInS5SYpOaahlOk/oDF3JKRH4TuSUMjRMu9XRGkkPTXGvtJTE1ZszId2tBrie5kvQBpNyTM4mA1gZDfu8qgB2MaOThu89JDpiiQFW2HX2fiuicFqcmO88gX96LMG/2cpCv6gatJuTgBXHWlbkX6s+x5XH4mNk5GFK8eFm7Z79/+8eN1gtuR5Z9l3ENqz2drLm1hxiYllyC9MsMRCSQDtmVOcsXfVe1IQhBJPrcUquzqwA0OHJ10eHo75we5qiG6c7YfXJoDpkI3vJYYk0N0RApyX/j1u+PlMVGUqtKI+zqfyLobKeEHU9/FBnAOeVFIC9vFTGwBOuS72qsE04/JIiEO+PxOX/j1ZcH+cDPwSWYGJl2V8x2jXQ/pxvVB5/aNpmSpnZyy7P/wp/jqJ8+sWYgZtio52pm7cc5mRZNpKLy1p+NWsxtAmydIcSraJD8TYVgXihZlZz0hZECd/yGSNi3qrqma/zeFyYXt9DTD7PqauvE706sljq6JQ3kKWEx3hGOg660wCpmOv9d3tND2PQ2dkfmusT9tAo9ZPdfz+tn1RAz4cxJILIUMtrYlXUU3UGwbYA0O1C3+XIzuDNdrpqeoRFjKzJd03t4SMvXT1ZNVxrWq82sqQacGbDwrn0EjrIjdAafebXjznuEHcLsZO7MS3FPIAgjYeLw0pHQR+G3/SrMHZNB6TOyyjxXODxc/YFExbWOuEKO4dUZoV6WeqsfFEryROxca62ymQhtuiyC1MpvbbtM2ggppvPPfiqpYL9sqty/icJNqRTXJ5EanXhwJAzF5rGUgcYgtL6mLCzOuSXwHm+r6AetrRx3mxf/L7ZQ5w7ZRF5oYG/TR+FYAeXWevasxIfonth9ajJkKuPkCjOaicIKEsXSG0wjd8KZTPRRPU3tFotNTKNfbKQWMsQU4B5JDNjeuSZYa/IG7pLjmtMObfcWxaVBB4Nb7J3Souen01SEcSBQZvthDGHeRYJGGxYvNT/AijuS6l1Qb15IZnTOocIxWZouxmtTQPFpU98nYJsFNt1cMcVx9sZ4u/wMuhH2G5Hu4TB4aQ+8yBrH2K10hBhfvCYeKxDNxjDFXYpG8u7JKqqrS6T9W3jR1SeWUhFiY/av0rxiRgcWz++INdIMPkwHvfj7WgahMiYkwKarQ+Tyys/5hZ6rxt6Re7TeSgqHLaWHbdZknTFVDxMo/2wN6kEKV1+N1lKUFLXHfjwi+zkizbay8rAxwf+m X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 112b197b-e773-47e9-6801-08dbf680c6ce X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:35.6859 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: d3JDOiP2wTtAy35xYvVEFRSuqoTH8QPJsVOuOP5mgRVsRDUxJ4nuXnYdzIvBL/Vp X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092858_524241_50A4E190 X-CRM114-Status: GOOD ( 18.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The SVA BTM and shared cd code was the only thing keeping this as a global array. Now that is out of the way we can move it to per-smmu. Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 2 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 39 +++++++++---------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 +-- 3 files changed, 22 insertions(+), 24 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 19b4d52be7ab66..d5772ce27d48b2 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -408,7 +408,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, smmu_domain->domain.ops = &arm_smmu_sva_domain_ops; smmu_domain->smmu = smmu; - ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain, + ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain, XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); if (ret) goto err_free; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 83e31177215533..e30de36044cac5 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -71,9 +71,6 @@ struct arm_smmu_option_prop { const char *prop; }; -DEFINE_XARRAY_ALLOC1(arm_smmu_asid_xa); -DEFINE_MUTEX(arm_smmu_asid_lock); - static struct arm_smmu_option_prop arm_smmu_options[] = { { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" }, { ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"}, @@ -2245,9 +2242,9 @@ void arm_smmu_domain_free_id(struct arm_smmu_domain *smmu_domain) arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid); /* Prevent SVA from touching the CD while we're freeing it */ - mutex_lock(&arm_smmu_asid_lock); - xa_erase(&arm_smmu_asid_xa, smmu_domain->cd.asid); - mutex_unlock(&arm_smmu_asid_lock); + mutex_lock(&smmu->asid_lock); + xa_erase(&smmu->asid_map, smmu_domain->cd.asid); + mutex_unlock(&smmu->asid_lock); } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S2 && smmu_domain->s2_cfg.vmid) { struct arm_smmu_cmdq_ent cmd = { @@ -2277,11 +2274,11 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu, struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; /* Prevent SVA from modifying the ASID until it is written to the CD */ - mutex_lock(&arm_smmu_asid_lock); - ret = xa_alloc(&arm_smmu_asid_xa, &asid, smmu_domain, + mutex_lock(&smmu->asid_lock); + ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain, XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); cd->asid = (u16)asid; - mutex_unlock(&arm_smmu_asid_lock); + mutex_unlock(&smmu->asid_lock); return ret; } @@ -2545,7 +2542,7 @@ static int arm_smmu_attach_prepare(struct arm_smmu_master *master, * arm_smmu_master_domain contents otherwise it could randomly write one * or the other to the CD. */ - lockdep_assert_held(&arm_smmu_asid_lock); + lockdep_assert_held(&master->smmu->asid_lock); master_domain = kzalloc(sizeof(*master_domain), GFP_KERNEL); if (!master_domain) @@ -2584,7 +2581,7 @@ static int arm_smmu_attach_prepare(struct arm_smmu_master *master, static void arm_smmu_attach_commit(struct arm_smmu_master *master, ioasid_t ssid, struct attach_state *state) { - lockdep_assert_held(&arm_smmu_asid_lock); + lockdep_assert_held(&master->smmu->asid_lock); if (!state->want_ats) { WARN_ON(master->ats_enabled); @@ -2647,12 +2644,12 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) * This allows the STE and the smmu_domain->devices list to * be inconsistent during this routine. */ - mutex_lock(&arm_smmu_asid_lock); + mutex_lock(&smmu->asid_lock); ret = arm_smmu_attach_prepare(master, smmu_domain, IOMMU_NO_PASID, &state); if (ret) { - mutex_unlock(&arm_smmu_asid_lock); + mutex_unlock(&smmu->asid_lock); return ret; } @@ -2677,7 +2674,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } arm_smmu_attach_commit(master, IOMMU_NO_PASID, &state); - mutex_unlock(&arm_smmu_asid_lock); + mutex_unlock(&smmu->asid_lock); return 0; } @@ -2701,7 +2698,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, if (!cdptr) return -ENOMEM; - mutex_lock(&arm_smmu_asid_lock); + mutex_lock(&master->smmu->asid_lock); ret = arm_smmu_attach_prepare(master, smmu_domain, pasid, &state); if (ret) goto out_unlock; @@ -2711,7 +2708,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, arm_smmu_attach_commit(master, pasid, &state); out_unlock: - mutex_unlock(&arm_smmu_asid_lock); + mutex_unlock(&master->smmu->asid_lock); return ret; } @@ -2727,12 +2724,12 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) smmu_domain = to_smmu_domain(domain); - mutex_lock(&arm_smmu_asid_lock); + mutex_lock(&master->smmu->asid_lock); arm_smmu_clear_cd(master, pasid); if (master->ats_enabled) arm_smmu_atc_inv_master(master, pasid); arm_smmu_remove_master_domain(master, smmu_domain, pasid); - mutex_unlock(&arm_smmu_asid_lock); + mutex_unlock(&master->smmu->asid_lock); } static int arm_smmu_attach_dev_ste(struct device *dev, @@ -2749,7 +2746,7 @@ static int arm_smmu_attach_dev_ste(struct device *dev, * Do not allow any ASID to be changed while are working on the STE, * otherwise we could miss invalidations. */ - mutex_lock(&arm_smmu_asid_lock); + mutex_lock(&master->smmu->asid_lock); /* * The SMMU does not support enabling ATS with bypass/abort. When the @@ -2778,7 +2775,7 @@ static int arm_smmu_attach_dev_ste(struct device *dev, master->ats_enabled = false; - mutex_unlock(&arm_smmu_asid_lock); + mutex_unlock(&master->smmu->asid_lock); /* * This has to be done after removing the master from the @@ -3436,6 +3433,8 @@ static int arm_smmu_init_strtab(struct arm_smmu_device *smmu) smmu->strtab_cfg.strtab_base = reg; ida_init(&smmu->vmid_map); + xa_init_flags(&smmu->asid_map, XA_FLAGS_ALLOC1); + mutex_init(&smmu->asid_lock); return 0; } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index c7c4f4fda31297..efc6bc11bbb838 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -675,6 +675,8 @@ struct arm_smmu_device { #define ARM_SMMU_MAX_ASIDS (1 << 16) unsigned int asid_bits; + struct xarray asid_map; + struct mutex asid_lock; #define ARM_SMMU_MAX_VMIDS (1 << 16) unsigned int vmid_bits; @@ -768,9 +770,6 @@ to_smmu_domain_safe(struct iommu_domain *domain) return NULL; } -extern struct xarray arm_smmu_asid_xa; -extern struct mutex arm_smmu_asid_lock; - struct arm_smmu_domain *arm_smmu_domain_alloc(void); void arm_smmu_clear_cd(struct arm_smmu_master *master, int ssid); From patchwork Wed Dec 6 17:28:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482113 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AA5F5C4167B for ; Wed, 6 Dec 2023 17:30:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=OsnTaxncMfF5K/20WASV4CRedQxUa8tEtgIFpeYsyYc=; b=yxr4K6q1bGVg24 Y9jIKH8qt/PrO0NA8i0vp+RatH3QIz4nlqjujPHRgrTIDh/BWo1Ht0SCQKZg9COIxQsLLu8iRDJPA C6ynef2xObN+hXUaDIW6ucfWJJRczntXLMAoQaFJpq4n7FKWJn8RTXqEwzBZ09nO1z3jXpZkwwQtY oWCKm6zmVVYAva4kL9LDmyz237l5ZF50R+6j+e1gTJCs2P5ruWzUt9Hllv+iS/q+WwAAKLutmqkf/ ECYr1uCQB+QgwirOUxOg+tZDmyc7pSaHxeBZ+vgrD4s21AtCnrPYpeBosUUcSJEHR32v6nCraUJNB mnzEgcp8a+8j1uIcWSqw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAviX-00AuGC-1D; Wed, 06 Dec 2023 17:29:49 +0000 Received: from mail-mw2nam12on20609.outbound.protection.outlook.com ([2a01:111:f400:fe5a::609] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhl-00AtIF-0k for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:29:09 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Jb/Z70oOmrwwUMKo2USTfL1gO5hlbphwNuSrxHJEW6JAdJqlMAXc+fuOe7TUGU8fGk7ibkk3k6dauqbvl7OD220kF4Kq+A9UZH4gVunC0HSFERPIZVD+1oQepxpCmMVrN2xFTD+FR/j3gAKYsFEB/TTJlNB9dzYjCnwztPgSu5fzV8d5M4ulwVvS6se9thVYygZA6YeP04Eexy4eET+jKavsCfMEdzmdvM6XSQHcNw17vZR9RKEyxcyaSd7vV7FNvL8fXyT8yvqjXKBQCn7i6BrIGR/0toQQZdpFXaoKOwVj1XHe3CaXOiTJeZOXBx+Ys3It3hSIKTX/+ybu47EOlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=71n1HqtL37oKBJQMf5vh6u3uyKKmodyLm/E0Hfw4+MQ=; b=dknM65pf2+lzgn6kHNSY17byZzXriRyBsgq+R+M3HEHGWA3u/h0p+ReLCep21AfEfGiu3ZLZm7cmdultfJrCr4uaq32J56nyPmzXIrknB9Hc+hhjrPJZf2Sby1UoFkpdaxDd0PngsWdxBRLtI8KB0vaVHHD4tNLZA1ACLEp0F3KNGaLsm10eLXK6HFSVZ4FtZwoRtXrTRj8IBfA9zr/VoCS3BZX4yZrlsHG310kzHICfK+ynoPDcI5qLZHEf7TfvUKiwNFebvWwqxpcSBCSbFCQdZmqo5Lhs2fWoPGxnJcF8HeRt65jHjcZOK/CJi/6VBWRumC6NddI+IbM4byYsTw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=71n1HqtL37oKBJQMf5vh6u3uyKKmodyLm/E0Hfw4+MQ=; b=k73kaAj9PKl3APzCo4wu0dqUemBpB4Pfsx6e7mvxh9TzbxEuxJIFqL84t5dfZrk8iws38O7pm2E3sWhz6enekDo9YcxUGAwACU7RqpWdBEn7ZSs3TAvKkzhqLx2tc65O2rTjNS15fsSaFyr7MwSAhLuM5MdlJ6faCwTLUXr1UrNkOhiZHUVkil2DhisRUtsHn1ykuvFQ9Lt6W6CUdJteksVfZM+FyQjp9AkQ/F2zCdkMhBP/pIFThIFfEc6Vv3JbsdO0gYZfqyXXWmO1PkTlf+n01/ymoBw5lF92qDUUMaefb7yE5cZU9dl5eQGOP0jbE9AbMvdeWrIJgfHsqs7Kxw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:44 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:44 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 24/27] iommu/arm-smmu-v3: Bring back SVA BTM support Date: Wed, 6 Dec 2023 13:28:30 -0400 Message-ID: <24-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR22CA0017.namprd22.prod.outlook.com (2603:10b6:208:238::22) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: 25db4a49-20b9-4c4f-c7e4-08dbf680c70e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: nFq77Hm5THDw3+fd+3q/K4azB7bTpoosZ+GXzV6KiZ1KTJghtSTwuweRea6aD/7muE01Txsl4EaAenRe1vp85vtMdS1EvAqyXi7CBuDNsTtDyiYLzAWme5FRRPsFU6YUlgxncqF1BqhRlOvnw1UEDz3L5bSRb5a4USNdW/O1D2IzUGuFuVcRegtc7EfW+nStZfOcwtkKJYurUAgrDgLz0NUyOr51k1DO48SjWT7uTxQ12pBOEoWwdRMGDUX+uz+LcZRALDmSlpexUrBFWmVKdSQopVdPODQRJHQ+EjtVO5g+dFNBHM9uF8JEgjTBisf9i9IH6FJ+0FHFrF79tE4TFJ+cxuDJRWNo0clm2R9ot2VF6wfyBuVHfVNkbi6HdpnFhQiT5HjFeYG9UNeECc7+p53qXWrxjt2kMwiQG6nonCcnWJYs6SsobWSDzsu0VdE93coL+ejpv/hHr0PR75QxjFN3kkLF1qexZBT5iE1bCFHBRDcSqg6uX7rysVTvvO3gmlMjkeWa7fm9bNESv6q+HysyOjudeygzNMsZXoqjBs30828QD6Q+e4YQdXDsrN4qynUyluuy75R8j/QmTMEi/r/ZzaBjeToNwi3PfFojEHixTMoIyt+oNNk6LcMEj4PY X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(230273577357003)(230173577357003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(30864003)(4326008)(54906003)(7416002)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: DRAHc03d65L7zNPq4fCfKGbmQC464lHy3c/xhOZu4Bw1DSVe03RRdppCUYRptEfkhZQC6Bd2xi23rHNbeIqIjEgi3OBglSXe7aW3gwilikAGw13XUsma10omhXVx1LiOjrl9X7+n14Cw0ezikhoWXvbBgZAyjf59uL8iOPezRHNprXJVZOdekdaXiomn/ctpmwv9Bps19V9iHLHiXduNW3zmcRgYkJMYVGLCd5X/pUrJQy+NETtPwTGp59eeo9YntL0lHrzsKEhjVmn50Mu+hH8wYBxq9PL4oO/xcVkY3NvCIOTRDomV8MoPOcuOtF0Iai+Eu0reJmmTXjBLBmhiQ7191hDrus2OwEz1a+TKHDO7zHoN+zbIRrjclnOPIXxB34ZCH6c/Mh8ymmFeCMBGpJ80u/+m6bEJfteZlGFBtQbV6G9acNCcksHFlVju0sJ3TZBZVgVCWGqm2B1m5dWIlvCKuYtjqVo3aE+u6w3jix76kaoAeYwsB3aHi6Mw17BCYKuKwMetU6RTQ3y0zQrQZa9UKiaHzk92y5gZD+kE5kwbU5ic0o1d0TVuLzFMRxQLNNH0/Qbd1gGOLhGrnEik9d2rHpcyOU/6HmGpQYIgrSBObaxV/PI8LS0LheEku1r21Tr+8aCku8B9Gmd+H9icw81SHaPXvQ/aTSbM1azIDcUfKmZXm7/AkLpeOe1nTWF7E+L6Y/oM7rGRPggE4h74eUEhACU0bCw0I2pGewvfDPU92Q4mlWMsa/44MTWNvaf6Udk+Zmcp1CKkbjXbTE52VATMveTyavqoyNG6uKtHEMVczWgWnOxtDFqHimi9UMfAL2RDD+xXGOMVvraK5B+yuNhO9GsFytFVjXFyrrxCvvECMrmMH9+E17ck1q8v9clVtnJIXA6MK+5OqOHptb3vvyEu7RvFwSDkItZkPPrG0zndK3Lph+zieMLdZCPXwDw9s2Y9pv8jnG1WXCUpUzi6kHEQpcnOsS1v8GUa1DfHHNRdMu+jeUmv4UKGF+3tDRhZ47u9uQ0Qjv4uYSxSgSZR379rmenB/5VL98CprUDpmjJN43/j0M7dsKxYmaFI6P3QAJ3dZ3L8SNaiMvXFAMyD5qn6iCScgBbH5MfO7zwKJ5H+JfPxNb0SOTDgNnC8zGFVHKk/Tm1vZF5DiaoVG3fLFuroo5wgXRT0qhLfF0v5DEmoufzgOwfackt+NN2nkvUCjtsDmDUn54bVexCrHWUEn5I1bWDRkqEOGGjpDjPP92GERXp0PG0spqTfLLVtnQJz2cFyUAWu5syYof3qO1rWQgKNG7fTLmOtZ5g8FCuYWQdw7PWxSIgY500Xazzkpk+H95pUqZ9G2xVZ4nwB54YqWSQ1BhU//zCiKLjFzICG9XsccCy6f1jxvGQgvFH3bwmRw+VxGhlFz8J7C3lj4JGxe3YKtFU2bDQ+wElemp9uI/MqBeNMSiT+VRIQMlrv3QAWlNF1n0WJ6ZsbT8dACX3lhNc7+JpFmRH56FGukiRQnxA18suKOOXZ+2OU1OWUc58cQ5UVAzeyaj7qtLlT6eRuV1VMaj/VIofyXRI+B7ETzGHKn8HuP1I1J0jzagZGF6wG X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 25db4a49-20b9-4c4f-c7e4-08dbf680c70e X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:36.0524 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: O3sydVfj1GvRwlk+EcN22nLxW8JM218uMd+dmK3UlV7TxuEWG+pnUr698xW7E+aj X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092901_330012_CD0C59B1 X-CRM114-Status: GOOD ( 29.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org BTM support is a feature where the CPU TLB invalidation can be forwarded to the IOMMU and also invalidate the IOTLB. For this to work the CPU and IOMMU ASID must be the same. Retain the prior SVA design here of keeping the ASID allocator for the IOMMU private to SMMU and force SVA domains to set an ASID that matches the CPU ASID. This requires changing the ASID assigned to a S1 domain if it happens to be overlapping with the required CPU ASID. We hold on to the CPU ASID so long as the SVA iommu_domain exists, so SVA domain conflict is not possible. With the asid per-smmu we no longer have a problem that two per-smmu iommu_domain's would need to share a CPU ASID entry in the IOMMU's xarray. Use the same ASID move algorithm for the S1 domains as before with some streamlining around how the xarray is being used. Do not synchronize the ASID's if BTM mode is not supported. Just leave BTM features off everywhere. Audit all the places that touch cd->asid and think carefully about how the locking works with the change to the cd->asid by the move algorithm. Use xarray internal locking during xa_alloc() instead of double locking. Add a note that concurrent S1 invalidation doesn't fully work. Note that this is all still dead code, ARM_SMMU_FEAT_BTM is never set. Signed-off-by: Jason Gunthorpe --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 133 ++++++++++++++++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 15 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 +- 3 files changed, 129 insertions(+), 21 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index d5772ce27d48b2..e1df7dca853615 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -15,12 +15,33 @@ static DEFINE_MUTEX(sva_lock); -static void __maybe_unused -arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) +static int arm_smmu_realloc_s1_domain_asid(struct arm_smmu_device *smmu, + struct arm_smmu_domain *smmu_domain) { struct arm_smmu_master_domain *master_domain; + u32 old_asid = smmu_domain->cd.asid; struct arm_smmu_cd target_cd; unsigned long flags; + int ret; + + lockdep_assert_held(&smmu->asid_lock); + + /* + * FIXME: The unmap and invalidation path doesn't take any locks but + * this is not fully safe. Since updating the CD tables is not atomic + * there is always a hole where invalidating only one ASID of two active + * ASIDs during unmap will cause the IOTLB to become stale. + * + * This approach is to hopefully shift the racing CPUs to the new ASID + * before we start programming the HW. This increases the chance that + * racing IOPTE changes will pick up an invalidation for the new ASID + * and we achieve eventual consistency. For the brief period where the + * old ASID is still in the CD entries it will become incoherent. + */ + ret = xa_alloc(&smmu->asid_map, &smmu_domain->cd.asid, smmu_domain, + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); + if (ret) + return ret; spin_lock_irqsave(&smmu_domain->devices_lock, flags); list_for_each_entry(master_domain, &smmu_domain->devices, devices_elm) { @@ -36,6 +57,10 @@ arm_smmu_update_s1_domain_cd_entry(struct arm_smmu_domain *smmu_domain) &target_cd); } spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + + /* Clean the ASID we are about to assign to a new translation */ + arm_smmu_tlb_inv_asid(smmu, old_asid); + return 0; } static u64 page_size_to_cd(void) @@ -138,12 +163,12 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, } if (!smmu_domain->btm_invalidation) { + ioasid_t asid = READ_ONCE(smmu_domain->cd.asid); + if (!size) - arm_smmu_tlb_inv_asid(smmu_domain->smmu, - smmu_domain->cd.asid); + arm_smmu_tlb_inv_asid(smmu_domain->smmu, asid); else - arm_smmu_tlb_inv_range_asid(start, size, - smmu_domain->cd.asid, + arm_smmu_tlb_inv_range_asid(start, size, asid, PAGE_SIZE, false, smmu_domain); } @@ -172,6 +197,8 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid); if (WARN_ON(!cdptr)) continue; + + /* An SVA ASID never changes, no asid_lock required */ arm_smmu_make_sva_cd(&target, master, NULL, smmu_domain->cd.asid, smmu_domain->btm_invalidation); @@ -378,6 +405,8 @@ static void arm_smmu_sva_domain_free(struct iommu_domain *domain) * unnecessary invalidation. */ arm_smmu_domain_free_id(smmu_domain); + if (smmu_domain->btm_invalidation) + arm64_mm_context_put(domain->mm); /* * Actual free is defered to the SRCU callback @@ -391,13 +420,97 @@ static const struct iommu_domain_ops arm_smmu_sva_domain_ops = { .free = arm_smmu_sva_domain_free }; +static int arm_smmu_share_asid(struct arm_smmu_device *smmu, + struct arm_smmu_domain *smmu_domain, + struct mm_struct *mm) +{ + struct arm_smmu_domain *old_s1_domain; + int ret; + + /* + * Notice that BTM is never currently enabled, this is all dead code. + * The specification cautions: + * + * Note: Arm expects that SMMU stage 2 address spaces are generally + * shared with their respective PE virtual machine stage 2 + * configuration. If broadcast invalidation is required to be avoided + * for a particular SMMU stage 2 address space, Arm recommends that a + * hypervisor configures the STE with a VMID that is not allocated for + * virtual machine use on the PEs + * + * However, in Linux, both KVM and SMMU think they own the VMID pool. + * Unfortunately the ARM design is problematic for Linux as we do not + * currently share the S2 table with KVM. This creates a situation where + * the S2 needs to have the same VMID as KVM in order to allow the guest + * to use BTM, however we must still invalidate the S2 directly since it + * is a different radix tree. What Linux would like is something like + * ASET for the STE to disable BTM only for the S2. + * + * Arguably in a system with BTM the driver should prefer to use a S1 + * table in all cases execpt when explicitly asked to create a nesting + * parent. Then it should use the VMID of KVM to enable BTM in the + * guest. We cannot optimize away the resulting double invalidation of + * the S2 :( Or we simply ignore BTM entirely as we are doing now. + */ + if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM)) + return xa_alloc(&smmu->asid_map, &smmu_domain->cd.asid, + smmu_domain, + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), + GFP_KERNEL); + + /* At this point the caller ensures we have a mmget() */ + smmu_domain->cd.asid = arm64_mm_context_get(mm); + + mutex_lock(&smmu->asid_lock); + old_s1_domain = xa_store(&smmu->asid_map, smmu_domain->cd.asid, + smmu_domain, GFP_KERNEL); + if (xa_err(old_s1_domain)) { + ret = xa_err(old_s1_domain); + goto out_put_asid; + } + + /* + * In BTM mode the CPU ASID and the IOMMU ASID have to be the same. + * Unfortunately we run separate allocators for this and the IOMMU + * ASID can already have been assigned to a S1 domain. SVA domains + * always align to their CPU ASIDs. In this case we change + * the S1 domain's ASID, update the CD entry and flush the caches. + * + * This is a bit tricky, all the places writing to a S1 CD, reading the + * S1 ASID, or doing xa_erase must hold the asid_lock or xa_lock to + * avoid IOTLB incoherence. + */ + if (old_s1_domain) { + if (WARN_ON(old_s1_domain->domain.type == IOMMU_DOMAIN_SVA)) { + ret = -EINVAL; + goto out_restore_s1; + } + ret = arm_smmu_realloc_s1_domain_asid(smmu, old_s1_domain); + if (ret) + goto out_restore_s1; + } + + smmu_domain->btm_invalidation = true; + + ret = 0; + goto out_unlock; + +out_restore_s1: + xa_store(&smmu->asid_map, smmu_domain->cd.asid, old_s1_domain, + GFP_KERNEL); +out_put_asid: + arm64_mm_context_put(mm); +out_unlock: + mutex_unlock(&smmu->asid_lock); + return ret; +} + struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, struct mm_struct *mm) { struct arm_smmu_master *master = dev_iommu_priv_get(dev); struct arm_smmu_device *smmu = master->smmu; struct arm_smmu_domain *smmu_domain; - u32 asid; int ret; smmu_domain = arm_smmu_domain_alloc(); @@ -408,12 +521,10 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, smmu_domain->domain.ops = &arm_smmu_sva_domain_ops; smmu_domain->smmu = smmu; - ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain, - XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); + ret = arm_smmu_share_asid(smmu, smmu_domain, mm); if (ret) goto err_free; - smmu_domain->cd.asid = asid; smmu_domain->mmu_notifier.ops = &arm_smmu_mmu_notifier_ops; ret = mmu_notifier_register(&smmu_domain->mmu_notifier, mm); if (ret) @@ -423,6 +534,8 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev, err_asid: arm_smmu_domain_free_id(smmu_domain); + if (smmu_domain->btm_invalidation) + arm64_mm_context_put(mm); err_free: kfree(smmu_domain); return ERR_PTR(ret); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index e30de36044cac5..3907cf4db3efe1 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1217,6 +1217,8 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr; + lockdep_assert_held(&master->smmu->asid_lock); + memset(target, 0, sizeof(*target)); target->data[0] = cpu_to_le64( @@ -2026,7 +2028,7 @@ static void arm_smmu_tlb_inv_context(void *cookie) * careful, 007. */ if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { - arm_smmu_tlb_inv_asid(smmu, smmu_domain->cd.asid); + arm_smmu_tlb_inv_asid(smmu, READ_ONCE(smmu_domain->cd.asid)); } else { cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; @@ -2269,17 +2271,10 @@ static void arm_smmu_domain_free(struct iommu_domain *domain) static int arm_smmu_domain_finalise_s1(struct arm_smmu_device *smmu, struct arm_smmu_domain *smmu_domain) { - int ret; - u32 asid; struct arm_smmu_ctx_desc *cd = &smmu_domain->cd; - /* Prevent SVA from modifying the ASID until it is written to the CD */ - mutex_lock(&smmu->asid_lock); - ret = xa_alloc(&smmu->asid_map, &asid, smmu_domain, - XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); - cd->asid = (u16)asid; - mutex_unlock(&smmu->asid_lock); - return ret; + return xa_alloc(&smmu->asid_map, &cd->asid, smmu_domain, + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); } static int arm_smmu_domain_finalise_s2(struct arm_smmu_device *smmu, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index efc6bc11bbb838..96b01db47b0ea5 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -586,7 +586,7 @@ struct arm_smmu_strtab_l1_desc { }; struct arm_smmu_ctx_desc { - u16 asid; + u32 asid; }; struct arm_smmu_l1_ctx_desc { From patchwork Wed Dec 6 17:28:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482105 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9546CC4167B for ; Wed, 6 Dec 2023 17:29:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=AZPr/5AFWoSI9L/qjEUDFLjrDbdaFbGEJ0b33XrLBL8=; b=pnVVzBsGQawxYC 4CquXr8/2zLKfZqICU2DEev8RC8lUpoeaEa/8lz4z4VA8wOaU+VsmmQi0Tgy6vVS91SivXLvM5DdI RdUpyOPNEfzHqWwQu2LA9piztK+GMkC6z35nT9A1fw76Fqyh4fKcUKiPfv+YOOS3Gq9KfeezhBEDB dKhlwqb+fyHRNMOAY4oruQSQXf9saIe8jQflO6pYI9b5V6+c+xOldg80VAEQKiC/PWSDjukgx4xo/ BNnDl7I9yZ6xT37ojdHRnpuAG4EDp7wkWX5BjaSEiJSv/X62a+rEp52hEzySlYhZNGW3rzEgiMZEL RXnYpgz8xYL7n9pCup9A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhx-00Atlu-13; Wed, 06 Dec 2023 17:29:13 +0000 Received: from mail-dm6nam11on20615.outbound.protection.outlook.com ([2a01:111:f400:7eaa::615] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvha-00AtKY-05 for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:53 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VGD50jk97eN1l3fBTQ0F90IGM0FqNKUtxYHKmv4ibvvTWcSDe/FT0dQgqZrfPWs79Y7B0SFvoeIvbXUyGIJ7gsqj4Vf19YBUPCub1kdrJUlPw09NUi+BFDGDFdx8V/Uoh3eHrbWmFSxBf9kwyzw/uT5twaAUMuvUSTW3UaerwSc6yXjZDKm+DGEo9MdU6qi/qPctsKluJpYAo0BFu778baEGEsB6ynluUCGwCiAdd0hT0clZ4XFJ8a7cCwgpps6KJiDI+E6avHglFj7c34qe702c0s+JF8JCrzD+vxlxJ7thu96xfZFIRFBn7jETqnQ1qtnMvNd4XrXqL7bYygRbAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IUime/3yKgW4y4/G58R6ZKEJX25ieum4+gTWMfe2tW8=; b=OqUCVwYeD64dP0Vt6frBHaEfdvwKrb3fztmTVHQ3yUOAj67ONCAdNLPQPsu8Z+ZQ6oUzkieQCWcouytQ3E4C1YCjSiqShgSs3pzwZ2j/R8KCVx/CNIXFeTjnzJkulUkrxiteayHyNtSMdH3NUxGwZr+UKsWf2vz/UFEgr0u6cN5cnxMvquVWmSxMtUK9bO2DiChBK+fwGf0W3iQW61rgOs3TMB8eEUE9zLEMg0sT+F1pya3mb2db/Cgl74FWPdU64JnMUXpodHHM7RqBaPtQg+mWo5Z4dqrNi1QaixrZZk8BzUCE6Ynfe4UxsjLANWzZxMHNc7KVE/gQBSIOsiKc4g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IUime/3yKgW4y4/G58R6ZKEJX25ieum4+gTWMfe2tW8=; b=g5B4dTe/xQkE759nZkCPzq9x0uwSBOiTknjou3ODXE8eooa7nv1uRvf40NctOqt1YZywxHH1xntD6YtfPgBMuA3A+26gf1wQvPE7A00hHppCGf2tjEFkkZW/RBb/jGux4yFEHyyFcHuRl034mjgaQY6+7lSkSbTdLMmb91/0ZhLA24alWkLcr6P5nait62dHYflblmrh5kgXkpkABZ/wg5lKFvpMU/Uozsx00sWv5tLV2gmTuku19LIbcNrfPn4ui3eSwtD5da81NCI2l3f0Fx8qi1QtiXrE0OjWVCLg0MF/U/k0QjHI0tTSaJyeJwCfVSO24a/ICMEWsqOoDl5kjQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:41 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:41 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 25/27] iommu/arm-smmu-v3: Allow IDENTITY/BLOCKED to be set while PASID is used Date: Wed, 6 Dec 2023 13:28:31 -0400 Message-ID: <25-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR1501CA0017.namprd15.prod.outlook.com (2603:10b6:207:17::30) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: 45c20fb3-301e-4350-43b2-08dbf680c689 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fqQ995YdNad6fSbiOSacivaysPxeF3nHLYsFGR/1V0DX/8iAeQJtCSF1SX5WH4OXQuIghweLGqXNSR/ckDZK99CvtDinzI3JNL6N0pdqv8tpUWRcqIBZinTfP0538ByO1ONBkpz+koo2B7kldB0QifdTjZPMQrzZjUQfaDycVbY4sHiu6cuKF7cWciMXQGjiitRv1Y7qf1ctGhsiuBHhz8GHcRhvZRBJgwQ3iNl1zjmKS1w+5Ez4kk6ZdOTQPZ2gypwXxtveI9XVP1L6Nf6vZk/yJ3hQfSdGms9AAVbugB+xgBGr2AucOQjv8S/HUPk9BBbfuVPdVWNyeCzJCSlezMpkNWQiGXvTyihG4p9ZgSrW8RGJL1T+J5Kc5R3ZTVfNRwkTc9elet2rOMV9WbQiZ5g/k2EQIu0gR75OVI/c1q6vzmK5mLcQiNeBHJSAqjRCwlkHN28WeelEmRLGqFac+sS5Js2MUfEAiFH/lFUibDBlGvZ5TcDgjwY4SsNSKOqt/dbcpRLUyXycjtysjx4ISCHLxVrDLzwDxGmc2CO9mNet42EPXfJQh+GYYor+aFbJ8ZPMorP74W2xdYwMVVRBs5T/fEmmb1s737c29/Bd46bg1wdCiqwu+VKBRQJFwGR68X63E9ycW5Pu7uD7QJlelA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(4326008)(54906003)(7416002)(15650500001)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005)(414714003)(473944003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: g2S5xz3I4hpMr1mxbHdS7HyojUrQQwVZjPld2RcbWV7F7qh3HePVnWyexA5ZpsBG5FqKesj9vCQqB0JjYGaS5YxMV+3FSj0F7ByjGFzMYPOpOo/iP8YwWvrAv73YameytlSFFEz+GS1ohF6AV4DLJmAHBYl58kxuUeZPKYmK14QrMd0UhQpYjBsZ6y5IWH7Lv6HdiZL1Ba1otFvwrQQvZE4uzDdHdYxW2T3gDNbzM4JzlgRXumzgm6F+6RfRm5GSdmPTIMdoNnW62ya9ps/tHbWEOssDjQwCNdLtgyarGO4S0nDwucO4DMvq4/VBLxque/JLzCQrsXuSvq0qnZu4J00z09qZONmG5CMwXtlnAC9dm1zXfKVO8ykaWc4jdkPoNTi2PviWG2/BzYbbqy1e08K4/3c6bAv3tfB15MQyskop9N9mXvd6Gt5Ryc9i6lLAfZQKQJGjlpRIL9gwdtHW6e6ofmqp5pusqF3ZChzt6UnX63sWsdJS3EvY+TXzCUXTUVWd3zuyhdfwRpiJIRdVSAIWa13+KPqAtz1a66qtDy9kLBNc1AzOdP9a+DOAfjvb56wzzuHlfqhgM8myQjep80Hl/eI/DniJrzt++FNKoeN5vNTYup+Sbe1URBfUcXDIGvuDI+u6+g3S7qf+ueorsUP9uCYXaAcRaFqwflJa74s+kdXzV+19xMTBmU/NEsxzNBtWQ8HoBZOolRZlviVlzKCqhQIPHhyE2LvP5A81jUMcg0lBS0zS/FzDpCvWVfW2xDr4EXd76W/g+v2w3CxPGI7v2KulVhc2Nw9V9w0tc7v72auyL/kwOic9IGpwQL6o8NllN9UOaaSFxnkhUsytBOoiGDFTRLq8jinczAAndFLeaDfEnGH8LSDzz4ZP7T0c0UnXwFsRWuQPfuPaxdKy8K7yCjj85GWELXgHlYg4I3yBBoI09+X2QnigvHyc9/s6hr5gRmJNp6oJ/bZ/RIYc2P0uUhUxsRO4akpXg/96JcuTGxzwwCr5Kk4+V8LvxSGJdu7hr97kM6porPap99ktd6l2Fb0VxU0mifNsMW18271F72vt5twW2KI0cYGMOf2O1tP3vq2c0Q65W5R45e7eQv3oOgqseMUniKMl5T9Va1/uibBAYoGZiFbOVtJWcGYgCI38HdvJZeGBwq72BZEoe0GGmfPGPmQJekof/lgW3C9rOTVmCJonTnJid7Rc1V/P0thrWvnN99Zx3gyCiglm6QWrkkOpj+sewR+kG4b4m7NOMFGQqtcyLqCJEN4zFNsi+VzYa1xUtCzEU2Sh9g4Y9Xo+FNp1KHrPMbXK95KCNpVVjTLRYMTYc3MhCIhMfZSzGhpqKYzdOiRB3GCYxmMyFQdxd2YtAguULUlpvRX0mENXLib9I9Rz+sEMz7tpbx0xQIeyuX51rj68osD4a4JMFyNMx4saUzGdJSLXfUgjHLgmGu56OJdR8KnCmiw7MiZXAkhPTJV0SweRq5elQlFu54UZyp15F/H+52c7tsA8KuHeXie2N9zCMrSnQnpcfvnHYZMFCe7JWSj384la/pzMO+sR0uZIMkCTr6Dd9GmdQrnc/roPx/68s/ylUPT7xY39 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 45c20fb3-301e-4350-43b2-08dbf680c689 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:35.1450 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: J35XH6Gx0zdzsuN59HIs6LCRDH3ekKrlhc8U9rc/PqK82AwEfsZpCY8jTtKNG1Cq X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092850_123977_88CFA4B4 X-CRM114-Status: GOOD ( 24.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The HW supports this, use the S1DSS bits to configure the behavior of SSID=0 which is the RID's translation. If SSID's are currently being used in the CD table then just update the S1DSS bits in the STE, remove the master_domain and leave ATS alone. For iommufd the driver design has a small problem that all the unused CD table entries are set with V=0 which will generate an event if VFIO userspace tries to use the CD entry. This patch extends this problem to include the RID as well if PASID is being used. For BLOCKED with used PASIDs the F_STREAM_DISABLED (STRTAB_STE_1_S1DSS_TERMINATE) event is generated on untagged traffic and a substream CD table entry with V=0 (removed pasid) will generate C_BAD_CD. Arguably there is no advantage to using S1DSS over the CD entry 0 with V=0. As we don't yet support PASID in iommufd this is a problem to resolve later, possibly by using EPD0 for unused CD table entries instead of V=0, and not using S1DSS for BLOCKED. Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 66 ++++++++++++++------- 1 file changed, 43 insertions(+), 23 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 3907cf4db3efe1..f97fac1de0b103 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1485,7 +1485,7 @@ static void arm_smmu_make_bypass_ste(struct arm_smmu_ste *target) static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, struct arm_smmu_master *master, struct arm_smmu_ctx_desc_cfg *cd_table, - bool ats_enabled) + bool ats_enabled, unsigned int s1dss) { struct arm_smmu_device *smmu = master->smmu; @@ -1498,7 +1498,7 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, FIELD_PREP(STRTAB_STE_0_S1CDMAX, cd_table->s1cdmax)); target->data[1] = cpu_to_le64( - FIELD_PREP(STRTAB_STE_1_S1DSS, STRTAB_STE_1_S1DSS_SSID0) | + FIELD_PREP(STRTAB_STE_1_S1DSS, s1dss) | FIELD_PREP(STRTAB_STE_1_S1CIR, STRTAB_STE_1_S1C_CACHE_WBRA) | FIELD_PREP(STRTAB_STE_1_S1COR, STRTAB_STE_1_S1C_CACHE_WBRA) | FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_ISH) | @@ -1511,7 +1511,11 @@ static void arm_smmu_make_cdtable_ste(struct arm_smmu_ste *target, FIELD_PREP(STRTAB_STE_1_STRW, (smmu->features & ARM_SMMU_FEAT_E2H) ? STRTAB_STE_1_STRW_EL2 : - STRTAB_STE_1_STRW_NSEL1)); + STRTAB_STE_1_STRW_NSEL1) | + FIELD_PREP(STRTAB_STE_1_SHCFG, + s1dss == STRTAB_STE_1_S1DSS_BYPASS ? + STRTAB_STE_1_SHCFG_INCOMING : + 0)); } static void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target, @@ -2656,7 +2660,8 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) arm_smmu_write_cd_entry(master, IOMMU_NO_PASID, cdptr, &target_cd); arm_smmu_make_cdtable_ste(&target, master, &master->cd_table, - state.want_ats); + state.want_ats, + STRTAB_STE_1_S1DSS_SSID0); arm_smmu_install_ste_for_dev(master, &target); break; } @@ -2727,16 +2732,14 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) mutex_unlock(&master->smmu->asid_lock); } -static int arm_smmu_attach_dev_ste(struct device *dev, - struct arm_smmu_ste *ste) +static void arm_smmu_attach_dev_ste(struct device *dev, + struct arm_smmu_ste *ste, + unsigned int s1dss) { struct arm_smmu_master *master = dev_iommu_priv_get(dev); struct arm_smmu_domain *old_domain = to_smmu_domain_safe(iommu_get_domain_for_dev(master->dev)); - if (arm_smmu_ssids_in_use(&master->cd_table)) - return -EBUSY; - /* * Do not allow any ASID to be changed while are working on the STE, * otherwise we could miss invalidations. @@ -2744,19 +2747,34 @@ static int arm_smmu_attach_dev_ste(struct device *dev, mutex_lock(&master->smmu->asid_lock); /* - * The SMMU does not support enabling ATS with bypass/abort. When the - * STE is in bypass (STE.Config[2:0] == 0b100), ATS Translation Requests - * and Translated transactions are denied as though ATS is disabled for - * the stream (STE.EATS == 0b00), causing F_BAD_ATS_TREQ and - * F_TRANSL_FORBIDDEN events (IHI0070Ea 5.2 Stream Table Entry). + * If the CD table is not in use we can use the provided STE, otherwise + * we use a cdtable STE with the provided S1DSS. */ - if (master->ats_enabled) { - pci_disable_ats(to_pci_dev(master->dev)); + if (!arm_smmu_ssids_in_use(&master->cd_table)) { /* - * Ensure ATS is disabled at the endpoint before we issue the - * ATC invalidation via the SMMU. + * The SMMU does not support enabling ATS with bypass/abort. + * When the STE is in bypass (STE.Config[2:0] == 0b100), ATS + * Translation Requests and Translated transactions are denied + * as though ATS is disabled for the stream (STE.EATS == 0b00), + * causing F_BAD_ATS_TREQ and F_TRANSL_FORBIDDEN events + * (IHI0070Ea 5.2 Stream Table Entry). */ - wmb(); + if (master->ats_enabled) { + pci_disable_ats(to_pci_dev(master->dev)); + /* + * Ensure ATS is disabled at the endpoint before we + * issue the ATC invalidation via the SMMU. + */ + wmb(); + } + } else { + /* + * It also does not support ATS with S1DSS = bypass but we have + * no idea what the other PASIDs are doing so it has to be left + * on. + */ + arm_smmu_make_cdtable_ste(ste, master, &master->cd_table, + master->ats_enabled, s1dss); } arm_smmu_install_ste_for_dev(master, ste); @@ -2768,7 +2786,8 @@ static int arm_smmu_attach_dev_ste(struct device *dev, IOMMU_NO_PASID); } - master->ats_enabled = false; + if (!arm_smmu_ssids_in_use(&master->cd_table)) + master->ats_enabled = false; mutex_unlock(&master->smmu->asid_lock); @@ -2778,7 +2797,6 @@ static int arm_smmu_attach_dev_ste(struct device *dev, * descriptor from arm_smmu_share_asid(). */ arm_smmu_clear_cd(master, IOMMU_NO_PASID); - return 0; } static int arm_smmu_attach_dev_identity(struct iommu_domain *domain, @@ -2787,7 +2805,8 @@ static int arm_smmu_attach_dev_identity(struct iommu_domain *domain, struct arm_smmu_ste ste; arm_smmu_make_bypass_ste(&ste); - return arm_smmu_attach_dev_ste(dev, &ste); + arm_smmu_attach_dev_ste(dev, &ste, STRTAB_STE_1_S1DSS_BYPASS); + return 0; } static const struct iommu_domain_ops arm_smmu_identity_ops = { @@ -2805,7 +2824,8 @@ static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain, struct arm_smmu_ste ste; arm_smmu_make_abort_ste(&ste); - return arm_smmu_attach_dev_ste(dev, &ste); + arm_smmu_attach_dev_ste(dev, &ste, STRTAB_STE_1_S1DSS_TERMINATE); + return 0; } static const struct iommu_domain_ops arm_smmu_blocked_ops = { From patchwork Wed Dec 6 17:28:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482147 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2C54FC10DCE for ; Wed, 6 Dec 2023 18:31:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+p8djLLCWlDMhPNGZ7JnpgtmUGv0OZy0rJGuzNuf0SE=; b=C4BMuHKXIB3Gjh 2hGYwVC7eqvoYmZre4/VAtxgsl+I7ren49o7U268QtIm7c9mmnejBQHlCPpf61k3SiB91HMu8UsmK AZ3AkgFP45jAB7Ls4Z5TN0A8sjPp8f9GZlN2x8HcWDyhHhD/9Yva2SoaI1m/a3AnbNKZJ4I0qBu0u dUSofDOfBkOIKGfYshsDZp9rB8GHQqyrGjRVrAFtN5DqqCu7xCCqnMv/Kijesj0/RAAmgpF6xIruE 9GBwzv34zsfPyaOsqQekteJxrn4yfgT9/PRSdVfccRxlSW4qzQxZwg1cmedPsb/HFWP8M2zHMbOFd 8Pj2SN3qk8PYXVQgDhoA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAwfS-00B2Cr-0k; Wed, 06 Dec 2023 18:30:42 +0000 Received: from mail-dm6nam11on20615.outbound.protection.outlook.com ([2a01:111:f400:7eaa::615] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhy-00AtKY-0g for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:29:25 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=InBaeYUL5y/l1ablo9joRlRY5b7q3oFjdmwMJbzssQ58KvP6/WyvO7uYe/nDSZE036+6ZkI8mnQ+0P+35jrKN86iKGUKlABNyWS6nc+cI8kZ7yPqNZbD9A8cdTWplb3+ojbPOxKBP+8Wh9CjWkjxoXXmqt/2VeVu+DyYWxDsQeEfu86IX6xbHYxUkMeFEPk/w9h0xcQFU4cwR6BsKTG7nGXZaSfHjOe6EqBcY6RLj1/Xip67BeCZxstLd6PIHgOEFW2jdEZtgsX3U4meA4APi0iHpFh1MMJQKJxeTiVcLSQnIKKKhWD2+2RS0BOTrD8ofJT2ZR8ov3OkdYme+ey0+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dE8YPqXFkgqbxJ+kWciNCG/M1/X9RBAap59Ml6R3fgs=; b=WfvZg12pOpod2vpZUAzaGFIxDSlkDhIQ0LE2BI5VJNC9wtTF12rsAwxG0VQu4QLeNKgmfRia6sffmKdbzuWrVu/w0CyTkKtWG39juPsuKpJxQWczfaKcf64zZr3E4DtU3LlVXZPqs3TYyhUSns43Kcro/mh3L4SFY5Hq1kW9nf7LyK5A51Y7TNfsKjdlzcQ7U6AbXd0lSYmxZnIG/uwngsdQyfTYVzN9W+5LT3t/ffSVFzZ8virA8R/M35GLr0vbmYJjNmUVkk8GREn9NAHiwlvsmFsXJIE2Ug4fJphY7RDh1wznSCLrjuIkzBFwCsACWIpFCayf2OR11sgD0osfLA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dE8YPqXFkgqbxJ+kWciNCG/M1/X9RBAap59Ml6R3fgs=; b=ryITzgQRcqeKUM6Rrw3bNWt2dPUSxPemxdF+Q4h7ISleEq0ZHQw03JOwVOd5F2J2qGaTfdq0sX3CJo5l4a7pNMlujah/nzkgso0HRxbbK+C8izGahxFwFvjwCSn0F77oqminCJF6/rwXSOBK0qPJ6M2wfD/K+iOFwU635terDISNuJvTjVtgr1wZhGfgOXjTijDvD61OELeafcQwjl5X+tU/um17BTjS1x2F9WF51s2tP0VqPXIRljOnKJPKGrU5zzGbG/LSxoEDrDqRRG7t4Jk/Qi7ueQMbNI+HoT+raYM51YCQl1wypBb4Sf4gqHfCy7px13vc9uDT9Cf0TfrH1g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:45 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:45 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 26/27] iommu/arm-smmu-v3: Allow a PASID to be set when RID is IDENTITY/BLOCKED Date: Wed, 6 Dec 2023 13:28:32 -0400 Message-ID: <26-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR22CA0003.namprd22.prod.outlook.com (2603:10b6:208:238::8) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: 58cb03a5-9387-45dd-6127-08dbf680c769 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: +Q111PE6d4t52VQh+D4wGVvnYgR9mUW9s4RKwUYTBoVbeWMzwTv7+c16EXs0PnWtHpU7HwSz9v0jgndyS8TtjJ0w5Mtstu/KNBoXeV/Apsa+D4tsENaCqMlfXx26IRJP0h38nVWv2jgWysKGX7LK08TlBSIH0yO+t6w1mXlDE1FAdzQcoZULU8322hCvdNRaryRsF/oahUPdv4lDYRMp2MqWwm9yxvGy/fTScJhZWbLtPE11uhW1Fb4uaMxUuDMDD3f4a7CLfprvGjaRRH+esPBouy2R2x+ySlhMOmMRhZ+xUVvYvhsTNricnPozdpK6DzdOxjtAQqZugloGUg6lXqtNmWwyGYMKtxx7ZVFETI9qnCp05PZvoOKwh42v3SPUDbWKfj00R9b+K3NEZspI+DlREx1NBMuDr56srAVYqL7Ky6eZ5VB0J5SzZQRG8RnN+F4DdqihWw1yt+tDK8Fl2zhlVSpyelII2aLrxf+JV5bpouVbIh/eYKxjHahrcyIKeDRCsmPXBdqnEO3eSyBdU+YTb1/Zqvo5tad+VvKsPwz8gbyZ9dNqDM1jcKp80EuOKfHCPErJfOknIJzqHRIGGbF5JBY237XnwFe+6lYzdoYX0v7HeyZHF5SCBblz6K2jDSIL72kGC0mBmIOHavqUzg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(4326008)(54906003)(7416002)(15650500001)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005)(414714003)(473944003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: yGGlPx6FNTtFMS5cQ1O+tV3/WF/5ikQthUsH3T6pwjuljfoFzv6KJ4eqG1m82fDjIONHbdtr/QHuxmpMw+3enjK3e8GjJPqSrvT8okxvN/pwC2zHwFp/T8FrhFx007ebEAXP/c6V2WbtTZWW7BD0x7iViJkar5vOCgtvJm3cqoMoop8G0hRIrAPSDOu2jeUEX15h72hFZ/aoltbjKiaIDvIdf3t759EBS2KstbhryB1RBg11hxNaGUGf9xbKuzS3Nh1So9U1QCK5JpzIl7Aepvjce/UZAjD1EstJgAOgz/7MiqBXkitHSZDfyWz5T8x/vc1dcGDfchio0x7kmfxRFujxcGQmVvPupNAiJRfNxdRzhhBBjZIy0T6NikqY0r7E6SuD7ShAiICljIwrFk8zCd+E3LCmTyVbuwzwqICDgb/ABHyBkvWKBfLl2Nz3rptGPoEqW4fPQzLvd0uR87kh+Dxh+XJRLBWaK34WXyoJLQojSzdrpSUrRT0Z4rNMDvEu61V6qYRN4da2hl2XxYFyzgiJuYTC+onVIvs3bDE5Xz/uO97uvp2dgAbOUlB5Rr5r51tz+bUcThNf2vB93XsU0BbWXrqva/25Za8ncxVQ2Cn41gRZ7F1JTmBkX9ERpPuQj2msZ1+3Cz6bgO18VDqNsgWYDiP2uyNr1Vn4uEgj4H9OI5SjhEaedDxC5DUAIgSwwdHXnMc58xbqI4azHV3ujGxmYiM6Ojnk7lCMqbbdKmFYbjtGDcEmbTOsKShmg/HHTet7QSsbgnhG6JWhvpwLhRuCQtG3o3xLmc2Goa1b4K3LFB82xBtBQSv8wEAcEuisvy8MdSc2kdvbfnJdaJSxQFsLgBGmg8Plk55e0SrdRv4KRqRaIDsy/r7kQpexd3nwtsf3/PXwMwff/dWmYiAn1P5x9IOU4CVOR3u7K/gjRdmueuA64fC2appVnsgxo9JO3M7PWZ8vmVUVQbPti6HWrLbBU4cO/o4t4aB4jviJ5VXX40ZWva2aFXkVQnhKYIPaMyRFhR8IVDZShY3yhCV8F7E4gGHjydidVOfcas3uO3naJ8tsKKb5a3yAjyJ5gpQiSuE6ezssBGrfc4pdhr6im+KCMcM6fCwqAMq+v1kEaPb5c18WN6uMHjbwlBhSwvH/SG7e2BAIyLKDjcFboBL9sB1EfTHyCuNIN87K9BX/pTScfsOnNYq6eMVS3ELLr1abyLBhj3CowLa14D1FJNvsvwqvIuO7CQhw0LGl24SBUak6sWfZ2yqse2D7hYvOaxTSlEZJ+4tV0McD4u+6xT6ZQhXeWJIdoWbpnn7J1Jcawrza6FlBPmfna/mjKiIog5lMcVzuHrm457jf5Oi9mbrdQIt4waLNNNRk3K1NN2bVdrGGMj+A0YzAnF88JAa4FY6ExCKwzrkc/onwylFJcGce3jCk1Vjg9IaiPfkO+ouzeVoet+aVgq3O8z3PbmMwTxx28ntxvQUUt5T7b/ljcYCy1UKMzQ2L7u/OyZ/nKP6KbBooEmRdVbjEdKf/1/tBbiUfYGOIWFPF6al5lVPw+KJv71UacO6tN7ldCIonQw+WweONIuq10oi9BkjWi/sNsgVI X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 58cb03a5-9387-45dd-6127-08dbf680c769 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:36.6695 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: VtIJX5MbGYFnZ6xQ1hEI5ikWAQBBJ7NwR7yotfOsxyZL7UAe137dyHyv66NrBcL7 X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092914_302861_0481FAF4 X-CRM114-Status: GOOD ( 21.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org If the STE doesn't point to the CD table we can upgrade it by reprogramming the STE with the appropriate S1DSS. We may also need to turn on ATS at the same time. Keep track if the installed STE is pointing at the cd_table and the ATS state to trigger this path. Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 57 +++++++++++++++++++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 ++- 2 files changed, 56 insertions(+), 7 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index f97fac1de0b103..7e5721490b7f4c 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2385,6 +2385,13 @@ static void arm_smmu_install_ste_for_dev(struct arm_smmu_master *master, int i, j; struct arm_smmu_device *smmu = master->smmu; + master->cd_table.in_ste = + FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(target->data[0])) == + STRTAB_STE_0_CFG_S1_TRANS; + master->ste_ats_enabled = + FIELD_GET(STRTAB_STE_1_EATS, le64_to_cpu(target->data[1])) == + STRTAB_STE_1_EATS_TRANS; + for (i = 0; i < master->num_streams; ++i) { u32 sid = master->streams[i].id; struct arm_smmu_ste *step = @@ -2678,21 +2685,48 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) return 0; } +static void arm_smmu_update_ste(struct arm_smmu_master *master, + struct iommu_domain *sid_domain, + bool want_ats) +{ + unsigned int s1dss = STRTAB_STE_1_S1DSS_TERMINATE; + struct arm_smmu_ste ste; + + if (master->cd_table.in_ste && master->ste_ats_enabled == want_ats) + return; + + if (sid_domain->type == IOMMU_DOMAIN_IDENTITY) + s1dss = STRTAB_STE_1_S1DSS_BYPASS; + else + WARN_ON(sid_domain->type != IOMMU_DOMAIN_BLOCKED); + + /* + * Change the STE into a cdtable one with SID IDENTITY/BLOCKED behavior + * using s1dss if necessary. The cd_table is already installed then + * the S1DSS is correct and this will just update the EATS. Otherwise + * it installs the entire thing. This will be hitless. + */ + arm_smmu_make_cdtable_ste(&ste, master, &master->cd_table, want_ats, + s1dss); + arm_smmu_install_ste_for_dev(master, &ste); +} + int arm_smmu_set_pasid(struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain, ioasid_t pasid, const struct arm_smmu_cd *cd) { - struct arm_smmu_domain *sid_smmu_domain = - to_smmu_domain_safe(iommu_get_domain_for_dev(master->dev)); + struct iommu_domain *sid_domain = iommu_get_domain_for_dev(master->dev); struct arm_smmu_cd *cdptr; struct attach_state state; int ret; - if (smmu_domain->smmu != master->smmu) + if (smmu_domain->smmu != master->smmu || pasid == IOMMU_NO_PASID) return -EINVAL; - if (!sid_smmu_domain || !master->cd_table.used_sid) - return -ENODEV; + if (!master->cd_table.in_ste && + sid_domain->type != IOMMU_DOMAIN_IDENTITY && + sid_domain->type != IOMMU_DOMAIN_BLOCKED) + return -EINVAL; cdptr = arm_smmu_get_cd_ptr(master, pasid); if (!cdptr) @@ -2704,6 +2738,7 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, goto out_unlock; arm_smmu_write_cd_entry(master, pasid, cdptr, cd); + arm_smmu_update_ste(master, sid_domain, state.want_ats); arm_smmu_attach_commit(master, pasid, &state); @@ -2717,6 +2752,7 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) struct arm_smmu_master *master = dev_iommu_priv_get(dev); struct arm_smmu_domain *smmu_domain; struct iommu_domain *domain; + bool last_ssid = master->cd_table.used_ssids == 1; domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA); if (WARN_ON(IS_ERR(domain)) || !domain) @@ -2730,6 +2766,17 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) arm_smmu_atc_inv_master(master, pasid); arm_smmu_remove_master_domain(master, smmu_domain, pasid); mutex_unlock(&master->smmu->asid_lock); + + /* + * When the last user of the CD table goes away downgrade the STE back + * to a non-cd_table one. + */ + if (last_ssid && !master->cd_table.used_sid) { + struct iommu_domain *sid_domain = + iommu_get_domain_for_dev(master->dev); + + sid_domain->ops->attach_dev(sid_domain, master->dev); + } } static void arm_smmu_attach_dev_ste(struct device *dev, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 96b01db47b0ea5..19628340ccd632 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -600,7 +600,8 @@ struct arm_smmu_ctx_desc_cfg { struct arm_smmu_l1_ctx_desc *l1_desc; unsigned int num_l1_ents; unsigned int used_ssids; - bool used_sid; + u8 used_sid; + u8 in_ste; u8 s1fmt; /* log2 of the maximum number of CDs supported by this table */ u8 s1cdmax; @@ -708,7 +709,8 @@ struct arm_smmu_master { /* Locked by the iommu core using the group mutex */ struct arm_smmu_ctx_desc_cfg cd_table; unsigned int num_streams; - bool ats_enabled; + bool ats_enabled : 1; + bool ste_ats_enabled : 1; bool stall_enabled; bool sva_enabled; bool iopf_enabled; From patchwork Wed Dec 6 17:28:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13482104 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F0F24C10F04 for ; Wed, 6 Dec 2023 17:29:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QCZiZ9cEyIhTT3RR91IOOTXioBwJ0SQ4ffTbI/dyOmI=; b=ZhPTtfEJMXNU5A cFgQTnBeAdaeQ5DeThBOv1M0+UYCzPkAfj1FOhjpo0VvfXnrhwxMjbUU2mTXnY+GjBYYWF5rNL3nA WOwfhziDkiUEOigPeTpVpTOfxfQY5u++iq18HECSZ4GEyuJv5vjgsjDXzB7jJ5A8nQ40g5fxsFED/ 6om/PsiMPmmaTnIfETBBdOzNC89jqLRGJBjbTTl297dPgboJ2h4cV37QxAuxXhuJOvFg7XN6rJ7vK luFTQQBb9fOOIxt++i+Fo2KnNUCph4YyGo9raWjCr50REG0fGNoanygo/9BwZa/+OkaoMXBPXTt1m 0VcKwfx7Vo9hFhmTfQag==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhq-00AtfF-1O; Wed, 06 Dec 2023 17:29:06 +0000 Received: from mail-mw2nam12on20609.outbound.protection.outlook.com ([2a01:111:f400:fe5a::609] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAvhZ-00AtIF-2q for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 17:28:52 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LqesJlyxNnkVz1HWlSHSLAc6ghx1WmpfXP8FYotwu5cPZ1EraEAKv42tDbZ3zp9pe9gQPd2aTU0CxBZ+8PYLawP7iuXIiqTwn30uZk8z/lx+AenJ9Qp2fqesak/8G2zNnv3bjQYykyIKWriM0SQuMPPt7GJ+o9B/K4ks5fVpfkXepa2YiYFMg+bJnY0pySzHuxPIF0mn03t/LEkEe9hlG2/iaupF+CoMBX8yBf87c2FpIoeDo04o+boBAXeGYejl4bFRZsQTiK2pvQ0W13Y0W1G1ygrWpWFXtZGPPrLyB9sh2cw5vQHb9XKWsDNPe37fl5PCT19xCgMUx19cePVPRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UGD+ydkWK/CdvwN0X7mLTOXVK5t1R7pl0Hu7fIddLjk=; b=Ire812VwnO9BtaUNJ0fYwrnLia0S3S9tuFoM0Rie3xh6MXLfO3Y8+Bf8hnksuDdwmfCDcKFEyH16M7EWp03qREPKp4pkRWIQ2eBodqBSS9OT1DUauiFQgT+HcCYMX4VyTSaBrvOvAkqJCS1EvJUyTr927W6mEfGR5/xqE5HNMyzLaparh8f/694XfoJQzRsajW3nw/Bf7L/qzLmMP7BW6OKUqmgaM0KghQsvX1gBIUCo81+5ZnfGgB4oWZWEY38JN7xh3vf56MUBofvmI5maUCJw5Z2eLczyH30mc4/EaL9H2IhODjziglbWUvORoQIMsQbIBG2ewduQNv9bv4op7w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UGD+ydkWK/CdvwN0X7mLTOXVK5t1R7pl0Hu7fIddLjk=; b=RoBiKD0BNxBQNUn53mMRXx47U8EE69Pnyk1WOsDD6GV3CJLKQVB75pOs/RXog09l7MekvNm6R478VVq40VsuXzgUG+OY1AIEOPqfdtiIDZbdOE7pM3p+NsMYtCRuz6iMRCQDRlMEURjgPasNmadCNcZy/GDBTFOSowfckZST7I/8eA/158XWv40CSOzbYK75buOXCwZmajN6omt4nKF2rkNom1ZZB+8DM97DrYWR2QRYhzgi4ecrk0F9bhibxhQRsea3SWsgxiHCKqqhnIiRPA2AiAJuke5195rA3iRCboKSZRokuKx72wWbxj1X8xZXnpHYdYL0E/k5ZuPrSpiKHQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8360.namprd12.prod.outlook.com (2603:10b6:208:3d8::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Wed, 6 Dec 2023 17:28:41 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7046.034; Wed, 6 Dec 2023 17:28:41 +0000 From: Jason Gunthorpe To: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon Cc: Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: [PATCH v3 27/27] iommu/arm-smmu-v3: Allow setting a S1 domain to a PASID Date: Wed, 6 Dec 2023 13:28:33 -0400 Message-ID: <27-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> In-Reply-To: <0-v3-9083a9368a5c+23fb-smmuv3_newapi_p2_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR0102CA0037.prod.exchangelabs.com (2603:10b6:208:25::14) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8360:EE_ X-MS-Office365-Filtering-Correlation-Id: 8da6173e-ef89-405c-8097-08dbf680c665 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: KnWPdkknfxsI8fVyZ+Q3CYlIT4frCIC7+PR4GSG6LsaqP1QvZb1lQl8/ZgFB/DmPby1TAhykAv6cE0CnEGBHPWMI8oZurHjT9wPS5V4Fx+JRsZDc0UzmjM6eulgT6KTP+6tDNaKwdes3seQNv02wvmF7Tl8dPD9HCbOG4B10HJzyvzGmwNPUiWGer+g3om7DWNMaUyvC/wkcfc1Y56WFihNgLpZ9K7y8+JgHEUQQ33fuUQAmyXFL8JZkIITrIaRQs4NdqT7Unh1i3V6aj4TJM3CBAOugio4dyoGk9M8Nu0bfjAkc5u71JwNbdjrSvJu7PAi7WTIF0kKa5VkIEyBpNjavLRzp7+RVtcWek0CqchDMtE8K+LoG/mFp/G7GvK29M7Kepce4Q5Cz+jWxxsvt7Dvl3xAPqdG7gQXrqscti+yhzM3Ic6YYKNkoPASuIXwiKA+uHnf5+l1s6gRS2uYGh1dO5qbYvKgP0WJhyGBbuEVqsx017JoeBTzBoA3TDpW3axZaIMM2morSPXWSdA9Rw/ZUy2sJojMe+VYxLqhjoVFyqaLK9OkmOst6X92pzUvp X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(346002)(376002)(136003)(366004)(396003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(6666004)(6486002)(478600001)(38100700002)(83380400001)(66946007)(6512007)(2616005)(66476007)(6506007)(8676002)(2906002)(4326008)(54906003)(7416002)(5660300002)(86362001)(36756003)(41300700001)(66556008)(8936002)(316002)(110136005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: okknbTKUQuBKxdTKF5flvpOQODk3mlKr79avucgtD+oEvF5fl58q+V634DO9VYadG/ZWW1eyKf0Cy9W0/1o1Nap++Q6U0g4X6OdjPGncGisr0Y7c2Jv/5bqW29gFUDnqUWZM0ZFA9egB6wYegIOTXZSuBa0AGqIOYbxwxnMIZTNNwBI3rV0NdjiFJb3sOR6wihJ3uaRQ/12RNTvHDTa+biYE3kO0n/8vyJS2v2o1MyVFfcp5J69Z8UEIqMXvA4R9bIp3GPG1FQ0Yub70vxsXomKgsGWjF38PuoNVVdzGYDPd/lSPpSG5fu5lGn5IWFA5cKzW9zJ7ZJWV6aPEJ2gpGJzsb5pZzT8ZjaFeck6SD2rLfTrMj0f/bMJKto2WMoXGuJIlHpioVK0XV2biahckPuKxCHf9TECKRO2mVHLJ1EFfqTgSWm8Ssb/Rq+zeU5yb9ppLDtdG2ayRuMIJaZUdOO13h6wk1wBZQPtdPdbFK6fZRYojKmlx10EdemiGk2K5UTGb134W0b5Bn8RgEraZOMrT0hf3FiKXt+yCdOvygbvIwUi1DA+2naJEdVxmBus13PSoKStmCNAQStUtOM5KKiLmAED8ajnyH+fvkmqKI3J691nvLxPMCX44ZMkxkJdHE5mfMuHWcWuaFThtRVQssy7El9JQfZm9XG44TuX9OCymdDnkhuX25qitBUGN4jpAc6V7i/MefYNwYmPnJgxnMfWu4xpSv5UvXyQlI63nCYdC0F9ga6vDRwLSywi5HtZ+cqHSH8wmWnX3tGCHFejbbHKv4fu+Nnnf3XaPtKKvBLEo3ugX7A8/EnM5kJmpIHG2ZrY5mi8hbv6Yn3W2sziLlIyK658pVlJvwhR2LzrLUgaZB8eATZHVAxl+ASAigtcEvt212uJe2QHUdWR6wJ5WrNKZp1HOdO+qWMGpz8BJJF5Hbei/E6z+g0oLPeEi3DZNMoikYfkue6IsiGS5m8bblJP8i4p8g0JxTZvUr5HP3CFyvCLbMJV88EZ0XVcnoTn4T3qlzdwRIt/TT0gZjZZozr/A6x6KLIatHlJFv4+R/kOtq8ASwpxYnJPBFhX2AgqVJHXxlxU27TTOkhJme48VEzPGdVxy9P46JC/cNjw2DSZcZeKEhhlhJc9IlI+MahTLZqlmOTMDZN2Q5gA5H2MC+MzR9rdcDpfG71+3IqQLOYxR/ApWk8nNjIWIbOKr3Ug2gE30jIkJRKTC7DXcHtAOLGkADacWDH3r/qRAkQCdvsZFdOaud6zZHzF6MuaVkwKVNtJFryPPBhDEkX8ySL7Hn9AdqtAuzvFM20wvf2zVHIsxIOnvr0R3GtQFJi93c5zHfoQzSpr+tcHuulCLY2iZHrIlEEfjXjHQ0s8zCj5LxW56PZ6Pb8ZHyBxIcqOh/+cqSPWhefxDDP3KoI7iAWbNEqw1BG+O0WQ5DLZjoiUfScV25iLFhvvlZvhHOYnAUPFWKKH9x7Tucyu6E+/Ho924O5F/LoLx1UcUEqI1WXJbcoz6eSYmladzfKAS7d2pM9XwaOC4IPXSUyugiNofpA/x4A/0w0OCjw3+cllBCd2ev+BdJPwbhmN+vIPe+q+1OMK3 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8da6173e-ef89-405c-8097-08dbf680c665 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2023 17:28:34.9605 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: COJQFTrbj/HxR9Rg3YyJYbtwqVVPKRIHbZsJuASHKSMwIL3HwtmsS2K/NlkRMn24 X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8360 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_092849_949078_E1A40F09 X-CRM114-Status: GOOD ( 17.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The SVA cleanup made the SSID logic entirely general so all we need to do is call it with the correct cd table entry for a S1 domain. This is slightly tricky because of the ASID and how the locking works, the simple fix is to just update the ASID once we get the right locks. Signed-off-by: Jason Gunthorpe --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 45 +++++++++++++++++++-- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 +- 2 files changed, 42 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 7e5721490b7f4c..5a0f0ea30c3f5b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1217,8 +1217,6 @@ void arm_smmu_make_s1_cd(struct arm_smmu_cd *target, typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr; - lockdep_assert_held(&master->smmu->asid_lock); - memset(target, 0, sizeof(*target)); target->data[0] = cpu_to_le64( @@ -2685,6 +2683,36 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) return 0; } +static int arm_smmu_s1_set_dev_pasid(struct iommu_domain *domain, + struct device *dev, ioasid_t id) +{ + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct arm_smmu_device *smmu = master->smmu; + struct arm_smmu_cd target_cd; + int ret = 0; + + mutex_lock(&smmu_domain->init_mutex); + if (!smmu_domain->smmu) + ret = arm_smmu_domain_finalise(smmu_domain, smmu); + else if (smmu_domain->smmu != smmu) + ret = -EINVAL; + mutex_unlock(&smmu_domain->init_mutex); + if (ret) + return ret; + + if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1) + return -EINVAL; + + /* + * We can read cd.asid outside the lock because arm_smmu_set_pasid() + * will fix it + */ + arm_smmu_make_s1_cd(&target_cd, master, smmu_domain); + return arm_smmu_set_pasid(master, to_smmu_domain(domain), id, + &target_cd); +} + static void arm_smmu_update_ste(struct arm_smmu_master *master, struct iommu_domain *sid_domain, bool want_ats) @@ -2713,7 +2741,7 @@ static void arm_smmu_update_ste(struct arm_smmu_master *master, int arm_smmu_set_pasid(struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain, ioasid_t pasid, - const struct arm_smmu_cd *cd) + struct arm_smmu_cd *cd) { struct iommu_domain *sid_domain = iommu_get_domain_for_dev(master->dev); struct arm_smmu_cd *cdptr; @@ -2737,6 +2765,14 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master, if (ret) goto out_unlock; + /* + * We don't want to obtain to the asid_lock too early, so fix up the + * caller set ASID under the lock in case it changed. + */ + cd->data[0] &= ~cpu_to_le64(CTXDESC_CD_0_ASID); + cd->data[0] |= cpu_to_le64( + FIELD_PREP(CTXDESC_CD_0_ASID, smmu_domain->cd.asid)); + arm_smmu_write_cd_entry(master, pasid, cdptr, cd); arm_smmu_update_ste(master, sid_domain, state.want_ats); @@ -2754,7 +2790,7 @@ static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) struct iommu_domain *domain; bool last_ssid = master->cd_table.used_ssids == 1; - domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA); + domain = iommu_get_domain_for_dev_pasid(dev, pasid, 0); if (WARN_ON(IS_ERR(domain)) || !domain) return; @@ -3280,6 +3316,7 @@ static struct iommu_ops arm_smmu_ops = { .owner = THIS_MODULE, .default_domain_ops = &(const struct iommu_domain_ops) { .attach_dev = arm_smmu_attach_dev, + .set_dev_pasid = arm_smmu_s1_set_dev_pasid, .map_pages = arm_smmu_map_pages, .unmap_pages = arm_smmu_unmap_pages, .flush_iotlb_all = arm_smmu_flush_iotlb_all, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 19628340ccd632..91b23437f41055 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -786,7 +786,7 @@ void arm_smmu_write_cd_entry(struct arm_smmu_master *master, int ssid, int arm_smmu_set_pasid(struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain, ioasid_t pasid, - const struct arm_smmu_cd *cd); + struct arm_smmu_cd *cd); void arm_smmu_remove_pasid(struct arm_smmu_master *master, struct arm_smmu_domain *smmu_domain, ioasid_t pasid);