From patchwork Sun Jun 27 14:34:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sven Peter X-Patchwork-Id: 12346853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D2BAC49EAB for ; Sun, 27 Jun 2021 14:39:39 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 33BAC6128A for ; Sun, 27 Jun 2021 14:39:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 33BAC6128A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=svenpeter.dev Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=vLrQCPGBdL1oaf9aIIb7LKPJAeTZViEVlGQPwjHsmXU=; b=DyVpAeRgH9mqyg OIKew4lWaFJMrgd68H8+2d05ZeFuoRgz97WceCBDtQWY4DFx0ZoDNQBxBUSilNVyuSDZ2OAp+Vpf3 pl3L7wOcjaJ588OcM9rpxGBbYGlOdHU9DB0YJcgnOJmB94Bi4V6glAHKVP2cXM+tbibkVD64I8cd4 /NyEXagReTJ8aoMI691PajxKDZXvmYHfo+ab5HAz2gT+1i8b3HeSlA2eOmoDVws7mbjXTuy/yLxUe SEKZ1ZirMMU39a+RbHF6aVgjIj0+GBz+JEuunDqZFroWZHPQaKD/OSq5LXXGDQNg+jpHxZb0wfBTX AW/+lbXjMTCgJRxb0gxA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lxVug-005hCW-Po; Sun, 27 Jun 2021 14:37:35 +0000 Received: from new4-smtp.messagingengine.com ([66.111.4.230]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lxVuc-005hAR-Ii for linux-arm-kernel@lists.infradead.org; Sun, 27 Jun 2021 14:37:32 +0000 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailnew.nyi.internal (Postfix) with ESMTP id 452C15806DA; Sun, 27 Jun 2021 10:37:26 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Sun, 27 Jun 2021 10:37:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=6dgROPTG2FgZP QhdtR6kWJqhz8OvQLzcBN89UwKC3T4=; b=v622dmoMM7/W3OCI/R8QGIKw6vVl7 NU5dPRqXCz6M4PDrvC3+PC/1Z1xex058OW1ydvH+qE/97e83UOiZOzQMpHwuJuze ZmVDmf54wFrVu/qKpTq8iqZGohx28MRq8n2i8x9LImSVwigNdtPXjKHS8IQ/A1xT S+hg9JIeH5w1QBVuKn55OIO4IGzlKe7sSk/MWe5nazvig9MFRRvIOlhKcILeo3qE grvKG/0S5df19blbtI8LTEVaKDrKXqAUWrTNwfskdpanGj+0SZ9lIKtUxltEXiAR TPb7HfVVqXte/J04ZEGKF7QdQy6Mpfeu2pzRU8mh68wXLx6u8K348KGnw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=6dgROPTG2FgZPQhdtR6kWJqhz8OvQLzcBN89UwKC3T4=; b=gjLKQ1ic fguSAuUwVKEenXTzK/coo3q+DXBo++s14UcAE9+vxdUJRvVYoZpqMXVkaCXe6IzV fg2gKosTMYMzp1XkpoB9y/CPaODu2kj7F9SgNoSABXtIgc/qJelOFmh3NCR8wJJ5 we2pQD4uDSDzA+b9+pFpraA6LsPVb6Tigc9v0upeiTidD5d8tmumJNakD5iPyEFL MaVEY5dUjmACpRPvSfVLw4qvCFV+xUBEYCELKAHNDYLPe+V4TDpmRAyivaKOZLPT PZdj1tRWnChBUziJibXDNrE3Woc1Pe++m/poweQC4SuTtBlDLKTMIduY/2uf6VYg Ax2P8LKVNXE6Zw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeehvddgjeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnhcu rfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthhtvg hrnheptedvkeetleeuffffhfekteetffeggffgveehieelueefvddtueffveevlefhfeej necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepshhvvg hnsehsvhgvnhhpvghtvghrrdguvghv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 27 Jun 2021 10:37:23 -0400 (EDT) From: Sven Peter To: Will Deacon , Robin Murphy , Joerg Roedel Cc: Sven Peter , Arnd Bergmann , devicetree@vger.kernel.org, Hector Martin , linux-kernel@vger.kernel.org, Marc Zyngier , Mohamed Mediouni , Stan Skowronek , linux-arm-kernel@lists.infradead.org, Mark Kettenis , iommu@lists.linux-foundation.org, Alexander Graf , Alyssa Rosenzweig , Rob Herring , r.czerwinski@pengutronix.de Subject: [PATCH v4 1/3] iommu: io-pgtable: add DART pagetable format Date: Sun, 27 Jun 2021 16:34:03 +0200 Message-Id: <20210627143405.77298-2-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20210627143405.77298-1-sven@svenpeter.dev> References: <20210627143405.77298-1-sven@svenpeter.dev> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210627_073730_730928_A974645C X-CRM114-Status: GOOD ( 18.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Apple's DART iommu uses a pagetable format that shares some similarities with the ones already implemented by io-pgtable.c. Add a new format variant to support the required differences so that we don't have to duplicate the pagetable handling code. Signed-off-by: Sven Peter Reviewed-by: Alexander Graf Reviewed-by: Alyssa Rosenzweig Reviewed-by: Robin Murphy --- drivers/iommu/io-pgtable-arm.c | 62 ++++++++++++++++++++++++++++++++++ drivers/iommu/io-pgtable.c | 1 + include/linux/io-pgtable.h | 7 ++++ 3 files changed, 70 insertions(+) diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index 87def58e79b5..1dd5c45b4b5b 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -127,6 +127,9 @@ #define ARM_MALI_LPAE_MEMATTR_IMP_DEF 0x88ULL #define ARM_MALI_LPAE_MEMATTR_WRITE_ALLOC 0x8DULL +#define APPLE_DART_PTE_PROT_NO_WRITE (1<<7) +#define APPLE_DART_PTE_PROT_NO_READ (1<<8) + /* IOPTE accessors */ #define iopte_deref(pte,d) __va(iopte_to_paddr(pte, d)) @@ -381,6 +384,15 @@ static arm_lpae_iopte arm_lpae_prot_to_pte(struct arm_lpae_io_pgtable *data, { arm_lpae_iopte pte; + if (data->iop.fmt == ARM_APPLE_DART) { + pte = 0; + if (!(prot & IOMMU_WRITE)) + pte |= APPLE_DART_PTE_PROT_NO_WRITE; + if (!(prot & IOMMU_READ)) + pte |= APPLE_DART_PTE_PROT_NO_READ; + return pte; + } + if (data->iop.fmt == ARM_64_LPAE_S1 || data->iop.fmt == ARM_32_LPAE_S1) { pte = ARM_LPAE_PTE_nG; @@ -1043,6 +1055,51 @@ arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie) return NULL; } +static struct io_pgtable * +apple_dart_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie) +{ + struct arm_lpae_io_pgtable *data; + int i; + + if (cfg->oas > 36) + return NULL; + + data = arm_lpae_alloc_pgtable(cfg); + if (!data) + return NULL; + + /* + * Apple's DART always requires three levels with the first level being + * stored in four MMIO registers. We always concatenate the first and + * second level so that we only have to setup the MMIO registers once. + * This results in an effective two level pagetable. + */ + if (data->start_level < 1) + return NULL; + if (data->start_level == 1 && data->pgd_bits > 2) + return NULL; + if (data->start_level > 1) + data->pgd_bits = 0; + data->start_level = 2; + cfg->apple_dart_cfg.n_ttbrs = 1 << data->pgd_bits; + data->pgd_bits += data->bits_per_level; + + data->pgd = __arm_lpae_alloc_pages(ARM_LPAE_PGD_SIZE(data), GFP_KERNEL, + cfg); + if (!data->pgd) + goto out_free_data; + + for (i = 0; i < cfg->apple_dart_cfg.n_ttbrs; ++i) + cfg->apple_dart_cfg.ttbr[i] = + virt_to_phys(data->pgd + i * ARM_LPAE_GRANULE(data)); + + return &data->iop; + +out_free_data: + kfree(data); + return NULL; +} + struct io_pgtable_init_fns io_pgtable_arm_64_lpae_s1_init_fns = { .alloc = arm_64_lpae_alloc_pgtable_s1, .free = arm_lpae_free_pgtable, @@ -1068,6 +1125,11 @@ struct io_pgtable_init_fns io_pgtable_arm_mali_lpae_init_fns = { .free = arm_lpae_free_pgtable, }; +struct io_pgtable_init_fns io_pgtable_apple_dart_init_fns = { + .alloc = apple_dart_alloc_pgtable, + .free = arm_lpae_free_pgtable, +}; + #ifdef CONFIG_IOMMU_IO_PGTABLE_LPAE_SELFTEST static struct io_pgtable_cfg *cfg_cookie __initdata; diff --git a/drivers/iommu/io-pgtable.c b/drivers/iommu/io-pgtable.c index 6e9917ce980f..fd8e6bd6caf9 100644 --- a/drivers/iommu/io-pgtable.c +++ b/drivers/iommu/io-pgtable.c @@ -20,6 +20,7 @@ io_pgtable_init_table[IO_PGTABLE_NUM_FMTS] = { [ARM_64_LPAE_S1] = &io_pgtable_arm_64_lpae_s1_init_fns, [ARM_64_LPAE_S2] = &io_pgtable_arm_64_lpae_s2_init_fns, [ARM_MALI_LPAE] = &io_pgtable_arm_mali_lpae_init_fns, + [ARM_APPLE_DART] = &io_pgtable_apple_dart_init_fns, #endif #ifdef CONFIG_IOMMU_IO_PGTABLE_ARMV7S [ARM_V7S] = &io_pgtable_arm_v7s_init_fns, diff --git a/include/linux/io-pgtable.h b/include/linux/io-pgtable.h index 4d40dfa75b55..a4bfac7f85f7 100644 --- a/include/linux/io-pgtable.h +++ b/include/linux/io-pgtable.h @@ -16,6 +16,7 @@ enum io_pgtable_fmt { ARM_V7S, ARM_MALI_LPAE, AMD_IOMMU_V1, + ARM_APPLE_DART, IO_PGTABLE_NUM_FMTS, }; @@ -136,6 +137,11 @@ struct io_pgtable_cfg { u64 transtab; u64 memattr; } arm_mali_lpae_cfg; + + struct { + u64 ttbr[4]; + u32 n_ttbrs; + } apple_dart_cfg; }; }; @@ -246,5 +252,6 @@ extern struct io_pgtable_init_fns io_pgtable_arm_64_lpae_s2_init_fns; extern struct io_pgtable_init_fns io_pgtable_arm_v7s_init_fns; extern struct io_pgtable_init_fns io_pgtable_arm_mali_lpae_init_fns; extern struct io_pgtable_init_fns io_pgtable_amd_iommu_v1_init_fns; +extern struct io_pgtable_init_fns io_pgtable_apple_dart_init_fns; #endif /* __IO_PGTABLE_H */ From patchwork Sun Jun 27 14:34:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sven Peter X-Patchwork-Id: 12346851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98827C48BC2 for ; Sun, 27 Jun 2021 14:39:38 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4A7A76128A for ; Sun, 27 Jun 2021 14:39:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4A7A76128A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=svenpeter.dev Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fpa/f4FGZozEV3QxvJq6KJWYh/px1leDVz3vNKtWTLU=; b=rFa/0dFcjGUOHN KRMoOXbx7MnyXjJbQBU7b9yQKtu740G0SwSG95Gyo1BNdFcrcV9xqedQGpJj1p5rd60Tt6giyLQuY sZDfwuP5AnIf0lOJ89bBu8sTYYS2LKqu7HijAE0qxIdBTKZfBBDmkmQCAugRuWuCTO0ggqW9Zam85 lZpY5WClTwlncP/aL1Ve+bRw3KfmNDqLHQa2ZUI/VMTtvIbeTQZf/MbZYm+QB4SbQ8q7ZNPW8LOFy 3qmrjLVZS7hPb+yfOHUPLYkavl++RVVYVVbNwLVz61xuIlR/ErrEI0btGIj/sqTh1DJxzIGpsC1Dp 4IfW+laHqjmORgDTz43g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lxVv0-005hE6-MK; Sun, 27 Jun 2021 14:37:54 +0000 Received: from new4-smtp.messagingengine.com ([66.111.4.230]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lxVue-005hB0-2G for linux-arm-kernel@lists.infradead.org; Sun, 27 Jun 2021 14:37:33 +0000 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailnew.nyi.internal (Postfix) with ESMTP id 76F8B580708; Sun, 27 Jun 2021 10:37:29 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Sun, 27 Jun 2021 10:37:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=zjKOSKJOLRXxE vp9vWIrYnPwlGb51esD71NNHk1QI88=; b=dG6VBXyBYtc/OpLWTLLnj2Qxw6lG6 J8mO4ATEh+pwa+Q+F/rYZtX7OOtgd8hpZJ0U8IBo9GfVIvqW+1XFCCLDnRXXFVD8 pvdEEZAk6SmBtVgbDjHsJcq/Gjb9PWjh2NFd74y5PtyAIpjZjE0iG+ogAEg7NMQC vtJR4KCkCNw1fIgYsAiVdz+odx/cy9j8okMbbl0r27LawSwVeAhvpeNK9wSkxjjV J1Mpos/iO2dIV/MR81LR6Xk1RKpx0HlxUf10H8dcBpMx0hfhtWFUkriF/d0P0qqt BQ07UThsk38ioDYhfxX9xQUYCk2QVLHa2zs1ZCiAKCINbU9JCFqxz9LMg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=zjKOSKJOLRXxEvp9vWIrYnPwlGb51esD71NNHk1QI88=; b=E/8SALpV Oe9O1AKtLIj8sU3pJMt0YukeRbOKat+h1gQtDBAd1UEVkxM5z3Es3XXfxeyL/YBf bsjUnPA7yRlk54AWPIEyWVBW0oiDEgauL/jpm3RnP4bzJPsApZ/NbOTp8kVM7O+x kKispA+EGBFZY27ZSZJ2XX0XcHOm2C/vxgbVBNi1TLWjmJoyaifoZklkOciok7A4 DpeWkgUN8nTlEYHGYpuE9mVx1bD3uyN+3PuNTjThrlnPJKysZ8zhM1ZkecmhWzLE 08h8Tlj/1omtbmDayTmEoDrpJet8RrjKrQXkbCWTMdwILsSYSvf2VqSKJnXvHPp/ l75m4Cnjbh9+KQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeehvddgjeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnhcu rfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthhtvg hrnhephfefffejvdekleeitdffiefhhfffveffieejteefhedutdfgffeftdejhedtteeh necuffhomhgrihhnpeguvghvihgtvghtrhgvvgdrohhrghenucevlhhushhtvghrufhiii gvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsvhgvnhesshhvvghnphgvthgvrhdr uggvvh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 27 Jun 2021 10:37:26 -0400 (EDT) From: Sven Peter To: Will Deacon , Robin Murphy , Joerg Roedel Cc: Sven Peter , Arnd Bergmann , devicetree@vger.kernel.org, Hector Martin , linux-kernel@vger.kernel.org, Marc Zyngier , Mohamed Mediouni , Stan Skowronek , linux-arm-kernel@lists.infradead.org, Mark Kettenis , iommu@lists.linux-foundation.org, Alexander Graf , Alyssa Rosenzweig , Rob Herring , r.czerwinski@pengutronix.de, Rob Herring Subject: [PATCH v4 2/3] dt-bindings: iommu: add DART iommu bindings Date: Sun, 27 Jun 2021 16:34:04 +0200 Message-Id: <20210627143405.77298-3-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20210627143405.77298-1-sven@svenpeter.dev> References: <20210627143405.77298-1-sven@svenpeter.dev> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210627_073732_216768_C1366FF1 X-CRM114-Status: GOOD ( 14.66 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DART (Device Address Resolution Table) is the iommu found on Apple ARM SoCs such as the M1. Reviewed-by: Rob Herring Signed-off-by: Sven Peter Reviewed-by: Alyssa Rosenzweig --- .../devicetree/bindings/iommu/apple,dart.yaml | 81 +++++++++++++++++++ MAINTAINERS | 6 ++ 2 files changed, 87 insertions(+) create mode 100644 Documentation/devicetree/bindings/iommu/apple,dart.yaml diff --git a/Documentation/devicetree/bindings/iommu/apple,dart.yaml b/Documentation/devicetree/bindings/iommu/apple,dart.yaml new file mode 100644 index 000000000000..94aa9e9afa59 --- /dev/null +++ b/Documentation/devicetree/bindings/iommu/apple,dart.yaml @@ -0,0 +1,81 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/iommu/apple,dart.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Apple DART IOMMU + +maintainers: + - Sven Peter + +description: |+ + Apple SoCs may contain an implementation of their Device Address + Resolution Table which provides a mandatory layer of address + translations for various masters. + + Each DART instance is capable of handling up to 16 different streams + with individual pagetables and page-level read/write protection flags. + + This DART IOMMU also raises interrupts in response to various + fault conditions. + +properties: + compatible: + const: apple,t8103-dart + + reg: + maxItems: 1 + + interrupts: + maxItems: 1 + + clocks: + description: + Reference to the gate clock phandle if required for this IOMMU. + Optional since not all IOMMUs are attached to a clock gate. + + '#iommu-cells': + const: 1 + description: + Has to be one. The single cell describes the stream id emitted by + a master to the IOMMU. + +required: + - compatible + - reg + - '#iommu-cells' + - interrupts + +additionalProperties: false + +examples: + - |+ + dart1: iommu@82f80000 { + compatible = "apple,t8103-dart"; + reg = <0x82f80000 0x4000>; + interrupts = <1 781 4>; + #iommu-cells = <1>; + }; + + master1 { + iommus = <&dart1 0>; + }; + + - |+ + dart2a: iommu@82f00000 { + compatible = "apple,t8103-dart"; + reg = <0x82f00000 0x4000>; + interrupts = <1 781 4>; + #iommu-cells = <1>; + }; + dart2b: iommu@82f80000 { + compatible = "apple,t8103-dart"; + reg = <0x82f80000 0x4000>; + interrupts = <1 781 4>; + #iommu-cells = <1>; + }; + + master2 { + iommus = <&dart2a 0>, <&dart2b 1>; + }; diff --git a/MAINTAINERS b/MAINTAINERS index 8c5ee008301a..29e5541c8f21 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1240,6 +1240,12 @@ L: linux-input@vger.kernel.org S: Odd fixes F: drivers/input/mouse/bcm5974.c +APPLE DART IOMMU DRIVER +M: Sven Peter +L: iommu@lists.linux-foundation.org +S: Maintained +F: Documentation/devicetree/bindings/iommu/apple,dart.yaml + APPLE SMC DRIVER M: Henrik Rydberg L: linux-hwmon@vger.kernel.org From patchwork Sun Jun 27 14:34:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sven Peter X-Patchwork-Id: 12346857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A196DC48BC2 for ; Sun, 27 Jun 2021 14:39:57 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 62D4E619F1 for ; Sun, 27 Jun 2021 14:39:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 62D4E619F1 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=svenpeter.dev Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=L58UMK9N5/lrpxp6vTRN/UGakFrTd/WgNzSeXbmkCQc=; b=PSqPY/Gvd3EuS9 I5EzjspxOEWjB8pXnNMux0MHdh3xeVaymSkMwR/xmgPZzNizJ03gV50M0cbyD0TII+3jZdG6KC0Fd Ietr9mIu8d1fUpHzbqHFUSSXur3qIANbxh0lNESXhtQe4mNgAa81qhEYf0UrGS4dUaWxNuy/u3B3D vIKmzKnvQ70OgT0fHCe6RjKYyJ7igtoE3WTXh+oZIiRQhEmmco23CXizTUuJkbP6EYlCYnZrz7SEx BEAwFQ2pB0hdxAM12kB16NVOq8pgDW78CxNKCnuzyewe/J5ISDI1DqqWbtiuZEfBwQBq8JZoeLtbO P8Pq4gXvJfXmFGgAVYWw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lxVvB-005hFF-7u; Sun, 27 Jun 2021 14:38:05 +0000 Received: from new4-smtp.messagingengine.com ([66.111.4.230]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lxVuf-005hC8-2r for linux-arm-kernel@lists.infradead.org; Sun, 27 Jun 2021 14:37:36 +0000 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailnew.nyi.internal (Postfix) with ESMTP id 66A8158070D; Sun, 27 Jun 2021 10:37:32 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Sun, 27 Jun 2021 10:37:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=oEA9go9wU7b6H mn5NofkFuUSkA6XvVEbXEtpGTPcAFQ=; b=4h4GwJA5XM+I1xkMmKSyGMmSP1n33 zuq/25QGKqaMH4/soDqzoHkSesQNmV6CItFh3ELmo1J1it4uFqAV+mWHv5RH3M0k 5Njp0MYgRMj5LRjH5StpEfPhOG5qFkJtlQ2vabNQ+PqhqEjKgFDtFB2oY37+xGsP 9z8GXNlSk73CYXPQJPWgpC32B1bE0ZtkKorQ2/8NY55fblMVLjgQ4qYvrByHd4NI LaIKJao6wlvCpGP+lM25xQPFioW0h+6uzn2HE8HH57ncF2WG6mQZr4lRpxQK8YXJ 80PStG/rCbdUKgKii+NWzt6hJ9jbk6a42g1+0STJmHD7P6A5WiKNfCOvw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=oEA9go9wU7b6Hmn5NofkFuUSkA6XvVEbXEtpGTPcAFQ=; b=R4cAx7Nz lVaNXV4ul8/0AnV0jCZqsbDwVx7CGQbS2QS/bdFQsGYazBdZbED9MZUSlep50UQY YjekX1gVETMrwdmIEEFjbPNa0vGdNxXtyjZcxRAaQ5szLMT+NeIdW77NOnU+Ti1w Kx4q9KOoCRLcAY9MEdKgDePaakn2k73lCScCx4TuuSHUt1J5nsIRP8e6M7wVWpqj UnBBDqNq8DZl7/vf2pvvlVm+myBo4cIDbsNaszVg5nQMJc/A3J0yRn7ihzqL9n7F UgPLh2KP0tsRu3rWG9lfXj2Mw+7oqxNhyIBflfOG2v9gdCBcOBNKPMFlBj926+ky XArse50dUktLWw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeehvddgjeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnhcu rfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthhtvg hrnheptedvkeetleeuffffhfekteetffeggffgveehieelueefvddtueffveevlefhfeej necuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepshhvvg hnsehsvhgvnhhpvghtvghrrdguvghv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 27 Jun 2021 10:37:29 -0400 (EDT) From: Sven Peter To: Will Deacon , Robin Murphy , Joerg Roedel Cc: Sven Peter , Arnd Bergmann , devicetree@vger.kernel.org, Hector Martin , linux-kernel@vger.kernel.org, Marc Zyngier , Mohamed Mediouni , Stan Skowronek , linux-arm-kernel@lists.infradead.org, Mark Kettenis , iommu@lists.linux-foundation.org, Alexander Graf , Alyssa Rosenzweig , Rob Herring , r.czerwinski@pengutronix.de Subject: [PATCH v4 3/3] iommu: dart: Add DART iommu driver Date: Sun, 27 Jun 2021 16:34:05 +0200 Message-Id: <20210627143405.77298-4-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20210627143405.77298-1-sven@svenpeter.dev> References: <20210627143405.77298-1-sven@svenpeter.dev> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210627_073733_331098_0DBAFA56 X-CRM114-Status: GOOD ( 33.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Apple's new SoCs use iommus for almost all peripherals. These Device Address Resolution Tables must be setup before these peripherals can act as DMA masters. Signed-off-by: Sven Peter --- MAINTAINERS | 1 + drivers/iommu/Kconfig | 15 + drivers/iommu/Makefile | 1 + drivers/iommu/apple-dart-iommu.c | 1058 ++++++++++++++++++++++++++++++ 4 files changed, 1075 insertions(+) create mode 100644 drivers/iommu/apple-dart-iommu.c diff --git a/MAINTAINERS b/MAINTAINERS index 29e5541c8f21..c1ffaa56b5f9 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1245,6 +1245,7 @@ M: Sven Peter L: iommu@lists.linux-foundation.org S: Maintained F: Documentation/devicetree/bindings/iommu/apple,dart.yaml +F: drivers/iommu/apple-dart-iommu.c APPLE SMC DRIVER M: Henrik Rydberg diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index 1f111b399bca..87882c628b46 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -249,6 +249,21 @@ config SPAPR_TCE_IOMMU Enables bits of IOMMU API required by VFIO. The iommu_ops is not implemented as it is not necessary for VFIO. +config IOMMU_APPLE_DART + tristate "Apple DART IOMMU Support" + depends on ARM64 || (COMPILE_TEST && !GENERIC_ATOMIC64) + select IOMMU_API + select IOMMU_IO_PGTABLE + select IOMMU_IO_PGTABLE_LPAE + default ARCH_APPLE + help + Support for Apple DART (Device Address Resolution Table) IOMMUs + found in Apple ARM SoCs like the M1. + This IOMMU is required for most peripherals using DMA to access + the main memory. + + Say Y here if you are using an Apple SoC with a DART IOMMU. + # ARM IOMMU support config ARM_SMMU tristate "ARM Ltd. System MMU (SMMU) Support" diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile index c0fb0ba88143..8c813f0ebc54 100644 --- a/drivers/iommu/Makefile +++ b/drivers/iommu/Makefile @@ -29,3 +29,4 @@ obj-$(CONFIG_HYPERV_IOMMU) += hyperv-iommu.o obj-$(CONFIG_VIRTIO_IOMMU) += virtio-iommu.o obj-$(CONFIG_IOMMU_SVA_LIB) += iommu-sva-lib.o io-pgfault.o obj-$(CONFIG_SPRD_IOMMU) += sprd-iommu.o +obj-$(CONFIG_IOMMU_APPLE_DART) += apple-dart-iommu.o diff --git a/drivers/iommu/apple-dart-iommu.c b/drivers/iommu/apple-dart-iommu.c new file mode 100644 index 000000000000..637ba6e7cef9 --- /dev/null +++ b/drivers/iommu/apple-dart-iommu.c @@ -0,0 +1,1058 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Apple DART (Device Address Resolution Table) IOMMU driver + * + * Copyright (C) 2021 The Asahi Linux Contributors + * + * Based on arm/arm-smmu/arm-ssmu.c and arm/arm-smmu-v3/arm-smmu-v3.c + * Copyright (C) 2013 ARM Limited + * Copyright (C) 2015 ARM Limited + * and on exynos-iommu.c + * Copyright (c) 2011,2016 Samsung Electronics Co., Ltd. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define DART_MAX_STREAMS 16 +#define DART_MAX_TTBR 4 + +#define DART_STREAM_ALL 0xffff + +#define DART_PARAMS1 0x00 +#define DART_PARAMS_PAGE_SHIFT GENMASK(27, 24) + +#define DART_PARAMS2 0x04 +#define DART_PARAMS_BYPASS_SUPPORT BIT(0) + +#define DART_STREAM_COMMAND 0x20 +#define DART_STREAM_COMMAND_BUSY BIT(2) +#define DART_STREAM_COMMAND_INVALIDATE BIT(20) + +#define DART_STREAM_SELECT 0x34 + +#define DART_ERROR 0x40 +#define DART_ERROR_STREAM GENMASK(27, 24) +#define DART_ERROR_CODE GENMASK(23, 0) +#define DART_ERROR_FLAG BIT(31) +#define DART_ERROR_READ_FAULT BIT(4) +#define DART_ERROR_WRITE_FAULT BIT(3) +#define DART_ERROR_NO_PTE BIT(2) +#define DART_ERROR_NO_PMD BIT(1) +#define DART_ERROR_NO_TTBR BIT(0) + +#define DART_CONFIG 0x60 +#define DART_CONFIG_LOCK BIT(15) + +#define DART_STREAM_COMMAND_BUSY_TIMEOUT 100 + +#define DART_STREAM_REMAP 0x80 + +#define DART_ERROR_ADDR_HI 0x54 +#define DART_ERROR_ADDR_LO 0x50 + +#define DART_TCR(sid) (0x100 + 4 * (sid)) +#define DART_TCR_TRANSLATE_ENABLE BIT(7) +#define DART_TCR_BYPASS0_ENABLE BIT(8) +#define DART_TCR_BYPASS1_ENABLE BIT(12) + +#define DART_TTBR(sid, idx) (0x200 + 16 * (sid) + 4 * (idx)) +#define DART_TTBR_VALID BIT(31) +#define DART_TTBR_SHIFT 12 + +/* + * Private structure associated with each DART device. + * + * @dev: device struct + * @regs: mapped MMIO region + * @irq: interrupt number, can be shared with other DARTs + * @clks: clocks associated with this DART + * @num_clks: number of @clks + * @lock: lock for @used_sids and hardware operations involving this dart + * @used_sids: bitmap of streams attached to a domain + * @pgsize: pagesize supported by this DART + * @supports_bypass: indicates if this DART supports bypass mode + * @force_bypass: force bypass mode due to pagesize mismatch? + * @sw_bypass_cpu_start: offset into cpu address space in software bypass mode + * @sw_bypass_dma_start: offset into dma address space in software bypass mode + * @sw_bypass_len: length of iova space in software bypass mode + * @iommu: iommu core device + */ +struct apple_dart { + struct device *dev; + + void __iomem *regs; + + int irq; + struct clk_bulk_data *clks; + int num_clks; + + spinlock_t lock; + + u32 used_sids; + u32 pgsize; + + u32 supports_bypass : 1; + u32 force_bypass : 1; + + u64 sw_bypass_cpu_start; + u64 sw_bypass_dma_start; + u64 sw_bypass_len; + + struct iommu_device iommu; +}; + +/* + * This structure is used to identify a single stream attached to a domain. + * It's used as a list inside that domain to be able to attach multiple + * streams to a single domain. Since multiple devices can use a single stream + * it additionally keeps track of how many devices are represented by this + * stream. Once that number reaches zero it is detached from the IOMMU domain + * and all translations from this stream are disabled. + * + * @dart: DART instance to which this stream belongs + * @sid: stream id within the DART instance + * @num_devices: count of devices attached to this stream + * @stream_head: list head for the next stream + */ +struct apple_dart_stream { + struct apple_dart *dart; + u32 sid; + + u32 num_devices; + + struct list_head stream_head; +}; + +/* + * This structure is attached to each iommu domain handled by a DART. + * A single domain is used to represent a single virtual address space. + * It is always allocated together with a page table. + * + * Streams are the smallest units the DART hardware can differentiate. + * These are pointed to the page table of a domain whenever a device is + * attached to it. A single stream can only be assigned to a single domain. + * + * Devices are assigned to at least a single and sometimes multiple individual + * streams (using the iommus property in the device tree). Multiple devices + * can theoretically be represented by the same stream, though this is usually + * not the case. + * + * We only keep track of streams here and just count how many devices are + * represented by each stream. When the last device is removed the whole stream + * is removed from the domain. + * + * @dart: pointer to the DART instance + * @pgtbl_ops: pagetable ops allocated by io-pgtable + * @type: domain type IOMMU_DOMAIN_IDENTITY_{IDENTITY,DMA,UNMANAGED,BLOCKED} + * @sw_bypass_cpu_start: offset into cpu address space in software bypass mode + * @sw_bypass_dma_start: offset into dma address space in software bypass mode + * @sw_bypass_len: length of iova space in software bypass mode + * @streams: list of streams attached to this domain + * @lock: spinlock for operations involving the list of streams + * @domain: core iommu domain pointer + */ +struct apple_dart_domain { + struct apple_dart *dart; + struct io_pgtable_ops *pgtbl_ops; + + unsigned int type; + + u64 sw_bypass_cpu_start; + u64 sw_bypass_dma_start; + u64 sw_bypass_len; + + struct list_head streams; + + spinlock_t lock; + + struct iommu_domain domain; +}; + +/* + * This structure is attached to devices with dev_iommu_priv_set() on of_xlate + * and contains a list of streams bound to this device as defined in the + * device tree. Multiple DART instances can be attached to a single device + * and each stream is identified by its stream id. + * It's usually reference by a pointer called *cfg. + * + * A dynamic array instead of a linked list is used here since in almost + * all cases a device will just be attached to a single stream and streams + * are never removed after they have been added. + * + * @num_streams: number of streams attached + * @streams: array of structs to identify attached streams and the device link + * to the iommu + */ +struct apple_dart_master_cfg { + int num_streams; + struct { + struct apple_dart *dart; + u32 sid; + + struct device_link *link; + } streams[]; +}; + +static struct platform_driver apple_dart_driver; +static const struct iommu_ops apple_dart_iommu_ops; +static const struct iommu_flush_ops apple_dart_tlb_ops; + +static struct apple_dart_domain *to_dart_domain(struct iommu_domain *dom) +{ + return container_of(dom, struct apple_dart_domain, domain); +} + +static void apple_dart_hw_enable_translation(struct apple_dart *dart, u16 sid) +{ + writel(DART_TCR_TRANSLATE_ENABLE, dart->regs + DART_TCR(sid)); +} + +static void apple_dart_hw_disable_dma(struct apple_dart *dart, u16 sid) +{ + writel(0, dart->regs + DART_TCR(sid)); +} + +static void apple_dart_hw_enable_bypass(struct apple_dart *dart, u16 sid) +{ + WARN_ON(!dart->supports_bypass); + writel(DART_TCR_BYPASS0_ENABLE | DART_TCR_BYPASS1_ENABLE, + dart->regs + DART_TCR(sid)); +} + +static void apple_dart_hw_set_ttbr(struct apple_dart *dart, u16 sid, u16 idx, + phys_addr_t paddr) +{ + writel(DART_TTBR_VALID | (paddr >> DART_TTBR_SHIFT), + dart->regs + DART_TTBR(sid, idx)); +} + +static void apple_dart_hw_clear_ttbr(struct apple_dart *dart, u16 sid, u16 idx) +{ + writel(0, dart->regs + DART_TTBR(sid, idx)); +} + +static void apple_dart_hw_clear_all_ttbrs(struct apple_dart *dart, u16 sid) +{ + int i; + + for (i = 0; i < 4; ++i) + apple_dart_hw_clear_ttbr(dart, sid, i); +} + +static int apple_dart_hw_stream_command(struct apple_dart *dart, u16 sid_bitmap, + u32 command) +{ + unsigned long flags; + int ret; + u32 command_reg; + + spin_lock_irqsave(&dart->lock, flags); + + writel(sid_bitmap, dart->regs + DART_STREAM_SELECT); + writel(command, dart->regs + DART_STREAM_COMMAND); + + ret = readl_poll_timeout_atomic( + dart->regs + DART_STREAM_COMMAND, command_reg, + !(command_reg & DART_STREAM_COMMAND_BUSY), 1, + DART_STREAM_COMMAND_BUSY_TIMEOUT); + + spin_unlock_irqrestore(&dart->lock, flags); + + if (ret) { + dev_err(dart->dev, + "busy bit did not clear after command %x for streams %x\n", + command, sid_bitmap); + return ret; + } + + return 0; +} + +static int apple_dart_hw_invalidate_tlb_global(struct apple_dart *dart) +{ + return apple_dart_hw_stream_command(dart, DART_STREAM_ALL, + DART_STREAM_COMMAND_INVALIDATE); +} + +static int apple_dart_hw_invalidate_tlb_stream(struct apple_dart *dart, u16 sid) +{ + return apple_dart_hw_stream_command(dart, 1 << sid, + DART_STREAM_COMMAND_INVALIDATE); +} + +static int apple_dart_hw_reset(struct apple_dart *dart) +{ + int sid; + u32 config; + + config = readl(dart->regs + DART_CONFIG); + if (config & DART_CONFIG_LOCK) { + dev_err(dart->dev, "DART is locked down until reboot: %08x\n", + config); + return -EINVAL; + } + + for (sid = 0; sid < DART_MAX_STREAMS; ++sid) { + apple_dart_hw_disable_dma(dart, sid); + apple_dart_hw_clear_all_ttbrs(dart, sid); + } + + /* restore stream identity map */ + writel(0x03020100, dart->regs + DART_STREAM_REMAP); + writel(0x07060504, dart->regs + DART_STREAM_REMAP + 4); + writel(0x0b0a0908, dart->regs + DART_STREAM_REMAP + 8); + writel(0x0f0e0d0c, dart->regs + DART_STREAM_REMAP + 12); + + /* clear any pending errors before the interrupt is unmasked */ + writel(readl(dart->regs + DART_ERROR), dart->regs + DART_ERROR); + + return apple_dart_hw_invalidate_tlb_global(dart); +} + +static void apple_dart_domain_flush_tlb(struct apple_dart_domain *domain) +{ + unsigned long flags; + struct apple_dart_stream *stream; + struct apple_dart *dart = domain->dart; + + if (!dart) + return; + + spin_lock_irqsave(&domain->lock, flags); + list_for_each_entry(stream, &domain->streams, stream_head) { + apple_dart_hw_invalidate_tlb_stream(stream->dart, stream->sid); + } + spin_unlock_irqrestore(&domain->lock, flags); +} + +static void apple_dart_flush_iotlb_all(struct iommu_domain *domain) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + + apple_dart_domain_flush_tlb(dart_domain); +} + +static void apple_dart_iotlb_sync(struct iommu_domain *domain, + struct iommu_iotlb_gather *gather) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + + apple_dart_domain_flush_tlb(dart_domain); +} + +static void apple_dart_iotlb_sync_map(struct iommu_domain *domain, + unsigned long iova, size_t size) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + + apple_dart_domain_flush_tlb(dart_domain); +} + +static void apple_dart_tlb_flush_all(void *cookie) +{ + struct apple_dart_domain *domain = cookie; + + apple_dart_domain_flush_tlb(domain); +} + +static void apple_dart_tlb_flush_walk(unsigned long iova, size_t size, + size_t granule, void *cookie) +{ + struct apple_dart_domain *domain = cookie; + + apple_dart_domain_flush_tlb(domain); +} + +static const struct iommu_flush_ops apple_dart_tlb_ops = { + .tlb_flush_all = apple_dart_tlb_flush_all, + .tlb_flush_walk = apple_dart_tlb_flush_walk, + .tlb_add_page = NULL, +}; + +static phys_addr_t apple_dart_iova_to_phys(struct iommu_domain *domain, + dma_addr_t iova) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + struct io_pgtable_ops *ops = dart_domain->pgtbl_ops; + + if (domain->type == IOMMU_DOMAIN_IDENTITY && + dart_domain->dart->supports_bypass) + return iova; + if (!ops) + return -ENODEV; + + return ops->iova_to_phys(ops, iova); +} + +static int apple_dart_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + struct io_pgtable_ops *ops = dart_domain->pgtbl_ops; + + if (!ops) + return -ENODEV; + if (prot & IOMMU_MMIO) + return -EINVAL; + if (prot & IOMMU_NOEXEC) + return -EINVAL; + + return ops->map(ops, iova, paddr, size, prot, gfp); +} + +static size_t apple_dart_unmap(struct iommu_domain *domain, unsigned long iova, + size_t size, struct iommu_iotlb_gather *gather) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + struct io_pgtable_ops *ops = dart_domain->pgtbl_ops; + + if (!ops) + return 0; + + return ops->unmap(ops, iova, size, gather); +} + +static int apple_dart_prepare_sw_bypass(struct apple_dart *dart, + struct apple_dart_domain *dart_domain, + struct device *dev) +{ + lockdep_assert_held(&dart_domain->lock); + + if (dart->supports_bypass) + return 0; + if (dart_domain->type != IOMMU_DOMAIN_IDENTITY) + return 0; + + // use the bus region from the first attached dev for the bypass range + if (!dart->sw_bypass_len) { + const struct bus_dma_region *dma_rgn = dev->dma_range_map; + + if (!dma_rgn) + return -EINVAL; + + dart->sw_bypass_len = dma_rgn->size; + dart->sw_bypass_cpu_start = dma_rgn->cpu_start; + dart->sw_bypass_dma_start = dma_rgn->dma_start; + } + + // ensure that we don't mix different bypass setups + if (dart_domain->sw_bypass_len) { + if (dart->sw_bypass_len != dart_domain->sw_bypass_len) + return -EINVAL; + if (dart->sw_bypass_cpu_start != + dart_domain->sw_bypass_cpu_start) + return -EINVAL; + if (dart->sw_bypass_dma_start != + dart_domain->sw_bypass_dma_start) + return -EINVAL; + } else { + dart_domain->sw_bypass_len = dart->sw_bypass_len; + dart_domain->sw_bypass_cpu_start = dart->sw_bypass_cpu_start; + dart_domain->sw_bypass_dma_start = dart->sw_bypass_dma_start; + } + + return 0; +} + +static int apple_dart_domain_needs_pgtbl_ops(struct apple_dart *dart, + struct iommu_domain *domain) +{ + if (domain->type == IOMMU_DOMAIN_DMA) + return 1; + if (domain->type == IOMMU_DOMAIN_UNMANAGED) + return 1; + if (!dart->supports_bypass && domain->type == IOMMU_DOMAIN_IDENTITY) + return 1; + return 0; +} + +static int apple_dart_finalize_domain(struct iommu_domain *domain) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + struct apple_dart *dart = dart_domain->dart; + struct io_pgtable_cfg pgtbl_cfg; + + lockdep_assert_held(&dart_domain->lock); + + if (dart_domain->pgtbl_ops) + return 0; + if (!apple_dart_domain_needs_pgtbl_ops(dart, domain)) + return 0; + + pgtbl_cfg = (struct io_pgtable_cfg){ + .pgsize_bitmap = dart->pgsize, + .ias = 32, + .oas = 36, + .coherent_walk = 1, + .tlb = &apple_dart_tlb_ops, + .iommu_dev = dart->dev, + }; + + dart_domain->pgtbl_ops = + alloc_io_pgtable_ops(ARM_APPLE_DART, &pgtbl_cfg, domain); + if (!dart_domain->pgtbl_ops) + return -ENOMEM; + + domain->pgsize_bitmap = pgtbl_cfg.pgsize_bitmap; + domain->geometry.aperture_start = 0; + domain->geometry.aperture_end = DMA_BIT_MASK(32); + domain->geometry.force_aperture = true; + + /* + * Some DARTs come without hardware bypass support but we may still + * be forced to use bypass mode (to e.g. allow kernels with 4K pages to + * boot). If we reach this point with an identity domain we have to setup + * bypass mode in software. This is done by creating a static pagetable + * for a linear map specified by dma-ranges in the device tree. + */ + if (domain->type == IOMMU_DOMAIN_IDENTITY) { + u64 offset; + int ret; + + for (offset = 0; offset < dart_domain->sw_bypass_len; + offset += dart->pgsize) { + ret = dart_domain->pgtbl_ops->map( + dart_domain->pgtbl_ops, + dart_domain->sw_bypass_dma_start + offset, + dart_domain->sw_bypass_cpu_start + offset, + dart->pgsize, IOMMU_READ | IOMMU_WRITE, + GFP_ATOMIC); + if (ret < 0) { + free_io_pgtable_ops(dart_domain->pgtbl_ops); + dart_domain->pgtbl_ops = NULL; + return -EINVAL; + } + } + } + + return 0; +} + +static void +apple_dart_stream_setup_translation(struct apple_dart_domain *domain, + struct apple_dart *dart, u32 sid) +{ + int i; + struct io_pgtable_cfg *pgtbl_cfg = + &io_pgtable_ops_to_pgtable(domain->pgtbl_ops)->cfg; + + for (i = 0; i < pgtbl_cfg->apple_dart_cfg.n_ttbrs; ++i) + apple_dart_hw_set_ttbr(dart, sid, i, + pgtbl_cfg->apple_dart_cfg.ttbr[i]); + for (; i < DART_MAX_TTBR; ++i) + apple_dart_hw_clear_ttbr(dart, sid, i); + + apple_dart_hw_enable_translation(dart, sid); + apple_dart_hw_invalidate_tlb_stream(dart, sid); +} + +static int apple_dart_attach_stream(struct apple_dart_domain *domain, + struct apple_dart *dart, u32 sid) +{ + unsigned long flags; + struct apple_dart_stream *stream; + int ret; + + lockdep_assert_held(&domain->lock); + + if (WARN_ON(dart->force_bypass && + domain->type != IOMMU_DOMAIN_IDENTITY)) + return -EINVAL; + + /* + * we can't mix and match DARTs that support bypass mode with those who don't + * because the iova space in fake bypass mode generally has an offset + */ + if (WARN_ON(domain->type == IOMMU_DOMAIN_IDENTITY && + (domain->dart->supports_bypass != dart->supports_bypass))) + return -EINVAL; + + list_for_each_entry(stream, &domain->streams, stream_head) { + if (stream->dart == dart && stream->sid == sid) { + stream->num_devices++; + return 0; + } + } + + spin_lock_irqsave(&dart->lock, flags); + + if (WARN_ON(dart->used_sids & BIT(sid))) { + ret = -EINVAL; + goto error; + } + + stream = kzalloc(sizeof(*stream), GFP_ATOMIC); + if (!stream) { + ret = -ENOMEM; + goto error; + } + + stream->dart = dart; + stream->sid = sid; + stream->num_devices = 1; + list_add(&stream->stream_head, &domain->streams); + + dart->used_sids |= BIT(sid); + spin_unlock_irqrestore(&dart->lock, flags); + + apple_dart_hw_clear_all_ttbrs(stream->dart, stream->sid); + + switch (domain->type) { + case IOMMU_DOMAIN_IDENTITY: + if (stream->dart->supports_bypass) + apple_dart_hw_enable_bypass(stream->dart, stream->sid); + else + apple_dart_stream_setup_translation( + domain, stream->dart, stream->sid); + break; + case IOMMU_DOMAIN_BLOCKED: + apple_dart_hw_disable_dma(stream->dart, stream->sid); + break; + case IOMMU_DOMAIN_UNMANAGED: + case IOMMU_DOMAIN_DMA: + apple_dart_stream_setup_translation(domain, stream->dart, + stream->sid); + break; + } + + return 0; + +error: + spin_unlock_irqrestore(&dart->lock, flags); + return ret; +} + +static void apple_dart_disable_stream(struct apple_dart *dart, u32 sid) +{ + unsigned long flags; + + apple_dart_hw_disable_dma(dart, sid); + apple_dart_hw_clear_all_ttbrs(dart, sid); + apple_dart_hw_invalidate_tlb_stream(dart, sid); + + spin_lock_irqsave(&dart->lock, flags); + dart->used_sids &= ~BIT(sid); + spin_unlock_irqrestore(&dart->lock, flags); +} + +static void apple_dart_detach_stream(struct apple_dart_domain *domain, + struct apple_dart *dart, u32 sid) +{ + struct apple_dart_stream *stream; + + lockdep_assert_held(&domain->lock); + + list_for_each_entry(stream, &domain->streams, stream_head) { + if (stream->dart == dart && stream->sid == sid) { + stream->num_devices--; + + if (stream->num_devices == 0) { + apple_dart_disable_stream(dart, sid); + list_del(&stream->stream_head); + kfree(stream); + } + return; + } + } +} + +static int apple_dart_attach_dev(struct iommu_domain *domain, + struct device *dev) +{ + int ret; + int i, j; + unsigned long flags; + struct apple_dart_master_cfg *cfg = dev_iommu_priv_get(dev); + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + struct apple_dart *dart = cfg->streams[0].dart; + + if (WARN_ON(dart->force_bypass && + dart_domain->type != IOMMU_DOMAIN_IDENTITY)) { + dev_warn( + dev, + "IOMMU must be in bypass mode but trying to attach to translated domain.\n"); + return -EINVAL; + } + + spin_lock_irqsave(&dart_domain->lock, flags); + + ret = apple_dart_prepare_sw_bypass(dart, dart_domain, dev); + if (ret) + goto out; + + if (!dart_domain->dart) + dart_domain->dart = dart; + + ret = apple_dart_finalize_domain(domain); + if (ret) + goto out; + + for (i = 0; i < cfg->num_streams; ++i) { + ret = apple_dart_attach_stream( + dart_domain, cfg->streams[i].dart, cfg->streams[i].sid); + if (ret) { + /* try to undo what we did before returning */ + for (j = 0; j < i; ++j) + apple_dart_detach_stream(dart_domain, + cfg->streams[j].dart, + cfg->streams[j].sid); + + goto out; + } + } + + ret = 0; + +out: + spin_unlock_irqrestore(&dart_domain->lock, flags); + return ret; +} + +static void apple_dart_detach_dev(struct iommu_domain *domain, + struct device *dev) +{ + int i; + unsigned long flags; + struct apple_dart_master_cfg *cfg = dev_iommu_priv_get(dev); + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + + spin_lock_irqsave(&dart_domain->lock, flags); + + for (i = 0; i < cfg->num_streams; ++i) + apple_dart_detach_stream(dart_domain, cfg->streams[i].dart, + cfg->streams[i].sid); + + spin_unlock_irqrestore(&dart_domain->lock, flags); +} + +static struct iommu_device *apple_dart_probe_device(struct device *dev) +{ + struct apple_dart_master_cfg *cfg = dev_iommu_priv_get(dev); + int i; + + if (!cfg) + return ERR_PTR(-ENODEV); + + for (i = 0; i < cfg->num_streams; ++i) { + cfg->streams[i].link = + device_link_add(dev, cfg->streams[i].dart->dev, + DL_FLAG_PM_RUNTIME | DL_FLAG_STATELESS); + } + + return &cfg->streams[0].dart->iommu; +} + +static void apple_dart_release_device(struct device *dev) +{ + struct apple_dart_master_cfg *cfg = dev_iommu_priv_get(dev); + int i; + + if (!cfg) + return; + + for (i = 0; i < cfg->num_streams; ++i) + device_link_del(cfg->streams[i].link); + + dev_iommu_priv_set(dev, NULL); + kfree(cfg); +} + +static struct iommu_domain *apple_dart_domain_alloc(unsigned int type) +{ + struct apple_dart_domain *dart_domain; + + if (type != IOMMU_DOMAIN_DMA && type != IOMMU_DOMAIN_UNMANAGED && + type != IOMMU_DOMAIN_IDENTITY && type != IOMMU_DOMAIN_BLOCKED) + return NULL; + + dart_domain = kzalloc(sizeof(*dart_domain), GFP_KERNEL); + if (!dart_domain) + return NULL; + + INIT_LIST_HEAD(&dart_domain->streams); + spin_lock_init(&dart_domain->lock); + iommu_get_dma_cookie(&dart_domain->domain); + dart_domain->type = type; + + return &dart_domain->domain; +} + +static void apple_dart_domain_free(struct iommu_domain *domain) +{ + struct apple_dart_domain *dart_domain = to_dart_domain(domain); + + WARN_ON(!list_empty(&dart_domain->streams)); + + kfree(dart_domain); +} + +static int apple_dart_of_xlate(struct device *dev, struct of_phandle_args *args) +{ + struct platform_device *iommu_pdev = of_find_device_by_node(args->np); + struct apple_dart_master_cfg *cfg = dev_iommu_priv_get(dev); + unsigned int num_streams = cfg ? cfg->num_streams : 0; + struct apple_dart_master_cfg *cfg_new; + struct apple_dart *dart = platform_get_drvdata(iommu_pdev); + + if (args->args_count != 1) + return -EINVAL; + + cfg_new = krealloc(cfg, struct_size(cfg, streams, num_streams + 1), + GFP_KERNEL); + if (!cfg_new) + return -ENOMEM; + + cfg = cfg_new; + dev_iommu_priv_set(dev, cfg); + + cfg->num_streams = num_streams; + cfg->streams[cfg->num_streams].dart = dart; + cfg->streams[cfg->num_streams].sid = args->args[0]; + cfg->num_streams++; + + return 0; +} + +static struct iommu_group *apple_dart_device_group(struct device *dev) +{ +#ifdef CONFIG_PCI + struct iommu_group *group; + + if (dev_is_pci(dev)) + group = pci_device_group(dev); + else + group = generic_device_group(dev); + + return group; +#else + return generic_device_group(dev); +#endif +} + +static int apple_dart_def_domain_type(struct device *dev) +{ + struct apple_dart_master_cfg *cfg = dev_iommu_priv_get(dev); + struct apple_dart *dart = cfg->streams[0].dart; + + if (dart->force_bypass) + return IOMMU_DOMAIN_IDENTITY; + if (!dart->supports_bypass) + return IOMMU_DOMAIN_DMA; + + return 0; +} + +static const struct iommu_ops apple_dart_iommu_ops = { + .domain_alloc = apple_dart_domain_alloc, + .domain_free = apple_dart_domain_free, + .attach_dev = apple_dart_attach_dev, + .detach_dev = apple_dart_detach_dev, + .map = apple_dart_map, + .unmap = apple_dart_unmap, + .flush_iotlb_all = apple_dart_flush_iotlb_all, + .iotlb_sync = apple_dart_iotlb_sync, + .iotlb_sync_map = apple_dart_iotlb_sync_map, + .iova_to_phys = apple_dart_iova_to_phys, + .probe_device = apple_dart_probe_device, + .release_device = apple_dart_release_device, + .device_group = apple_dart_device_group, + .of_xlate = apple_dart_of_xlate, + .def_domain_type = apple_dart_def_domain_type, + .pgsize_bitmap = -1UL, /* Restricted during dart probe */ +}; + +static irqreturn_t apple_dart_irq(int irq, void *dev) +{ + struct apple_dart *dart = dev; + static DEFINE_RATELIMIT_STATE(rs, DEFAULT_RATELIMIT_INTERVAL, + DEFAULT_RATELIMIT_BURST); + const char *fault_name = NULL; + u32 error = readl(dart->regs + DART_ERROR); + u32 error_code = FIELD_GET(DART_ERROR_CODE, error); + u32 addr_lo = readl(dart->regs + DART_ERROR_ADDR_LO); + u32 addr_hi = readl(dart->regs + DART_ERROR_ADDR_HI); + u64 addr = addr_lo | (((u64)addr_hi) << 32); + u8 stream_idx = FIELD_GET(DART_ERROR_STREAM, error); + + if (!(error & DART_ERROR_FLAG)) + return IRQ_NONE; + + if (error_code & DART_ERROR_READ_FAULT) + fault_name = "READ FAULT"; + else if (error_code & DART_ERROR_WRITE_FAULT) + fault_name = "WRITE FAULT"; + else if (error_code & DART_ERROR_NO_PTE) + fault_name = "NO PTE FOR IOVA"; + else if (error_code & DART_ERROR_NO_PMD) + fault_name = "NO PMD FOR IOVA"; + else if (error_code & DART_ERROR_NO_TTBR) + fault_name = "NO TTBR FOR IOVA"; + + if (WARN_ON(fault_name == NULL)) + fault_name = "unknown"; + + if (__ratelimit(&rs)) { + dev_err(dart->dev, + "translation fault: status:0x%x stream:%d code:0x%x (%s) at 0x%llx", + error, stream_idx, error_code, fault_name, addr); + } + + writel(error, dart->regs + DART_ERROR); + return IRQ_HANDLED; +} + +static int apple_dart_probe(struct platform_device *pdev) +{ + int ret; + u32 dart_params[2]; + struct resource *res; + struct apple_dart *dart; + struct device *dev = &pdev->dev; + + dart = devm_kzalloc(dev, sizeof(*dart), GFP_KERNEL); + if (!dart) + return -ENOMEM; + + dart->dev = dev; + spin_lock_init(&dart->lock); + + if (pdev->num_resources < 1) + return -ENODEV; + + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); + if (resource_size(res) < 0x4000) { + dev_err(dev, "MMIO region too small (%pr)\n", res); + return -EINVAL; + } + + dart->regs = devm_ioremap_resource(dev, res); + if (IS_ERR(dart->regs)) + return PTR_ERR(dart->regs); + + ret = devm_clk_bulk_get_all(dev, &dart->clks); + if (ret < 0) + return ret; + dart->num_clks = ret; + + ret = clk_bulk_prepare_enable(dart->num_clks, dart->clks); + if (ret) + return ret; + + ret = apple_dart_hw_reset(dart); + if (ret) + goto err_clk_disable; + + dart_params[0] = readl(dart->regs + DART_PARAMS1); + dart_params[1] = readl(dart->regs + DART_PARAMS2); + dart->pgsize = 1 << FIELD_GET(DART_PARAMS_PAGE_SHIFT, dart_params[0]); + dart->supports_bypass = dart_params[1] & DART_PARAMS_BYPASS_SUPPORT; + dart->force_bypass = dart->pgsize > PAGE_SIZE; + + dart->irq = platform_get_irq(pdev, 0); + if (dart->irq < 0) { + ret = -ENODEV; + goto err_clk_disable; + } + + ret = devm_request_irq(dart->dev, dart->irq, apple_dart_irq, + IRQF_SHARED, "apple-dart fault handler", dart); + if (ret) + goto err_clk_disable; + + platform_set_drvdata(pdev, dart); + + ret = iommu_device_sysfs_add(&dart->iommu, dev, NULL, "apple-dart.%s", + dev_name(&pdev->dev)); + if (ret) + goto err_clk_disable; + + ret = iommu_device_register(&dart->iommu, &apple_dart_iommu_ops, dev); + if (ret) + goto err_clk_disable; + + if (dev->bus->iommu_ops != &apple_dart_iommu_ops) { + ret = bus_set_iommu(dev->bus, &apple_dart_iommu_ops); + if (ret) + goto err_clk_disable; + } +#ifdef CONFIG_PCI + if (dev->bus->iommu_ops != pci_bus_type.iommu_ops) { + ret = bus_set_iommu(&pci_bus_type, &apple_dart_iommu_ops); + if (ret) + goto err_clk_disable; + } +#endif + + dev_info( + &pdev->dev, + "DART [pagesize %x, bypass support: %d, bypass forced: %d] initialized\n", + dart->pgsize, dart->supports_bypass, dart->force_bypass); + return 0; + +err_clk_disable: + clk_bulk_disable(dart->num_clks, dart->clks); + clk_bulk_unprepare(dart->num_clks, dart->clks); + + return ret; +} + +static int apple_dart_remove(struct platform_device *pdev) +{ + struct apple_dart *dart = platform_get_drvdata(pdev); + + devm_free_irq(dart->dev, dart->irq, dart); + + iommu_device_unregister(&dart->iommu); + iommu_device_sysfs_remove(&dart->iommu); + + clk_bulk_disable(dart->num_clks, dart->clks); + clk_bulk_unprepare(dart->num_clks, dart->clks); + + return 0; +} + +static void apple_dart_shutdown(struct platform_device *pdev) +{ + apple_dart_remove(pdev); +} + +static const struct of_device_id apple_dart_of_match[] = { + { .compatible = "apple,t8103-dart", .data = NULL }, + {}, +}; +MODULE_DEVICE_TABLE(of, apple_dart_of_match); + +static struct platform_driver apple_dart_driver = { + .driver = { + .name = "apple-dart", + .of_match_table = apple_dart_of_match, + }, + .probe = apple_dart_probe, + .remove = apple_dart_remove, + .shutdown = apple_dart_shutdown, +}; +module_platform_driver(apple_dart_driver); + +MODULE_DESCRIPTION("IOMMU API for Apple's DART"); +MODULE_AUTHOR("Sven Peter "); +MODULE_LICENSE("GPL v2");