From patchwork Tue Nov 6 19:44:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Geert Uytterhoeven X-Patchwork-Id: 10671331 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BEC191709 for ; Tue, 6 Nov 2018 19:44:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AFCE52A9A6 for ; Tue, 6 Nov 2018 19:44:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A0CEC2AA7B; Tue, 6 Nov 2018 19:44:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 81D452A9A6 for ; Tue, 6 Nov 2018 19:44:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Wdjxx8t1YVIHvTOI/wMLMzuc2WGc8oxoA5+9W+C+nd0=; b=HSA76V9nOG3gex 7cLeux9kv2fSu5Hi8zYbY9V6Jde8maM5shXYVnjC7ZAeXbb7C6Pa25OKr2KzG9Ce8tWUBvP+CVSME KRDugupngzYA5scoevUM8x7vlu2PN6f4QCD3JHvhyEHMtBLOCvdpcotnhVAhmVWHrFGsY9VOiLUjI th5VhmuZsfMNnKQjGf7KWkKPJyvzdhDvSaODbY/Vkd4XfYxzPQPKpz7Q2KUs8Lgf9bTvpGAfU2Ese 123hs2sKB7ZNkpwKNCghwUu6BJeS8/KZXwZZNRdUn6eJVpbXDAiONyk0CD9EQBTBXgr+TdOQZTfoV 53hHqlPLlGVYOIy5TZwQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gK7HE-0000B1-KQ; Tue, 06 Nov 2018 19:44:40 +0000 Received: from merlin.infradead.org ([2001:8b0:10b:1231::1]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gK7HC-0000Aj-Ga for linux-arm-kernel@bombadil.infradead.org; Tue, 06 Nov 2018 19:44:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Content-Type:Cc:To:Subject:Message-ID: Date:From:In-Reply-To:References:MIME-Version:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=7JGRqzpY61xvgiLG9i7Ot5TbM8ZIcd6GtrJKGgCegyw=; b=izxwuwPVqjAMz5avJZDxzFd6C uiXvxWkPSqwmy6yHvHZCS6fXmgJ1hdpvfArBBL2aszau2d6RId8+Q6PdzHxHDDsS1pFcTa+9u3fe/ Wm0zMNYp2o6EML5yAxhc2rt/Al1Fi0s40fwHCkGEwi4rOTPWD9XYaiXbxrMlbVJnW/fO4E890dWau sgo+OAgw3ybiAoQ32wq8wHeGUWL3CmqnW9HK5aSRah/zBz3e7tWdS6tN81ZEALUXEb4YYnG7xtKTi ggy1wMTMgLGxxnDrVKcqaYwt7cGAnM9bwV8JAktC7kqeoQL+grTuDAp/+0DJa9VfvTmmeGuDkYVq5 GBYPanycw==; Received: from mail-ua1-f65.google.com ([209.85.222.65]) by merlin.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gK7H9-0001n2-47 for linux-arm-kernel@lists.infradead.org; Tue, 06 Nov 2018 19:44:36 +0000 Received: by mail-ua1-f65.google.com with SMTP id e16so4995674uam.1 for ; Tue, 06 Nov 2018 11:44:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=7JGRqzpY61xvgiLG9i7Ot5TbM8ZIcd6GtrJKGgCegyw=; b=hBGgLfafHmTgX2chIoYaqb9y0QdX1QKxoA0ZEgNC6U/FA8zlPzZ4dPG7dXltuFV2cF gUSynFmJL1CY3ondlOoBB443Xd6FgNKfU9cENiAcASviksqfqraLzxK3Uu5zTyLLw6t6 8Vmp5otjQrJhJsU1VkTcAjqq4+lrTjHXRPjqHCDfZa+X/SvCItjl1CzPVBgBB8DM5Yce 66aLCOJ8RkPVZ1t7P8sideR0yOV9cGdd2gYOBVVseV1sroTIjdHQMnXSjF7qbq7tkulJ qBHT2OpubTCZqy2vr/DsWps0VH2xHjSr8+68acN8RT96OKr9QwNIXGsbWeVNW5Klkuld T82w== X-Gm-Message-State: AGRZ1gK3z815wqnnw0/RXW/tFHpDz1Kjc69toY4nHfffKBU0/Wm32z4D ijBAWk/6Hl9RQ56PWhans40MjtvQNqlPmDrqm9g= X-Google-Smtp-Source: AJdET5fZ2F+qaJfaPXPz4/vtBeBHF+eplRFskAJYxyKaQFfkfuapQMvDdv2y4QJedLOWQ1lnMNcbcHQ85kSAcEysTVE= X-Received: by 2002:ab0:210e:: with SMTP id d14mr941944ual.20.1541533460326; Tue, 06 Nov 2018 11:44:20 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Geert Uytterhoeven Date: Tue, 6 Nov 2018 20:44:07 +0100 Message-ID: Subject: IOMMU breakage on arm64 (was: Re: dma-direct: implement complete bus_dma_mask handling) To: Christoph Hellwig , Will Deacon , Robin Murphy , Joerg Roedel , Magnus Damm X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20181106_144435_198028_77F7F600 X-CRM114-Status: GOOD ( 21.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Linux-Renesas , Linux IOMMU , Linux Kernel Mailing List , Linux ARM Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Hi Christoph et al, On Tue, Oct 23, 2018 at 1:40 AM Linux Kernel Mailing List wrote: > Commit: b4ebe6063204da58e48600b810a97c29ae9e5d12 > Parent: 7d21ee4c719f00896767ce19c4c01a56374c2ced > Refname: refs/heads/master > Web: https://git.kernel.org/torvalds/c/b4ebe6063204da58e48600b810a97c29ae9e5d12 > Author: Christoph Hellwig > AuthorDate: Thu Sep 20 14:04:08 2018 +0200 > Committer: Christoph Hellwig > CommitDate: Mon Oct 1 07:28:03 2018 -0700 > > dma-direct: implement complete bus_dma_mask handling > > Instead of rejecting devices with a too small bus_dma_mask we can handle > by taking the bus dma_mask into account for allocations and bounce > buffering decisions. > > Signed-off-by: Christoph Hellwig I have bisected the following crash to the above commit: ipmmu-vmsa e67b0000.mmu: Cannot accommodate DMA translation for IOMMU page tables sata_rcar ee300000.sata: Unable to initialize IPMMU context iommu: Failed to add device ee300000.sata to group 0: -22 Unable to handle kernel NULL pointer dereference at virtual address 0000000000000038 Mem abort info: ESR = 0x96000004 Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0 EA = 0, S1PTW = 0 Data abort info: ISV = 0, ISS = 0x00000004 CM = 0, WnR = 0 [0000000000000038] user address but active_mm is swapper Internal error: Oops: 96000004 [#1] PREEMPT SMP CPU: 2 PID: 51 Comm: kworker/2:1 Not tainted 4.20.0-rc1-arm64-renesas-dirty #74 Hardware name: Renesas Salvator-X 2nd version board based on r8a7795 ES2.0+ (DT) Workqueue: events deferred_probe_work_func pstate: 60000005 (nZCv daif -PAN -UAO) pc : ipmmu_domain_free+0x1c/0xa0 lr : ipmmu_domain_free+0x14/0xa0 sp : ffff000009c9b990 x29: ffff000009c9b990 x28: 0000000000000000 x27: ffff000008dff000 x26: 0000000000000000 x25: ffff000008dd9000 x24: 0000000000000014 x23: ffff8006fffdbb20 x22: ffff000008dff000 x21: ffff8006f898e680 x20: ffff8006f8fa6c00 x19: ffff8006f8fa6c08 x18: 0000000000000037 x17: 0000000000000020 x16: ffff000008a8f780 x15: ffff8006fb096f10 x14: ffff8006f93731c8 x13: 0000000000000000 x12: ffff8006fb096f10 x11: ffff8006f9372fe8 x10: ffff000008dff708 x9 : ffff8006fb096f50 x8 : ffff000008dff708 x7 : ffff0000089f1858 x6 : 0000000000000000 x5 : 0000000000000000 x4 : ffff8006f89a3000 x3 : 00008006f7131000 x2 : ffff8006fb2b5700 x1 : 0000000000000000 x0 : 0000000000000028 Process kworker/2:1 (pid: 51, stack limit = 0x(____ptrval____)) Call trace: ipmmu_domain_free+0x1c/0xa0 iommu_group_release+0x48/0x68 kobject_put+0x74/0xe8 kobject_del.part.0+0x3c/0x50 kobject_put+0x60/0xe8 iommu_group_get_for_dev+0xa8/0x1f0 ipmmu_add_device+0x1c/0x40 of_iommu_configure+0x118/0x190 of_dma_configure+0xcc/0x1f0 platform_dma_configure+0x18/0x28 really_probe+0x94/0x2a8 driver_probe_device+0x58/0x100 __device_attach_driver+0x90/0xd0 bus_for_each_drv+0x64/0xc8 __device_attach+0xd8/0x130 device_initial_probe+0x10/0x18 bus_probe_device+0x98/0xa0 deferred_probe_work_func+0x74/0xb0 process_one_work+0x294/0x6f0 worker_thread+0x238/0x460 kthread+0x120/0x128 ret_from_fork+0x10/0x1c Code: aa0003f3 97ffeb1a d1002274 f85f8261 (f9401c20) ---[ end trace 4c46c7fd7cd07245 ]--- Reproducing on v4.20-rc1 requires CONFIG_IPMMU_VMSA=y, and whitelisting SATA for IOMMU use (https://patchwork.kernel.org/patch/10544059/). For older versions you also have to backport commit 3a0832d093693ede ("arm64: dts: renesas: salvator-xs: enable SATA"). Actually there are two issues at hand: 1) drivers/iommu/io-pgtable-arm.c:__arm_lpae_alloc_pages() allocates memory (above 4 GiB) and maps it for DMA use, but it is rejected due to: dma_map_single(dev, pages, size, DMA_TO_DEVICE) != virt_to_phys(pages) > --- a/include/linux/dma-direct.h > +++ b/include/linux/dma-direct.h > @@ -27,7 +27,8 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size) > if (!dev->dma_mask) > return false; > > - return addr + size - 1 <= *dev->dma_mask; > + return addr + size - 1 <= > + min_not_zero(*dev->dma_mask, dev->bus_dma_mask); *dev->dma_mask = 0xffffffffff (40-bit) dev->bus_dma_mask = 0xffffffff (32-bit) Hence before, we had (in __arm_lpae_alloc_pages()): arm-lpae io-pgtable: pages = ffff8006f88f3000 arm-lpae io-pgtable: dma = 0x00000007388f3000 arm-lpae io-pgtable: virt_to_phys(pages) = 0x00000007388f3000 After this change, we have: arm-lpae io-pgtable: pages = ffff8006f882b000 arm-lpae io-pgtable: dma = 0x0000000074009000 arm-lpae io-pgtable: virt_to_phys(pages) = 0x000000073882b000 And SATA runs without using the IOMMU. 2) The Renesas IPMMU driver doesn't handle the above failure well, leading to a NULL pointer dereference. This can be fixed using the (gmail-whitespace-damaged) patch below: } I expect drivers/iommu/arm-smmu-v3.c and drivers/iommu/arm-smmu.c need similar fixes. I didn't check all drivers, but e.g. drivers/iommu/amd_iommu.c has a similar check. Does the IOMMU work on other arm64 platforms in v4.20-rc1? Thanks for your comments! Gr{oetje,eeting}s, Geert --- a/drivers/iommu/ipmmu-vmsa.c +++ b/drivers/iommu/ipmmu-vmsa.c @@ -635,7 +635,8 @@ static void ipmmu_domain_free(struct iommu_domain *io_domain) * been detached. */ iommu_put_dma_cookie(io_domain); - ipmmu_domain_destroy_context(domain); + if (domain->mmu) + ipmmu_domain_destroy_context(domain); free_io_pgtable_ops(domain->iop); kfree(domain);