From patchwork Fri May 18 13:03:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Brodkin X-Patchwork-Id: 10410733 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 9022D602CB for ; Fri, 18 May 2018 13:11:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7E95328982 for ; Fri, 18 May 2018 13:11:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 71E4128986; Fri, 18 May 2018 13:11:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id D944E28982 for ; Fri, 18 May 2018 13:11:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Content-ID:In-Reply-To: References:Message-ID:Date:Subject:To:From:Reply-To:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UmhEEInNPgK/xtI1A62+E7ymzsOCgVDBReOs0dE4rKA=; b=Af++5Qod9ug1gF NlIhBdyPGRLn30yRj8tBaSX3aXTERyj7bmR08d0DpoH7zQ12n+XpEUo73HkfAh02OQO5ycpiNqVy4 VEOxtte+pNbS6KdTTePZ9ZNsDPSEO4avr55P97/NA6bdNkoS8sgsNQybH2jhBiLERyx5MKBu9waZY DapeDFd5du41pxESM4X4wsf/ONUFi/E6GDTYc+ImpfrGYbIELgJv2a548lgNj6HbalD9o2rolSKT0 FslS5bWIrL5jAYtL3fF4mldD8oANtCEQEfjdHAOnR8NjqG4th+0OvNHDQebvOtG6++mViuF/6ERAL 9K3D7wGTPYko0eFpm/sg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fJfAK-0005Lx-Ip; Fri, 18 May 2018 13:11:24 +0000 Received: from merlin.infradead.org ([2001:8b0:10b:1231::1]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fJfAH-0005JH-Lv; Fri, 18 May 2018 13:11:21 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=MIME-Version:Content-Transfer-Encoding: Content-ID:Content-Type:In-Reply-To:References:Message-ID:Date:Subject:CC:To: From:Sender:Reply-To:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=ib2y13Mp85nTR/5DcTzPO+DO4N/8vEB++VmGyaAclIU=; b=vhIU2um1Gk5pEZtoqQJKI+a5Rf 2PVVtX2+990Pp9cFn173rk0flyYd/FEquLgJNuEr6yitPschlpBOZPPoF+dcVKaneFAtFf2RsdB1F NxzNnIUV0TNtZZu+ymrech/Jv2zdNJnO8k09XH9FuDUF+86nbhFoNzvcPr+EcXzpKQf5rOfylweTx 6Q94qeCZRDxHESwxCc+Z8O17TvKtk6qBmoqbKK+fAMH/ljffNJlaOUs9vjQypMchlnFM9Rfq7T92X t4BaCn0JLn3vhBXRY9CUpsM71f5xbESgUHjMxCEVNJfP/QFEvu8ccJ/Bxum6snZx1+wUvEBXMr8/w 9etIPY+w==; Received: from us01smtprelay-2.synopsys.com ([198.182.60.111] helo=smtprelay.synopsys.com) by merlin.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fJf3J-00057E-2N; Fri, 18 May 2018 13:04:11 +0000 Received: from mailhost.synopsys.com (mailhost3.synopsys.com [10.12.238.238]) by smtprelay.synopsys.com (Postfix) with ESMTP id ED37110C0779; Fri, 18 May 2018 06:03:50 -0700 (PDT) Received: from US01WEHTC3.internal.synopsys.com (us01wehtc3.internal.synopsys.com [10.15.84.232]) by mailhost.synopsys.com (Postfix) with ESMTP id 4D9DD30C3; Fri, 18 May 2018 06:03:49 -0700 (PDT) Received: from DE02WEHTCB.internal.synopsys.com (10.225.19.94) by US01WEHTC3.internal.synopsys.com (10.15.84.232) with Microsoft SMTP Server (TLS) id 14.3.361.1; Fri, 18 May 2018 06:03:49 -0700 Received: from DE02WEMBXB.internal.synopsys.com ([fe80::95ce:118a:8321:a099]) by DE02WEHTCB.internal.synopsys.com ([::1]) with mapi id 14.03.0361.001; Fri, 18 May 2018 15:03:47 +0200 From: Alexey Brodkin To: "hch@lst.de" Subject: Re: [PATCH 02/20] dma-mapping: provide a generic dma-noncoherent implementation Thread-Topic: [PATCH 02/20] dma-mapping: provide a generic dma-noncoherent implementation Thread-Index: AQHT6P5rOSqikBCSDk25UJEllltUWqQ1XcqA Date: Fri, 18 May 2018 13:03:46 +0000 Message-ID: References: <20180511075945.16548-1-hch@lst.de> <20180511075945.16548-3-hch@lst.de> In-Reply-To: <20180511075945.16548-3-hch@lst.de> Accept-Language: en-US, ru-RU Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.121.8.87] Content-ID: <994FD47D919A664AB275AA5865AC4C92@internal.synopsys.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180518_090409_353216_BBC144C0 X-CRM114-Status: GOOD ( 17.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "linux-arch@vger.kernel.org" , "linux-xtensa@linux-xtensa.org" , "monstr@monstr.eu" , "linux-snps-arc@lists.infradead.org" , "linux-c6x-dev@linux-c6x.org" , "linux-parisc@vger.kernel.org" , "linux-sh@vger.kernel.org" , "linux-hexagon@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "iommu@lists.linux-foundation.org" , "linux-m68k@lists.linux-m68k.org" , "openrisc@lists.librecores.org" , "green.hu@gmail.com" , "linux-alpha@vger.kernel.org" , "sparclinux@vger.kernel.org" , "nios2-dev@lists.rocketboards.org" , "deanbo422@gmail.com" , "linux-arm-kernel@lists.infradead.org" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Hi Christoph, On Fri, 2018-05-11 at 09:59 +0200, Christoph Hellwig wrote: [snip] There seems to be one subtle issue with map/unmap code. While investigating problems on ARC I added instrumentation as below: ---------------------------------------->8------------------------------------ You seem to lost an offset in the page so if we happen to have a buffer not aligned to a page boundary then we were obviously corrupting data outside our data :) -Alexey --- a/arch/arc/mm/dma.c +++ b/arch/arc/mm/dma.c @@ -152,14 +152,37 @@ static void _dma_cache_sync(struct device *dev, phys_addr_t paddr, size_t size, } } +static const char *dir_to_str(enum dma_data_direction dir) +{ + switch (dir) { + case DMA_BIDIRECTIONAL: return "DMA_BIDIRECTIONAL"; + case DMA_TO_DEVICE: return "DMA_TO_DEVICE"; + case DMA_FROM_DEVICE: return "DMA_FROM_DEVICE"; + case DMA_NONE: return "DMA_NONE"; + default: return "WRONG_VALUE!"; + } +} + void arch_sync_dma_for_device(struct device *dev, phys_addr_t paddr, size_t size, enum dma_data_direction dir) { + if (dir != DMA_TO_DEVICE){ + dump_stack(); + printk(" *** %s@%d: DMA direction is %s instead of %s\n", + __func__, __LINE__, dir_to_str(dir), dir_to_str(DMA_TO_DEVICE)); + } + return _dma_cache_sync(dev, paddr, size, dir); } void arch_sync_dma_for_cpu(struct device *dev, phys_addr_t paddr, size_t size, enum dma_data_direction dir) { + if (dir != DMA_FROM_DEVICE) { + dump_stack(); + printk(" *** %s@%d: DMA direction is %s instead of %s\n", + __func__, __LINE__, dir_to_str(dir), dir_to_str(DMA_FROM_DEVICE)); + } + return _dma_cache_sync(dev, paddr, size, dir); } ---------------------------------------->8------------------------------------ And with that I noticed a bit unexpected output, see below: ---------------------------------------->8------------------------------------ Stack Trace: arc_unwind_core.constprop.1+0xd4/0xf8 dump_stack+0x68/0x80 arch_sync_dma_for_device+0x34/0xc4 dma_noncoherent_map_sg+0x80/0x94 __dw_mci_start_request+0x1ee/0x868 dw_mci_request+0x17e/0x1c8 mmc_wait_for_req+0x106/0x1ac mmc_app_sd_status+0x108/0x130 mmc_sd_setup_card+0xc6/0x2e8 mmc_attach_sd+0x1b6/0x394 mmc_rescan+0x2f4/0x3bc process_one_work+0x194/0x348 worker_thread+0xf2/0x478 kthread+0x120/0x13c ret_from_fork+0x18/0x1c *** arch_sync_dma_for_device@172: DMA direction is DMA_FROM_DEVICE instead of DMA_TO_DEVICE ... Stack Trace: arc_unwind_core.constprop.1+0xd4/0xf8 dump_stack+0x68/0x80 arch_sync_dma_for_device+0x34/0xc4 dma_noncoherent_map_page+0x86/0x8c usb_hcd_map_urb_for_dma+0x49e/0x53c usb_hcd_submit_urb+0x43c/0x8c4 usb_control_msg+0xbe/0x16c hub_port_init+0x5e0/0xb0c hub_event+0x4e6/0x1164 process_one_work+0x194/0x348 worker_thread+0xf2/0x478 kthread+0x120/0x13c ret_from_fork+0x18/0x1c mmcblk0: p1 p2 *** arch_sync_dma_for_device@172: DMA direction is DMA_FROM_DEVICE instead of DMA_TO_DEVICE ... and quite some more of the similar ... ---------------------------------------->8------------------------------------ In case of MMC/DW_MCI (AKA DesignWare MobileStorage controller) that's an execution flow: 1) __dw_mci_start_request() 2) dw_mci_pre_dma_transfer() 3) dma_map_sg(..., mmc_get_dma_dir(data)) Note mmc_get_dma_dir() is just "data->flags & MMC_DATA_WRITE ? DMA_TO_DEVICE : DMA_FROM_DEVICE". I.e. if we're preparing for sending data dma_noncoherent_map_sg() will have DMA_TO_DEVICE which is quite OK for passing to dma_noncoherent_sync_sg_for_device() but in case of reading we'll have DMA_FROM_DEVICE which we'll pass to dma_noncoherent_sync_sg_for_device() in dma_noncoherent_map_sg(). I'd say this is not entirely correct because IMHO arch_sync_dma_for_cpu() is supposed to only be used in case of DMA_FROM_DEVICE and arch_sync_dma_for_device() only in case of DMA_TO_DEVICE. > +static dma_addr_t dma_noncoherent_map_page(struct device *dev, struct page *page, > + unsigned long offset, size_t size, enum dma_data_direction dir, > + unsigned long attrs) > +{ > + dma_addr_t addr; > + > + addr = dma_direct_map_page(dev, page, offset, size, dir, attrs); > + if (!dma_mapping_error(dev, addr) && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) > + arch_sync_dma_for_device(dev, page_to_phys(page), size, dir); > + return addr; > +} > + > +static int dma_noncoherent_map_sg(struct device *dev, struct scatterlist *sgl, > + int nents, enum dma_data_direction dir, unsigned long attrs) > +{ > + nents = dma_direct_map_sg(dev, sgl, nents, dir, attrs); > + if (nents > 0 && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) > + dma_noncoherent_sync_sg_for_device(dev, sgl, nents, dir); > + return nents; > +} The same is for unmap functions. My guess is we need to respect direction in map/unmap functions and use either dma_noncoherent_sync_single_for_cpu(..., DMA_FROM_DEVICE) or dma_noncoherent_sync_single_for_device(...,DMA_TO_DEVICE). > +static void dma_noncoherent_unmap_page(struct device *dev, dma_addr_t addr, > + size_t size, enum dma_data_direction dir, unsigned long attrs) > +{ > + if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) > + dma_noncoherent_sync_single_for_cpu(dev, addr, size, dir); > +} > + > +static void dma_noncoherent_unmap_sg(struct device *dev, struct scatterlist *sgl, > + int nents, enum dma_data_direction dir, unsigned long attrs) > +{ > + if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) > + dma_noncoherent_sync_sg_for_cpu(dev, sgl, nents, dir); > +} > +#endif But the real fix of my problem is: ---------------------------------------->8------------------------------------ --- a/lib/dma-noncoherent.c +++ b/lib/dma-noncoherent.c @@ -35,7 +35,7 @@ static dma_addr_t dma_noncoherent_map_page(struct device *dev, struct page *page addr = dma_direct_map_page(dev, page, offset, size, dir, attrs); if (!dma_mapping_error(dev, addr) && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - arch_sync_dma_for_device(dev, page_to_phys(page), size, dir); + arch_sync_dma_for_device(dev, page_to_phys(page) + offset, size, dir); return addr; } ---------------------------------------->8------------------------------------