From patchwork Mon Jun 24 15:04:28 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konrad Rzeszutek Wilk X-Patchwork-Id: 2771851 Return-Path: X-Original-To: patchwork-dri-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id F1A70C0AB1 for ; Mon, 24 Jun 2013 15:05:11 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id B9E762027C for ; Mon, 24 Jun 2013 15:05:10 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) by mail.kernel.org (Postfix) with ESMTP id 0845B2027B for ; Mon, 24 Jun 2013 15:05:09 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id BE4D5E5FC1 for ; Mon, 24 Jun 2013 08:05:08 -0700 (PDT) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by gabe.freedesktop.org (Postfix) with ESMTP id D8666E5EA9 for ; Mon, 24 Jun 2013 08:04:49 -0700 (PDT) Received: from acsinet22.oracle.com (acsinet22.oracle.com [141.146.126.238]) by aserp1040.oracle.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.1) with ESMTP id r5OF4ZXK000846 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 24 Jun 2013 15:04:36 GMT Received: from userz7021.oracle.com (userz7021.oracle.com [156.151.31.85]) by acsinet22.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id r5OF4YOH005788 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 24 Jun 2013 15:04:34 GMT Received: from abhmt111.oracle.com (abhmt111.oracle.com [141.146.116.63]) by userz7021.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id r5OF4XaF003215; Mon, 24 Jun 2013 15:04:33 GMT Received: from localhost.localdomain (/50.195.21.189) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 24 Jun 2013 08:04:33 -0700 Date: Mon, 24 Jun 2013 11:04:28 -0400 From: Konrad Rzeszutek Wilk To: Chris Wilson , daniel@ffwll.ch Subject: Re: Regression introduced by 90797e6d1ec0dfde6ba62a48b9ee3803887d6ed4 ("drm/i915: create compact dma scatter lists for gem objects") Was:Re: i915 mapping large (3MB) scatter list, hitting limits on certain IOMMUs that can only map contingous regions up to 2MB. Message-ID: <20130624150428.GA2175@localhost.localdomain> References: <20130621192827.GA2044@localhost.localdomain> <20130622020343.GA19139@phenom.dumpdata.com> <20130622142259.GA3499@cantiga.alporthouse.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20130622142259.GA3499@cantiga.alporthouse.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: acsinet22.oracle.com [141.146.126.238] Cc: daniel.vetter@ffwll.ch, dri-devel@lists.freedesktop.org X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dri-devel-bounces+patchwork-dri-devel=patchwork.kernel.org@lists.freedesktop.org Errors-To: dri-devel-bounces+patchwork-dri-devel=patchwork.kernel.org@lists.freedesktop.org X-Spam-Status: No, score=-5.3 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Sat, Jun 22, 2013 at 03:22:59PM +0100, Chris Wilson wrote: > On Fri, Jun 21, 2013 at 10:03:43PM -0400, Konrad Rzeszutek Wilk wrote: > > On Fri, Jun 21, 2013 at 03:28:28PM -0400, Konrad Rzeszutek Wilk wrote: > > > Hey, > > > > CC-ing Imre, > > > > Imre, your patch 90797e6d1ec0dfde6ba62a48b9ee3803887d6ed4 > > ("drm/i915: create compact dma scatter lists for gem objects") is the cause > > of the regression. > > > > If I revert your patch it boots fine without any trouble. > > > > I am not entirely sure why that is - as I added some debug code in > > lib/swiotlb.c to trigger when it can't find 3MB area (which > > is what I thought initially was the issue) - but none of the debug > > code seems to be hit. > > > > Any thoughts? > > You should be hitting drivers/iommu/intel-iommu.c for the dma > translation. It looks like the contiguous 3MiB segment will be special > as it is the first sg that __domain_mapping() will attempt to allocate a > superpage (2MiB) for. What goes wrong at point, I am not sure, but I > would suggest peppering intel-iommu.c with printk to track down the error. I figured it out. The issue was that I am backed by the SWIOTLB which can only allocate up to 128*4K chunks of contingous bounce buffer (see IO_TLB_SEGSIZE) - which means it can only setup up to 512kB buffers. While one of the SG entries tries to give it one past that size (3MB). The change Imre introduced assume that the CPU addresses (virtual) are the same as the bus addresses. That is correct in most platforms, but some (for example when booting a Xen PV guest with i915 as PCI passthrough) the virt_to_phys() values != bus address. Which means that the nice check of: if (!i || page_to_pfn(page) != last_pfn + 1) { if (i) sg = sg_next(sg); st->nents++; sg_set_page(sg, page, PAGE_SIZE, 0); } else { sg->length += PAGE_SIZE; } is too simplistic. What it ought to do is consult the DMA API whether the next PFN (page_to_pfn(page)) is really contingous in the DMA space. And also that it does got past the DMA mask for said device. Unfortunatly such calls do not exist. Those checks are all done when dma_map_page_* is done (which is where it failed for me). The best (for an rc7 stage) fix I came up with is to revert just a bit of the old behavior and still retain the sg compact code behavior. See following patch: diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c index 970ad17..9edd2eb 100644 --- a/drivers/gpu/drm/i915/i915_gem.c +++ b/drivers/gpu/drm/i915/i915_gem.c @@ -1801,7 +1801,12 @@ i915_gem_object_get_pages_gtt(struct drm_i915_gem_object *obj) gfp |= __GFP_NORETRY | __GFP_NOWARN | __GFP_NO_KSWAPD; gfp &= ~(__GFP_IO | __GFP_WAIT); } - + if (swiotlb_nr_tbl()) { + st->nents++; + sg_set_page(sg, page, PAGE_SIZE, 0); + sg = sg_next(sg); + continue; + } if (!i || page_to_pfn(page) != last_pfn + 1) { if (i) sg = sg_next(sg); @@ -1813,7 +1818,8 @@ i915_gem_object_get_pages_gtt(struct drm_i915_gem_object *obj) last_pfn = page_to_pfn(page); } - sg_mark_end(sg); + if (!swiotlb_nr_tbl()) + sg_mark_end(sg); obj->pages = st; if (i915_gem_object_needs_bit17_swizzle(obj))