From patchwork Wed Jun 1 10:23:23 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 9147079 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B297560777 for ; Wed, 1 Jun 2016 10:23:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A1B5D1FF65 for ; Wed, 1 Jun 2016 10:23:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9669E26785; Wed, 1 Jun 2016 10:23:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6C6C11FF65 for ; Wed, 1 Jun 2016 10:23:41 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 739186E0A0; Wed, 1 Jun 2016 10:23:38 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from mail-wm0-x242.google.com (mail-wm0-x242.google.com [IPv6:2a00:1450:400c:c09::242]) by gabe.freedesktop.org (Postfix) with ESMTPS id 9CA996E0A0 for ; Wed, 1 Jun 2016 10:23:36 +0000 (UTC) Received: by mail-wm0-x242.google.com with SMTP id e3so5346779wme.2 for ; Wed, 01 Jun 2016 03:23:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=JAZRWjNB70KwUrkPSyuUXSc68xlMXSo+kS9XfD5T8us=; b=xssbWwageMEbEh4Yymgf3FJoMnqhJxXkL7MIJMP69dzBLaOp3kOumbWLRU3G6RhETt O74pg/LtKF66Ks924CBYi8FztTlTtH4Osw4kiDeYwrww2O/sOJzHd3oZWVx1+EDREWX2 Tf6ANqUv7qZX0nh4nCJZEzszXAtRxmZxRaXoUnrgvfZhFqXMKeqKPRN/6EywV6NOnEDF wwZwANe5K0JR8sIdYcgHj7LOQ+94dXRe6KFsJA7B/C/XmqreJgotcAi8QrwdvaPAeGGz z3i8Aj7jwUDxAqLiBgVtkUqDD1p/jIksGGVfeWIVNVU0w5F4DmR7//w5ukZjIMiCwoV8 WxUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=JAZRWjNB70KwUrkPSyuUXSc68xlMXSo+kS9XfD5T8us=; b=CDPlIPRm1ZUOtSCJBcMQ/lsHBGJiQdNqpe7ubw4R3E4MvxIqI4tulZ2LvUDVNlCcCL DzHn9Rdh5ebPtB3retujDieX2Xi89sEIXTNDvA3OIvRHzPk4uQM0q9xXiWdFwIPJ2tuS nvoeYKN9NvmkVC3k84EKREZNTEC1pyzr+oWOHz4wkVxNSZdZlX/dGvz51R6qCnW+H/t6 UP8zq0S16DE+MYiuPXk7aunV9FWkL87ghSNauV18AEaM2J5deorxu/2w78yCFmSW63qs VTbwWX3UJJrt3Y/c3IBxPsH2EpiGlReP2/nv5k5ctDGbeLOCNl6Vo+8EzMxQltARCkn3 hryQ== X-Gm-Message-State: ALyK8tIpias9PSA8aRrZnA/Jk1qNvjCZz2M25tlQPk8YvNhdu3Wz9um6vfOVoPqtYnxdpQ== X-Received: by 10.28.44.70 with SMTP id s67mr10029909wms.14.1464776614836; Wed, 01 Jun 2016 03:23:34 -0700 (PDT) Received: from haswell.alporthouse.com ([78.156.65.138]) by smtp.gmail.com with ESMTPSA id j6sm44628667wjb.29.2016.06.01.03.23.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 01 Jun 2016 03:23:33 -0700 (PDT) From: Chris Wilson To: Joerg Roedel Date: Wed, 1 Jun 2016 11:23:23 +0100 Message-Id: <1464776603-11998-1-git-send-email-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1464712470-26286-1-git-send-email-chris@chris-wilson.co.uk> References: <1464712470-26286-1-git-send-email-chris@chris-wilson.co.uk> Cc: linux-kernel@vger.kernel.org, intel-gfx@lists.freedesktop.org, iommu@lists.linux-foundation.org Subject: [Intel-gfx] [PATCH v2] iommu: Disable preemption around use of this_cpu_ptr() X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" X-Virus-Scanned: ClamAV using ClamSMTP Between acquiring the this_cpu_ptr() and using it, ideally we don't want to be preempted and work on another CPU's private data. If we disable preemption around this_cpu_ptr, we do not need the CPU local spinlock - so long as take care that no other CPU is running that code as do perform the cross-CPU cache flushing and teardown. [ 167.997877] BUG: using smp_processor_id() in preemptible [00000000] code: usb-storage/216 [ 167.997940] caller is debug_smp_processor_id+0x17/0x20 [ 167.997945] CPU: 7 PID: 216 Comm: usb-storage Tainted: G U 4.7.0-rc1-gfxbench-RO_Patchwork_1057+ #1 [ 167.997948] Hardware name: Hewlett-Packard HP Pro 3500 Series/2ABF, BIOS 8.11 10/24/2012 [ 167.997951] 0000000000000000 ffff880118b7f9c8 ffffffff8140dca5 0000000000000007 [ 167.997958] ffffffff81a3a7e9 ffff880118b7f9f8 ffffffff8142a927 0000000000000000 [ 167.997965] ffff8800d499ed58 0000000000000001 00000000000fffff ffff880118b7fa08 [ 167.997971] Call Trace: [ 167.997977] [] dump_stack+0x67/0x92 [ 167.997981] [] check_preemption_disabled+0xd7/0xe0 [ 167.997985] [] debug_smp_processor_id+0x17/0x20 [ 167.997990] [] alloc_iova_fast+0xb7/0x210 [ 167.997994] [] intel_alloc_iova+0x7f/0xd0 [ 167.997998] [] intel_map_sg+0xbd/0x240 [ 167.998002] [] ? debug_lockdep_rcu_enabled+0x1d/0x20 [ 167.998009] [] usb_hcd_map_urb_for_dma+0x4b9/0x5a0 [ 167.998013] [] usb_hcd_submit_urb+0xe9/0xaa0 [ 167.998017] [] ? mark_held_locks+0x6f/0xa0 [ 167.998022] [] ? __raw_spin_lock_init+0x1c/0x50 [ 167.998025] [] ? debug_lockdep_rcu_enabled+0x1d/0x20 [ 167.998028] [] usb_submit_urb+0x3f3/0x5a0 [ 167.998032] [] ? trace_hardirqs_on_caller+0x122/0x1b0 [ 167.998035] [] usb_sg_wait+0x67/0x150 [ 167.998039] [] usb_stor_bulk_transfer_sglist.part.3+0x82/0xd0 [ 167.998042] [] usb_stor_bulk_srb+0x4c/0x60 [ 167.998045] [] usb_stor_Bulk_transport+0x17e/0x420 [ 167.998049] [] usb_stor_invoke_transport+0x242/0x540 [ 167.998052] [] ? debug_lockdep_rcu_enabled+0x1d/0x20 [ 167.998058] [] usb_stor_transparent_scsi_command+0x9/0x10 [ 167.998061] [] usb_stor_control_thread+0x158/0x260 [ 167.998064] [] ? fill_inquiry_response+0x20/0x20 [ 167.998067] [] ? fill_inquiry_response+0x20/0x20 [ 167.998071] [] kthread+0xea/0x100 [ 167.998078] [] ret_from_fork+0x1f/0x40 [ 167.998081] [] ? kthread_create_on_node+0x1f0/0x1f0 v2: convert preempt_disable(); var = this_cpu_ptr() to var = get_cpu_ptr() Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=96293 Signed-off-by: Chris Wilson Cc: Joonas Lahtinen Cc: Joerg Roedel Cc: iommu@lists.linux-foundation.org Cc: linux-kernel@vger.kernel.org iova --- drivers/iommu/iova.c | 33 ++++++++++----------------------- 1 file changed, 10 insertions(+), 23 deletions(-) diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index ba764a0835d3..3464ed371eac 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -390,6 +390,11 @@ free_iova(struct iova_domain *iovad, unsigned long pfn) } EXPORT_SYMBOL_GPL(free_iova); +static void free_this_cached_iovas(void *info) +{ + free_cpu_cached_iovas(smp_processor_id(), info); +} + /** * alloc_iova_fast - allocates an iova from rcache * @iovad: - iova domain in question @@ -413,15 +418,12 @@ alloc_iova_fast(struct iova_domain *iovad, unsigned long size, retry: new_iova = alloc_iova(iovad, size, limit_pfn, true); if (!new_iova) { - unsigned int cpu; - if (flushed_rcache) return 0; /* Try replenishing IOVAs by flushing rcache. */ flushed_rcache = true; - for_each_online_cpu(cpu) - free_cpu_cached_iovas(cpu, iovad); + on_each_cpu(free_this_cached_iovas, iovad, true); goto retry; } @@ -645,7 +647,6 @@ struct iova_magazine { }; struct iova_cpu_rcache { - spinlock_t lock; struct iova_magazine *loaded; struct iova_magazine *prev; }; @@ -727,7 +728,6 @@ static void init_iova_rcaches(struct iova_domain *iovad) continue; for_each_possible_cpu(cpu) { cpu_rcache = per_cpu_ptr(rcache->cpu_rcaches, cpu); - spin_lock_init(&cpu_rcache->lock); cpu_rcache->loaded = iova_magazine_alloc(GFP_KERNEL); cpu_rcache->prev = iova_magazine_alloc(GFP_KERNEL); } @@ -747,10 +747,8 @@ static bool __iova_rcache_insert(struct iova_domain *iovad, struct iova_magazine *mag_to_free = NULL; struct iova_cpu_rcache *cpu_rcache; bool can_insert = false; - unsigned long flags; - cpu_rcache = this_cpu_ptr(rcache->cpu_rcaches); - spin_lock_irqsave(&cpu_rcache->lock, flags); + cpu_rcache = get_cpu_var(rcache->cpu_rcaches); if (!iova_magazine_full(cpu_rcache->loaded)) { can_insert = true; @@ -778,7 +776,7 @@ static bool __iova_rcache_insert(struct iova_domain *iovad, if (can_insert) iova_magazine_push(cpu_rcache->loaded, iova_pfn); - spin_unlock_irqrestore(&cpu_rcache->lock, flags); + put_cpu_var(rcache->cpu_rcaches); if (mag_to_free) { iova_magazine_free_pfns(mag_to_free, iovad); @@ -810,10 +808,8 @@ static unsigned long __iova_rcache_get(struct iova_rcache *rcache, struct iova_cpu_rcache *cpu_rcache; unsigned long iova_pfn = 0; bool has_pfn = false; - unsigned long flags; - cpu_rcache = this_cpu_ptr(rcache->cpu_rcaches); - spin_lock_irqsave(&cpu_rcache->lock, flags); + cpu_rcache = get_cpu_var(rcache->cpu_rcaches); if (!iova_magazine_empty(cpu_rcache->loaded)) { has_pfn = true; @@ -833,7 +829,7 @@ static unsigned long __iova_rcache_get(struct iova_rcache *rcache, if (has_pfn) iova_pfn = iova_magazine_pop(cpu_rcache->loaded, limit_pfn); - spin_unlock_irqrestore(&cpu_rcache->lock, flags); + put_cpu_var(rcache->cpu_rcaches); return iova_pfn; } @@ -862,17 +858,11 @@ static void free_cpu_iova_rcache(unsigned int cpu, struct iova_domain *iovad, struct iova_rcache *rcache) { struct iova_cpu_rcache *cpu_rcache = per_cpu_ptr(rcache->cpu_rcaches, cpu); - unsigned long flags; - - spin_lock_irqsave(&cpu_rcache->lock, flags); - iova_magazine_free_pfns(cpu_rcache->loaded, iovad); iova_magazine_free(cpu_rcache->loaded); iova_magazine_free_pfns(cpu_rcache->prev, iovad); iova_magazine_free(cpu_rcache->prev); - - spin_unlock_irqrestore(&cpu_rcache->lock, flags); } /* @@ -906,16 +896,13 @@ void free_cpu_cached_iovas(unsigned int cpu, struct iova_domain *iovad) { struct iova_cpu_rcache *cpu_rcache; struct iova_rcache *rcache; - unsigned long flags; int i; for (i = 0; i < IOVA_RANGE_CACHE_MAX_SIZE; ++i) { rcache = &iovad->rcaches[i]; cpu_rcache = per_cpu_ptr(rcache->cpu_rcaches, cpu); - spin_lock_irqsave(&cpu_rcache->lock, flags); iova_magazine_free_pfns(cpu_rcache->loaded, iovad); iova_magazine_free_pfns(cpu_rcache->prev, iovad); - spin_unlock_irqrestore(&cpu_rcache->lock, flags); } }