From patchwork Sun Jul 17 12:58:06 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 9233669 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B5A926086B for ; Sun, 17 Jul 2016 13:00:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A816E205AD for ; Sun, 17 Jul 2016 13:00:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9CC452094D; Sun, 17 Jul 2016 13:00:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1AF6020855 for ; Sun, 17 Jul 2016 13:00:16 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C136E6E1FC; Sun, 17 Jul 2016 13:00:14 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mail-wm0-x243.google.com (mail-wm0-x243.google.com [IPv6:2a00:1450:400c:c09::243]) by gabe.freedesktop.org (Postfix) with ESMTPS id 89BC96E1D8 for ; Sun, 17 Jul 2016 12:59:12 +0000 (UTC) Received: by mail-wm0-x243.google.com with SMTP id x83so8714707wma.3 for ; Sun, 17 Jul 2016 05:59:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=AOg1ADiY6u7pvvCYK5foFph73K7NFmO1y1TC1JORjZU=; b=iAJDXaaOQF1IZ79tGiUXBsBxirSX0bDqmPfOG3d+LSXv3aAehy3UFqeMgPH7qX+tWC mzHTql2hnNWuNmHZu5ii+I6Gp39pcn2jVyxKH/cuaj4BVCxemnTywpK4HLoNUs1t4ScJ JEKohbUF+GQUEcpclMI+uNle2bnHLaTwd/vUjQln8ng8ciPxMEjwid5d70pjm6WLK0yC H1QKbEbRpUHJ35d/7SXxk13CpcrjppRYS4tSNwcHQd8NgM8pLZHyKKeJ4CJYoDVMcrYH sgWUJ2Jga92zkZu4NpBgspZ/jP4B+P+Gh44gB3u4cApAr0V6N3TvTzkhg/23YTbt8+kL XSXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=AOg1ADiY6u7pvvCYK5foFph73K7NFmO1y1TC1JORjZU=; b=BDChMwcYf05ytOe/fOKPio8e0YwAi9sLzM989zjI6Z1IPPkmk90Lx38QlwLdxi34rl NW2J+pNXITpDaCq6tmeHI2QVbDIOX8zC5yh/9i+DFIM404dWFlly8I+XneapEccrHtv4 iSL1foaE0I5+yK4mu74KOlDs1Utb+aqXjVX/rcYi7Sq6AsJkSY8urUlYv3FrDTz0O9lG lAdwwqRxtJ0C3WV3MPRu36SzYK5J13T077ulefik19Fkci9Nl4j1VBOy+WeJzwFEXU+1 bgmhf2/p8B/sZdGQuvvDmaTLqCrl8XXlRdY3JnEuf/N0a/lBThhyMBNA9h5Yau0kmMe0 fOKQ== X-Gm-Message-State: ALyK8tLDvx65v71XCMFUl+S2h4uX7f9si5oUxvMw9ggNTr1M3DqoUi52mcwbD1AniCDUJw== X-Received: by 10.28.47.7 with SMTP id v7mr23927683wmv.38.1468760350613; Sun, 17 Jul 2016 05:59:10 -0700 (PDT) Received: from haswell.alporthouse.com ([78.156.65.138]) by smtp.gmail.com with ESMTPSA id p76sm8125371wmd.10.2016.07.17.05.59.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 17 Jul 2016 05:59:09 -0700 (PDT) From: Chris Wilson To: linux-kernel@vger.kernel.org Subject: [PATCH v2 6/7] async: Add execution barriers Date: Sun, 17 Jul 2016 13:58:06 +0100 Message-Id: <1468760287-731-7-git-send-email-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1468760287-731-1-git-send-email-chris@chris-wilson.co.uk> References: <1466759333-4703-1-git-send-email-chris@chris-wilson.co.uk> <1468760287-731-1-git-send-email-chris@chris-wilson.co.uk> Cc: "Peter Zijlstra \(Intel\)" , Daniel Vetter , Rasmus Villemoes , dri-devel@lists.freedesktop.org, Alexander Potapenko , Ingo Molnar , Davidlohr Bueso , Shuah Khan , Andrey Ryabinin , "Paul E. McKenney" , linux-media@vger.kernel.org, Kees Cook , Nikolay Aleksandrov , linaro-mm-sig@lists.linaro.org, Dan Williams , Andy Shevchenko , Dmitry Vyukov , Thomas Gleixner , Tejun Heo , Andrew Morton , "David S. Miller" X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" X-Virus-Scanned: ClamAV using ClamSMTP A frequent mode of operation is fanning out N tasks to execute in parallel, collating results, fanning out M tasks, rinse and repeat. This is also common to the notion of the async/sync kernel domain split. A barrier provides a mechanism by which all work queued after the barrier must wait (i.e. not be scheduled) until all work queued before the barrier is completed. Signed-off-by: Chris Wilson Cc: Sumit Semwal Cc: Shuah Khan Cc: Tejun Heo Cc: Daniel Vetter Cc: Andrew Morton Cc: Ingo Molnar Cc: Kees Cook Cc: Thomas Gleixner Cc: "Paul E. McKenney" Cc: Dan Williams Cc: Andrey Ryabinin Cc: Davidlohr Bueso Cc: Nikolay Aleksandrov Cc: "David S. Miller" Cc: "Peter Zijlstra (Intel)" Cc: Rasmus Villemoes Cc: Andy Shevchenko Cc: Dmitry Vyukov Cc: Alexander Potapenko Cc: linux-kernel@vger.kernel.org Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Cc: linaro-mm-sig@lists.linaro.org --- include/linux/async.h | 4 +++ kernel/async.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+) diff --git a/include/linux/async.h b/include/linux/async.h index e7d7289a9889..de44306f8cb7 100644 --- a/include/linux/async.h +++ b/include/linux/async.h @@ -26,6 +26,7 @@ struct async_work { struct async_domain { struct list_head pending; + struct kfence *barrier; unsigned registered:1; }; @@ -59,6 +60,9 @@ extern void async_synchronize_cookie(async_cookie_t cookie); extern void async_synchronize_cookie_domain(async_cookie_t cookie, struct async_domain *domain); +extern void async_barrier(void); +extern void async_barrier_domain(struct async_domain *domain); + extern bool current_is_async(void); extern struct async_work * diff --git a/kernel/async.c b/kernel/async.c index 0d695919a60d..5cfa398a19b2 100644 --- a/kernel/async.c +++ b/kernel/async.c @@ -154,6 +154,15 @@ struct async_work *async_work_create(async_func_t func, void *data, gfp_t gfp) } EXPORT_SYMBOL_GPL(async_work_create); +static void async_barrier_delete(struct async_domain *domain) +{ + if (!domain->barrier) + return; + + kfence_put(domain->barrier); + domain->barrier = NULL; +} + async_cookie_t queue_async_work(struct async_domain *domain, struct async_work *work, gfp_t gfp) @@ -174,6 +183,10 @@ async_cookie_t queue_async_work(struct async_domain *domain, async_pending_count++; spin_unlock_irqrestore(&async_lock, flags); + if (domain->barrier && + !kfence_await_kfence(&entry->base.fence, domain->barrier, gfp)) + async_barrier_delete(domain); + /* mark that this task has queued an async job, used by module init */ current->flags |= PF_USED_ASYNC; @@ -241,6 +254,63 @@ async_cookie_t async_schedule_domain(async_func_t func, void *data, } EXPORT_SYMBOL_GPL(async_schedule_domain); +static struct kfence *__async_barrier_create(struct async_domain *domain) +{ + struct kfence *fence; + struct async_entry *entry; + unsigned long flags; + int ret; + + fence = kmalloc(sizeof(*fence), GFP_KERNEL); + if (!fence) + goto out_sync; + + kfence_init(fence, NULL); + + ret = 0; + spin_lock_irqsave(&async_lock, flags); + list_for_each_entry(entry, &domain->pending, pending_link[0]) { + ret |= kfence_await_kfence(fence, + &entry->base.fence, + GFP_ATOMIC); + if (ret < 0) + break; + } + spin_unlock_irqrestore(&async_lock, flags); + if (ret <= 0) + goto out_put; + + if (domain->barrier) + kfence_await_kfence(fence, domain->barrier, GFP_KERNEL); + + kfence_complete(fence); + return fence; + +out_put: + kfence_complete(fence); + kfence_put(fence); +out_sync: + async_synchronize_full_domain(domain); + return NULL; +} + +void async_barrier(void) +{ + async_barrier_domain(&async_dfl_domain); +} +EXPORT_SYMBOL_GPL(async_barrier); + +void async_barrier_domain(struct async_domain *domain) +{ + struct kfence *barrier = __async_barrier_create(domain); + + if (domain->barrier) + kfence_put(domain->barrier); + + domain->barrier = barrier; +} +EXPORT_SYMBOL_GPL(async_barrier_domain); + /** * async_synchronize_full - synchronize all asynchronous function calls * @@ -264,6 +334,8 @@ EXPORT_SYMBOL_GPL(async_synchronize_full); void async_unregister_domain(struct async_domain *domain) { WARN_ON(!list_empty(&domain->pending)); + + async_barrier_delete(domain); domain->registered = 0; } EXPORT_SYMBOL_GPL(async_unregister_domain);