From patchwork Thu Aug 25 19:23:58 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konrad Rzeszutek Wilk X-Patchwork-Id: 9299687 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8DC0B607D8 for ; Thu, 25 Aug 2016 19:26:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7F80A29168 for ; Thu, 25 Aug 2016 19:26:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 740822919A; Thu, 25 Aug 2016 19:26:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4AAA128B74 for ; Thu, 25 Aug 2016 19:26:40 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bd0GD-00086w-5V; Thu, 25 Aug 2016 19:24:21 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bd0GB-00086K-V4 for xen-devel@lists.xenproject.org; Thu, 25 Aug 2016 19:24:20 +0000 Received: from [85.158.139.211] by server-3.bemta-5.messagelabs.com id 60/A4-01957-3E54FB75; Thu, 25 Aug 2016 19:24:19 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrJLMWRWlGSWpSXmKPExsXSO6nOVfeR6/5 wg5NNShbft0xmcmD0OPzhCksAYxRrZl5SfkUCa0bbp8msBb81K968/czSwPhfsYuRi0NIYCKT xNbFu9ggnN+MEov/nWOEcDYySvx4s4QdwpnGKLHlag9rFyMHB5uAicSbVY5djJwcIgKmEtsnn mQBqWEWmMYk8eLOFXaQhLCAt8TRrpdMIPUsAqoSi85pgoR5BdwkbnbvAhsjISAv8ex2PUiYU8 Bd4uaGPWBhIaCS1TOyQcISAoYSnzcuZZ7AyLeAkWEVo3pxalFZapGukV5SUWZ6RkluYmaOrqG BqV5uanFxYnpqTmJSsV5yfu4mRmCQMADBDsbvf5wOMUpyMCmJ8nrK7g8X4kvKT6nMSCzOiC8q zUktPsQow8GhJMG7zwUoJ1iUmp5akZaZAwxXmLQEB4+SCO8RZ6A0b3FBYm5xZjpE6hSjopQ4b wNInwBIIqM0D64NFiOXGGWlhHkZgQ4R4ilILcrNLEGVf8UozsGoJMw7B2QKT2ZeCdz0V0CLmY AWt9zfDbK4JBEhJdXAGPOSZ09Q/uLDvfb+s7f+aag4+Kd/3ZfZM+rrFO4Z1LLNWd3XufTz4+a baYZdkuWrppS+WO+0J2nrv0U3P99WXnzZjqt1557PLKdmX1+4t9ozTkviAotp5/YLbkEbandm /jpz6JeTb+p293kqIlFRzLwcDmbGnJ6T+fN5WRQqv+THfj6eknAuWYmlOCPRUIu5qDgRAEljn SOMAgAA X-Env-Sender: konrad.wilk@oracle.com X-Msg-Ref: server-15.tower-206.messagelabs.com!1472153056!42530430!1 X-Originating-IP: [141.146.126.69] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTQxLjE0Ni4xMjYuNjkgPT4gMjc3MjE4\n X-StarScan-Received: X-StarScan-Version: 8.84; banners=-,-,- X-VirusChecked: Checked Received: (qmail 18055 invoked from network); 25 Aug 2016 19:24:18 -0000 Received: from aserp1040.oracle.com (HELO aserp1040.oracle.com) (141.146.126.69) by server-15.tower-206.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 25 Aug 2016 19:24:18 -0000 Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u7PJOBBw000490 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Aug 2016 19:24:12 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0022.oracle.com (8.14.4/8.13.8) with ESMTP id u7PJOBmQ004800 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Aug 2016 19:24:11 GMT Received: from abhmp0013.oracle.com (abhmp0013.oracle.com [141.146.116.19]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id u7PJOAvk027970; Thu, 25 Aug 2016 19:24:11 GMT Received: from localhost.event.rightround.com (/75.98.193.200) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 25 Aug 2016 12:24:10 -0700 From: Konrad Rzeszutek Wilk To: xen-devel@lists.xenproject.org, konrad@kernel.org Date: Thu, 25 Aug 2016 15:23:58 -0400 Message-Id: <1472153041-14220-3-git-send-email-konrad.wilk@oracle.com> X-Mailer: git-send-email 2.4.11 In-Reply-To: <1472153041-14220-1-git-send-email-konrad.wilk@oracle.com> References: <1472153041-14220-1-git-send-email-konrad.wilk@oracle.com> X-Source-IP: userv0022.oracle.com [156.151.31.74] Cc: "Lan, Tianyu" , Kevin Tian , Jan Beulich , Konrad Rzeszutek Wilk , Andrew Cooper , Jun Nakajima Subject: [Xen-devel] [PATCH v2 2/5] tasklet: Add cross CPU feeding of per-cpu tasklets. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Since the per-cpu tasklets are lockfree and run only within their own CPU context - we need a mechanism for 'tasklet_schedule_on_cpu' from other CPUs to insert tasklets in the destination per-cpu lists. We use an IPI mechanism which function implements an per-cpu 'feeding' list and a global lock. When the IPI happens on the target CPU it will drain its per-cpu feeding list and add them in the per-cpu tasklet vector. With that in place we can now swap over all of the softirq_tasklet users to use it. That means we can also eliminate the percpu_tasklet scaffolding. Note that we don't do this tasklet schedule on other CPUs that often - during microcode update and when doing hypercall_continuation. This could be squashed in "tasklet: Introduce per-cpu tasklet for softirq." but the author thought it would be an easier aid in understanding the code with these parts split out. Signed-off-by: Konrad Rzeszutek Wilk --- RFC: First version v1: Posted, folks asked if ticketlocks fixed it. v2: Intel confirmed at XPDS 2016 that the problem is still present with large guests. Cc: Jan Beulich Cc: Andrew Cooper Cc: "Lan, Tianyu" Cc: Kevin Tian Cc: Jun Nakajima --- xen/arch/x86/hvm/hvm.c | 2 +- xen/common/tasklet.c | 50 +++++++++++++++++++++++++++++++++++++++-------- xen/include/xen/tasklet.h | 2 -- 3 files changed, 43 insertions(+), 11 deletions(-) diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c index d933ddd..0180f26 100644 --- a/xen/arch/x86/hvm/hvm.c +++ b/xen/arch/x86/hvm/hvm.c @@ -1511,7 +1511,7 @@ int hvm_vcpu_initialise(struct vcpu *v) if ( (rc = hvm_funcs.vcpu_initialise(v)) != 0 ) /* teardown: hvm_funcs.vcpu_destroy */ goto fail3; - percpu_tasklet_init( + softirq_tasklet_init( &v->arch.hvm_vcpu.assert_evtchn_irq_tasklet, (void(*)(unsigned long))hvm_assert_evtchn_irq, (unsigned long)v); diff --git a/xen/common/tasklet.c b/xen/common/tasklet.c index 319866f..d8f3cb3 100644 --- a/xen/common/tasklet.c +++ b/xen/common/tasklet.c @@ -30,8 +30,36 @@ static DEFINE_PER_CPU(struct list_head, softirq_tasklet_list); /* Protects all lists and tasklet structures. */ static DEFINE_SPINLOCK(tasklet_lock); +static DEFINE_SPINLOCK(feeder_lock); static DEFINE_PER_CPU(struct list_head, softirq_list); +static DEFINE_PER_CPU(struct list_head, tasklet_feeder); + +static void percpu_tasklet_feed(void *arg) +{ + unsigned long flags; + struct tasklet *t; + struct list_head *dst_list; + struct list_head *list = &__get_cpu_var(tasklet_feeder); + + spin_lock_irqsave(&feeder_lock, flags); + + if ( list_empty(list) ) + goto out; + + while ( !list_empty(list) ) + { + t = list_entry(list->next, struct tasklet, list); + BUG_ON(!t->is_percpu); + list_del(&t->list); + + dst_list = &__get_cpu_var(softirq_list); + list_add_tail(&t->list, dst_list); + } + raise_softirq(TASKLET_SOFTIRQ_PERCPU); +out: + spin_unlock_irqrestore(&feeder_lock, flags); +} static void tasklet_enqueue(struct tasklet *t) { @@ -44,7 +72,18 @@ static void tasklet_enqueue(struct tasklet *t) INIT_LIST_HEAD(&t->list); BUG_ON( !t->is_softirq ); - BUG_ON( cpu != smp_processor_id() ); /* Not implemented yet. */ + + if ( cpu != smp_processor_id() ) + { + spin_lock_irqsave(&feeder_lock, flags); + + list = &per_cpu(tasklet_feeder, cpu); + list_add_tail(&t->list, list); + + spin_unlock_irqrestore(&feeder_lock, flags); + on_selected_cpus(cpumask_of(cpu), percpu_tasklet_feed, NULL, 1); + return; + } local_irq_save(flags); @@ -314,13 +353,6 @@ void softirq_tasklet_init( { tasklet_init(t, func, data); t->is_softirq = 1; -} - -void percpu_tasklet_init( - struct tasklet *t, void (*func)(unsigned long), unsigned long data) -{ - tasklet_init(t, func, data); - t->is_softirq = 1; t->is_percpu = 1; } @@ -335,12 +367,14 @@ static int cpu_callback( INIT_LIST_HEAD(&per_cpu(tasklet_list, cpu)); INIT_LIST_HEAD(&per_cpu(softirq_tasklet_list, cpu)); INIT_LIST_HEAD(&per_cpu(softirq_list, cpu)); + INIT_LIST_HEAD(&per_cpu(tasklet_feeder, cpu)); break; case CPU_UP_CANCELED: case CPU_DEAD: migrate_tasklets_from_cpu(cpu, &per_cpu(tasklet_list, cpu)); migrate_tasklets_from_cpu(cpu, &per_cpu(softirq_tasklet_list, cpu)); migrate_tasklets_from_cpu(cpu, &per_cpu(softirq_list, cpu)); + migrate_tasklets_from_cpu(cpu, &per_cpu(tasklet_feeder, cpu)); break; default: break; diff --git a/xen/include/xen/tasklet.h b/xen/include/xen/tasklet.h index 9497c47..530a5e7 100644 --- a/xen/include/xen/tasklet.h +++ b/xen/include/xen/tasklet.h @@ -99,8 +99,6 @@ void tasklet_init( struct tasklet *t, void (*func)(unsigned long), unsigned long data); void softirq_tasklet_init( struct tasklet *t, void (*func)(unsigned long), unsigned long data); -void percpu_tasklet_init( - struct tasklet *t, void (*func)(unsigned long), unsigned long data); void tasklet_subsys_init(void); #endif /* __XEN_TASKLET_H__ */