From patchwork Thu Apr 17 15:04:00 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 4009401 Return-Path: X-Original-To: patchwork-kvm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 0BA5EC0DA2 for ; Thu, 17 Apr 2014 15:08:22 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 17EDE200F3 for ; Thu, 17 Apr 2014 15:08:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2507020380 for ; Thu, 17 Apr 2014 15:08:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752737AbaDQPHi (ORCPT ); Thu, 17 Apr 2014 11:07:38 -0400 Received: from g5t1627.atlanta.hp.com ([15.192.137.10]:43490 "EHLO g5t1627.atlanta.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751786AbaDQPEw (ORCPT ); Thu, 17 Apr 2014 11:04:52 -0400 Received: from g5t1633.atlanta.hp.com (g5t1633.atlanta.hp.com [16.201.144.132]) by g5t1627.atlanta.hp.com (Postfix) with ESMTP id 9176927A; Thu, 17 Apr 2014 15:04:49 +0000 (UTC) Received: from RHEL65.localdomain (unknown [16.99.12.168]) by g5t1633.atlanta.hp.com (Postfix) with ESMTP id 7313691; Thu, 17 Apr 2014 15:04:48 +0000 (UTC) From: Waiman Long To: Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , Peter Zijlstra Cc: linux-arch@vger.kernel.org, x86@kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, xen-devel@lists.xenproject.org, kvm@vger.kernel.org, Paolo Bonzini , Konrad Rzeszutek Wilk , "Paul E. McKenney" , Rik van Riel , Linus Torvalds , Raghavendra K T , David Vrabel , Oleg Nesterov , Gleb Natapov , Scott J Norton , Chegu Vinod , Waiman Long Subject: [PATCH v9 08/19] qspinlock: Make a new qnode structure to support virtualization Date: Thu, 17 Apr 2014 11:04:00 -0400 Message-Id: <1397747051-15401-9-git-send-email-Waiman.Long@hp.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1397747051-15401-1-git-send-email-Waiman.Long@hp.com> References: <1397747051-15401-1-git-send-email-Waiman.Long@hp.com> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In order to support additional virtualization features like unfair lock and para-virtualized spinlock, it is necessary to store additional CPU specific data into the queue node structure. As a result, a new qnode structure is created and the mcs_spinlock structure is now part of the new structure. It is also necessary to expand arch_mcs_spin_lock_contended() to the underlying while loop as additional code will need to be inserted into the loop. Signed-off-by: Waiman Long --- kernel/locking/qspinlock.c | 36 +++++++++++++++++++++++------------- 1 files changed, 23 insertions(+), 13 deletions(-) diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index 80fe9ee..994358c 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -61,12 +61,21 @@ #include "mcs_spinlock.h" /* + * To have additional features for better virtualization support, it is + * necessary to store additional data in the queue node structure. So + * a new queue node structure will have to be defined and used here. + */ +struct qnode { + struct mcs_spinlock mcs; +}; + +/* * Per-CPU queue node structures; we can never have more than 4 nested * contexts: task, softirq, hardirq, nmi. * * Exactly fits one cacheline. */ -static DEFINE_PER_CPU_ALIGNED(struct mcs_spinlock, mcs_nodes[4]); +static DEFINE_PER_CPU_ALIGNED(struct qnode, qnodes[4]); /* * We must be able to distinguish between no-tail and the tail at 0:0, @@ -83,12 +92,12 @@ static inline u32 encode_tail(int cpu, int idx) return tail; } -static inline struct mcs_spinlock *decode_tail(u32 tail) +static inline struct qnode *decode_tail(u32 tail) { int cpu = (tail >> _Q_TAIL_CPU_OFFSET) - 1; int idx = (tail & _Q_TAIL_IDX_MASK) >> _Q_TAIL_IDX_OFFSET; - return per_cpu_ptr(&mcs_nodes[idx], cpu); + return per_cpu_ptr(&qnodes[idx], cpu); } #define _Q_LOCKED_PENDING_MASK (_Q_LOCKED_MASK | _Q_PENDING_MASK) @@ -343,7 +352,7 @@ static inline int trylock_pending(struct qspinlock *lock, u32 *pval) */ void queue_spin_lock_slowpath(struct qspinlock *lock, u32 val) { - struct mcs_spinlock *prev, *next, *node; + struct qnode *prev, *next, *node; u32 old, tail; int idx; @@ -352,13 +361,13 @@ void queue_spin_lock_slowpath(struct qspinlock *lock, u32 val) if (trylock_pending(lock, &val)) return; /* Lock acquired */ - node = this_cpu_ptr(&mcs_nodes[0]); - idx = node->count++; + node = this_cpu_ptr(&qnodes[0]); + idx = node->mcs.count++; tail = encode_tail(smp_processor_id(), idx); node += idx; - node->locked = 0; - node->next = NULL; + node->mcs.locked = 0; + node->mcs.next = NULL; /* * We touched a (possibly) cold cacheline; attempt the trylock once @@ -381,9 +390,10 @@ void queue_spin_lock_slowpath(struct qspinlock *lock, u32 val) */ if (old & _Q_TAIL_MASK) { prev = decode_tail(old); - ACCESS_ONCE(prev->next) = node; + ACCESS_ONCE(prev->mcs.next) = (struct mcs_spinlock *)node; - arch_mcs_spin_lock_contended(&node->locked); + while (!smp_load_acquire(&node->mcs.locked)) + arch_mutex_cpu_relax(); } /* @@ -423,15 +433,15 @@ void queue_spin_lock_slowpath(struct qspinlock *lock, u32 val) /* * contended path; wait for next, release. */ - while (!(next = ACCESS_ONCE(node->next))) + while (!(next = (struct qnode *)ACCESS_ONCE(node->mcs.next))) arch_mutex_cpu_relax(); - arch_mcs_spin_unlock_contended(&next->locked); + arch_mcs_spin_unlock_contended(&next->mcs.locked); release: /* * release the node */ - this_cpu_dec(mcs_nodes[0].count); + this_cpu_dec(qnodes[0].mcs.count); } EXPORT_SYMBOL(queue_spin_lock_slowpath);