From patchwork Mon Feb 24 08:43:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Roger_Pau_Monn=C3=A9?= X-Patchwork-Id: 11399493 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 106E6109A for ; Mon, 24 Feb 2020 08:45:58 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E11FF2072D for ; Mon, 24 Feb 2020 08:45:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="fqhOvlhG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E11FF2072D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1j69Lz-0001jW-Na; Mon, 24 Feb 2020 08:44:39 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1j69Ly-0001jR-Vt for xen-devel@lists.xenproject.org; Mon, 24 Feb 2020 08:44:39 +0000 X-Inumbo-ID: e17293a8-56e1-11ea-8ad0-12813bfff9fa Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id e17293a8-56e1-11ea-8ad0-12813bfff9fa; Mon, 24 Feb 2020 08:44:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1582533874; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bS39wVl3XFNMZ12sjHRKZl3ESjEQTHScX0sJ5tSBhjI=; b=fqhOvlhGTCvYi39L43o4CBac3ztULqbmIUQp8NvZs/6lnnwVbAKISfe7 LfdePYjxUSBPxjxTokunj8qsgipo4qk7DY84U3pHJVPvZVS1VJwF67P+t w4axpx8A8EYT3Wa/FSC5syw0sokVhtYfY+AuB5x+xtRnB9k6a3XncU9xv 0=; Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=roger.pau@citrix.com; spf=Pass smtp.mailfrom=roger.pau@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa5.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: 7Ir8UrxunJwrmA6olcBUWE49QDgM4HiRW9MZ+2Oaw+05CfKMuuvNPmCoU5O4G/8i5dZgwhH2RY H3T63ZemWqBLMeqeRout7/He0+doCjtt9SfQet2dhE88Gn5of5c9H2EHRBYFeGj2XQo5f87Z0K /bQWccBFa2S0ZShPNfs1zzhx6uPVQ/Ruppcx/ucrHHThyoLoH8eUHVVHzJrBTHD5op7AxmBEUM gSETbpsKem/5sw2giyIOctw27YTEIup3weOSnjOsHn/3gbZQD1Ih8Zi611NX2TkejmhuUOFcrY p20= X-SBRS: 2.7 X-MesageID: 13251359 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.70,479,1574139600"; d="scan'208";a="13251359" From: Roger Pau Monne To: Date: Mon, 24 Feb 2020 09:43:59 +0100 Message-ID: <20200224084400.94482-2-roger.pau@citrix.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200224084400.94482-1-roger.pau@citrix.com> References: <20200224084400.94482-1-roger.pau@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v3 1/2] atomic: add atomic_and operations X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Stefano Stabellini , Julien Grall , Wei Liu , Andrew Cooper , Jan Beulich , Volodymyr Babchuk , Roger Pau Monne Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" To x86 and Arm. This performs an atomic AND operation against an atomic_t variable with the provided mask. Requested-by: Jan Beulich Signed-off-by: Roger Pau Monné Reviewed-by: Jan Beulich Reviewed-by: Julien Grall --- xen/include/asm-arm/arm32/atomic.h | 17 +++++++++++++++++ xen/include/asm-arm/arm64/atomic.h | 14 ++++++++++++++ xen/include/asm-x86/atomic.h | 8 ++++++++ 3 files changed, 39 insertions(+) diff --git a/xen/include/asm-arm/arm32/atomic.h b/xen/include/asm-arm/arm32/atomic.h index c03eb684cd..4637381bcc 100644 --- a/xen/include/asm-arm/arm32/atomic.h +++ b/xen/include/asm-arm/arm32/atomic.h @@ -96,6 +96,23 @@ static inline int atomic_sub_return(int i, atomic_t *v) return result; } +static inline void atomic_and(unsigned int m, atomic_t *v) +{ + unsigned long tmp; + int result; + + prefetchw(&v->counter); + __asm__ __volatile__("@ atomic_and\n" +"1: ldrex %0, [%3]\n" +" and %0, %0, %4\n" +" strex %1, %0, [%3]\n" +" teq %1, #0\n" +" bne 1b" + : "=&r" (result), "=&r" (tmp), "+Qo" (v->counter) + : "r" (&v->counter), "Ir" (m) + : "cc"); +} + static inline int atomic_cmpxchg(atomic_t *ptr, int old, int new) { int oldval; diff --git a/xen/include/asm-arm/arm64/atomic.h b/xen/include/asm-arm/arm64/atomic.h index bce38d4ca2..2f906f0265 100644 --- a/xen/include/asm-arm/arm64/atomic.h +++ b/xen/include/asm-arm/arm64/atomic.h @@ -91,6 +91,20 @@ static inline int atomic_sub_return(int i, atomic_t *v) return result; } +static inline void atomic_and(unsigned int m, atomic_t *v) +{ + unsigned long tmp; + int result; + + asm volatile("// atomic_and\n" +"1: ldxr %w0, %2\n" +" and %w0, %w0, %w3\n" +" stxr %w1, %w0, %2\n" +" cbnz %w1, 1b" + : "=&r" (result), "=&r" (tmp), "+Q" (v->counter) + : "Ir" (m)); +} + static inline int atomic_cmpxchg(atomic_t *ptr, int old, int new) { unsigned long tmp; diff --git a/xen/include/asm-x86/atomic.h b/xen/include/asm-x86/atomic.h index 682bcf91b1..21794bfe7b 100644 --- a/xen/include/asm-x86/atomic.h +++ b/xen/include/asm-x86/atomic.h @@ -224,6 +224,14 @@ static inline int atomic_add_unless(atomic_t *v, int a, int u) return c; } +static inline void atomic_and(unsigned int m, atomic_t *v) +{ + asm volatile ( + "lock; andl %1,%0" + : "=m" (*(volatile int *)&v->counter) + : "ir" (m), "m" (*(volatile int *)&v->counter) ); +} + #define atomic_xchg(v, new) (xchg(&((v)->counter), new)) #endif /* __ARCH_X86_ATOMIC__ */ From patchwork Mon Feb 24 08:44:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Roger_Pau_Monn=C3=A9?= X-Patchwork-Id: 11399497 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E9CC8109A for ; Mon, 24 Feb 2020 08:46:11 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C5F712072D for ; Mon, 24 Feb 2020 08:46:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="BjdAE+Zn" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C5F712072D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1j69M5-0001jv-15; Mon, 24 Feb 2020 08:44:45 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1j69M3-0001jl-Vt for xen-devel@lists.xenproject.org; Mon, 24 Feb 2020 08:44:44 +0000 X-Inumbo-ID: e2f305dc-56e1-11ea-8ad0-12813bfff9fa Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id e2f305dc-56e1-11ea-8ad0-12813bfff9fa; Mon, 24 Feb 2020 08:44:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1582533876; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hDOEAljPrqrZg8kT60UnipHFRoxjXCLudTIsaHhqbG8=; b=BjdAE+Zne1M/jhYa3swlXOmoZwt4VS3iIaEDcdPve/AbqT/Qbgo+HOsV rYafTUCDyAvzCUjVW6pINA8YnlmrIaJ7w4E8K9vz8FwIb64sWz45HaM9W FvU0q6JThGhiIehJoF7kQ5nDc4mGbhSyE53+1QLtj/1IJ825qbEBSWnCe 4=; Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=roger.pau@citrix.com; spf=Pass smtp.mailfrom=roger.pau@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa5.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: XjaTVafm1LWaHSlEwJuPvMlW1jLrLcmvab3P8F5roK5qA2R5dKT3R4zaz8Hgzjua2Xo2NjfZ77 0oJv+rNIWxj4e6eJWD4lXQ1dD+QL+lz5mqZ0o4qeobPUHSDEOenAoti3CaL6+gO2GcoCEFhgPM b7szfOhPdNpBjRah2J7NX6loEe/23V8uSfxxqPmrpKjaTfobgTWEzrfF4+4/X/J40VKfjpY6/F PRUX2jNbW3qFSf3hbABV/1BaQHn6AQ3RFyIX5j7jsKCVOC+JOsTN3vkSYltQMpgS0mMHSvrAOh NNE= X-SBRS: 2.7 X-MesageID: 13251361 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.70,479,1574139600"; d="scan'208";a="13251361" From: Roger Pau Monne To: Date: Mon, 24 Feb 2020 09:44:00 +0100 Message-ID: <20200224084400.94482-3-roger.pau@citrix.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200224084400.94482-1-roger.pau@citrix.com> References: <20200224084400.94482-1-roger.pau@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v3 2/2] rwlock: allow recursive read locking when already locked in write mode X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: =?utf-8?b?SsO8cmdlbiBHcm/Dnw==?= , Stefano Stabellini , Julien Grall , Wei Liu , Konrad Rzeszutek Wilk , George Dunlap , Andrew Cooper , Ian Jackson , Jan Beulich , Roger Pau Monne Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Allow a CPU already holding the lock in write mode to also lock it in read mode. There's no harm in allowing read locking a rwlock that's already owned by the caller (ie: CPU) in write mode. Allowing such accesses is required at least for the CPU maps use-case. In order to do this reserve 12bits of the lock, this allows to support up to 4096 CPUs. Also reduce the write lock mask to 2 bits: one to signal there are pending writers waiting on the lock and the other to signal the lock is owned in write mode. This reduces the maximum number of concurrent readers from 16777216 to 262144, I think this should still be enough, or else the lock field can be expanded from 32 to 64bits if all architectures support atomic operations on 64bit integers. Fixes: 5872c83b42c608 ('smp: convert the cpu maps lock into a rw lock') Reported-by: Jan Beulich Reported-by: Jürgen Groß Signed-off-by: Roger Pau Monné Reviewed-by: Jan Beulich Reviewed-by: Julien Grall --- Changes since v2: - Use atomic_and to clear the writers part of the lock when write-unlocking. - Reduce writer-related area to 14bits. - Include xen/smp.h for Arm64. Changes since v1: - Move the processor ID field to start at bit 0 and hence avoid the shift when checking it. - Enlarge write related structures to use 16bit, and hence allow them to be cleared using an atomic write. --- xen/common/rwlock.c | 4 ++-- xen/include/xen/rwlock.h | 52 ++++++++++++++++++++++++---------------- 2 files changed, 34 insertions(+), 22 deletions(-) diff --git a/xen/common/rwlock.c b/xen/common/rwlock.c index d568bbf6de..dadab372b5 100644 --- a/xen/common/rwlock.c +++ b/xen/common/rwlock.c @@ -69,7 +69,7 @@ void queue_write_lock_slowpath(rwlock_t *lock) /* Try to acquire the lock directly if no reader is present. */ if ( !atomic_read(&lock->cnts) && - (atomic_cmpxchg(&lock->cnts, 0, _QW_LOCKED) == 0) ) + (atomic_cmpxchg(&lock->cnts, 0, _write_lock_val()) == 0) ) goto unlock; /* @@ -93,7 +93,7 @@ void queue_write_lock_slowpath(rwlock_t *lock) cnts = atomic_read(&lock->cnts); if ( (cnts == _QW_WAITING) && (atomic_cmpxchg(&lock->cnts, _QW_WAITING, - _QW_LOCKED) == _QW_WAITING) ) + _write_lock_val()) == _QW_WAITING) ) break; cpu_relax(); diff --git a/xen/include/xen/rwlock.h b/xen/include/xen/rwlock.h index 3dfea1ac2a..82af36b1b4 100644 --- a/xen/include/xen/rwlock.h +++ b/xen/include/xen/rwlock.h @@ -2,6 +2,7 @@ #define __RWLOCK_H__ #include +#include #include #include @@ -20,21 +21,30 @@ typedef struct { #define DEFINE_RWLOCK(l) rwlock_t l = RW_LOCK_UNLOCKED #define rwlock_init(l) (*(l) = (rwlock_t)RW_LOCK_UNLOCKED) -/* - * Writer states & reader shift and bias. - * - * Writer field is 8 bit to allow for potential optimisation, see - * _write_unlock(). - */ -#define _QW_WAITING 1 /* A writer is waiting */ -#define _QW_LOCKED 0xff /* A writer holds the lock */ -#define _QW_WMASK 0xff /* Writer mask.*/ -#define _QR_SHIFT 8 /* Reader count shift */ +/* Writer states & reader shift and bias. */ +#define _QW_CPUMASK 0xfff /* Writer CPU mask */ +#define _QW_SHIFT 12 /* Writer flags shift */ +#define _QW_WAITING (1u << _QW_SHIFT) /* A writer is waiting */ +#define _QW_LOCKED (3u << _QW_SHIFT) /* A writer holds the lock */ +#define _QW_WMASK (3u << _QW_SHIFT) /* Writer mask */ +#define _QR_SHIFT 14 /* Reader count shift */ #define _QR_BIAS (1U << _QR_SHIFT) void queue_read_lock_slowpath(rwlock_t *lock); void queue_write_lock_slowpath(rwlock_t *lock); +static inline bool _is_write_locked_by_me(uint32_t cnts) +{ + BUILD_BUG_ON(_QW_CPUMASK < NR_CPUS); + return (cnts & _QW_WMASK) == _QW_LOCKED && + (cnts & _QW_CPUMASK) == smp_processor_id(); +} + +static inline bool _can_read_lock(uint32_t cnts) +{ + return !(cnts & _QW_WMASK) || _is_write_locked_by_me(cnts); +} + /* * _read_trylock - try to acquire read lock of a queue rwlock. * @lock : Pointer to queue rwlock structure. @@ -45,10 +55,10 @@ static inline int _read_trylock(rwlock_t *lock) u32 cnts; cnts = atomic_read(&lock->cnts); - if ( likely(!(cnts & _QW_WMASK)) ) + if ( likely(_can_read_lock(cnts)) ) { cnts = (u32)atomic_add_return(_QR_BIAS, &lock->cnts); - if ( likely(!(cnts & _QW_WMASK)) ) + if ( likely(_can_read_lock(cnts)) ) return 1; atomic_sub(_QR_BIAS, &lock->cnts); } @@ -64,7 +74,7 @@ static inline void _read_lock(rwlock_t *lock) u32 cnts; cnts = atomic_add_return(_QR_BIAS, &lock->cnts); - if ( likely(!(cnts & _QW_WMASK)) ) + if ( likely(_can_read_lock(cnts)) ) return; /* The slowpath will decrement the reader count, if necessary. */ @@ -115,6 +125,11 @@ static inline int _rw_is_locked(rwlock_t *lock) return atomic_read(&lock->cnts); } +static inline uint32_t _write_lock_val(void) +{ + return _QW_LOCKED | smp_processor_id(); +} + /* * queue_write_lock - acquire write lock of a queue rwlock. * @lock : Pointer to queue rwlock structure. @@ -122,7 +137,7 @@ static inline int _rw_is_locked(rwlock_t *lock) static inline void _write_lock(rwlock_t *lock) { /* Optimize for the unfair lock case where the fair flag is 0. */ - if ( atomic_cmpxchg(&lock->cnts, 0, _QW_LOCKED) == 0 ) + if ( atomic_cmpxchg(&lock->cnts, 0, _write_lock_val()) == 0 ) return; queue_write_lock_slowpath(lock); @@ -157,16 +172,13 @@ static inline int _write_trylock(rwlock_t *lock) if ( unlikely(cnts) ) return 0; - return likely(atomic_cmpxchg(&lock->cnts, 0, _QW_LOCKED) == 0); + return likely(atomic_cmpxchg(&lock->cnts, 0, _write_lock_val()) == 0); } static inline void _write_unlock(rwlock_t *lock) { - /* - * If the writer field is atomic, it can be cleared directly. - * Otherwise, an atomic subtraction will be used to clear it. - */ - atomic_sub(_QW_LOCKED, &lock->cnts); + ASSERT(_is_write_locked_by_me(atomic_read(&lock->cnts))); + atomic_and(~(_QW_CPUMASK | _QW_WMASK), &lock->cnts); } static inline void _write_unlock_irq(rwlock_t *lock)