From patchwork Mon Jun 10 19:14:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 10985335 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9F58E13AD for ; Mon, 10 Jun 2019 19:15:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8E5DF1FF7F for ; Mon, 10 Jun 2019 19:15:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 82E9327F7F; Mon, 10 Jun 2019 19:15:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D0621FF7F for ; Mon, 10 Jun 2019 19:15:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389141AbfFJTOa (ORCPT ); Mon, 10 Jun 2019 15:14:30 -0400 Received: from mail-ua1-f67.google.com ([209.85.222.67]:47081 "EHLO mail-ua1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389113AbfFJTOa (ORCPT ); Mon, 10 Jun 2019 15:14:30 -0400 Received: by mail-ua1-f67.google.com with SMTP id o19so3523786uap.13; Mon, 10 Jun 2019 12:14:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IyGtykocY14Ccf/Pb7GMn65evkFypDyeXC43FQG/uJM=; b=iGxLfyd0GjLWmcUlOrC7eeHsJSAboo5VdeQkIsunxAg8Ms7kDdxkVCEEDY3kP4xYD1 +21CeQ8hwl4zE/dufWp1gLf092ahOaIQdwza1RUot1CM49cZQTQNT9/dVLPVfcasOnw+ +1adBiOJ3veiPoJjxEjpxjfbBUZcHi5QRQy6V9nZEuA/cyYXY15AICjtSBm9eK9E7dma 1jTgJpw2/Y0uG2nFLGdtV4+tt/M+ODinHrM4f24ckfUknVGLFnjEvh0BCqNsNKc7U/WK NZjmf+Pe++kVmIGHHSyM4y3a6teR4axhv+uevtp6yuVPqIE3aJZHzbWVNvxLz9xXMAwO WtMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IyGtykocY14Ccf/Pb7GMn65evkFypDyeXC43FQG/uJM=; b=DqRJh2ZFfal2DBE/mWzTHfkg9M87MzF7a2NwTayF8P6wa6JZitx0wUhynRSx1+IIvX 5xj+42nS4/04HAfmBbheoffrqgD4pPefpypT3Ah6/9RdvCN0+ZmI1fx7aGjNt8Lj3DzV s/6hOQ/jW8pof75VnFcOMi3ju6O6qm/fjsTfpvT3vkyflpSX7FDGUmwPw0fAFmI8QMS7 4rI9iErG6C+r5kGMhUulVjniUSsemyY5ys0oUY0FC14qIGom1764d1N09PtlZtYr800R I6ZCK2ej+3K+mfD4nUaHWkrI3XaC3vgyGi1WfNaxnifhx2eQ7graW8dA2lmjEmtVdgfQ bJ6A== X-Gm-Message-State: APjAAAWjKFGtYeHeCIh6p5HD1N33oHMunboV1k0qa3wWUKUAN5L+Hw5d QPV5XFLKOufUZ6XFlHFNEK9ZuMdBtg== X-Google-Smtp-Source: APXvYqxBa5o5NGvWvlKZInrXsNGxwCGTRbVoWbIZEbgmX9sm4XiN2RN5BsLVB1jzr15bbkRrJnnk3Q== X-Received: by 2002:ab0:1388:: with SMTP id m8mr18502081uae.53.1560194068624; Mon, 10 Jun 2019 12:14:28 -0700 (PDT) Received: from kmo-pixel.hsd1.vt.comcast.net (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id t20sm4834014vkd.53.2019.06.10.12.14.27 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 10 Jun 2019 12:14:27 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 01/12] Compiler Attributes: add __flatten Date: Mon, 10 Jun 2019 15:14:09 -0400 Message-Id: <20190610191420.27007-2-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190610191420.27007-1-kent.overstreet@gmail.com> References: <20190610191420.27007-1-kent.overstreet@gmail.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Kent Overstreet --- include/linux/compiler_attributes.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/linux/compiler_attributes.h b/include/linux/compiler_attributes.h index 6b318efd8a..48b2c6ae6f 100644 --- a/include/linux/compiler_attributes.h +++ b/include/linux/compiler_attributes.h @@ -253,4 +253,9 @@ */ #define __weak __attribute__((__weak__)) +/* + * gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-flatten-function-attribute + */ +#define __flatten __attribute__((flatten)) + #endif /* __LINUX_COMPILER_ATTRIBUTES_H */ From patchwork Mon Jun 10 19:14:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 10985333 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BDA6613AD for ; Mon, 10 Jun 2019 19:15:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ACC3427F7F for ; Mon, 10 Jun 2019 19:15:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A150428305; Mon, 10 Jun 2019 19:15:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 611AD27F7F for ; Mon, 10 Jun 2019 19:15:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389170AbfFJTOd (ORCPT ); Mon, 10 Jun 2019 15:14:33 -0400 Received: from mail-ua1-f65.google.com ([209.85.222.65]:36351 "EHLO mail-ua1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389152AbfFJTOc (ORCPT ); Mon, 10 Jun 2019 15:14:32 -0400 Received: by mail-ua1-f65.google.com with SMTP id 94so3529970uam.3; Mon, 10 Jun 2019 12:14:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hXrfDEoF/55M53i73uGkMDRlja8di0X+2AypmqfVHNw=; b=HG+V5/zvYTs/OcY+6lCJro/4MjxQLE9fc2bGIAYrMOchwrcdpbBzc/ylWiMde+HDP0 Q7sdcOtN+mnS6ElF+MOaqEuoRwSMRGbt/wLFstfD3RrzI8Q2Fw9ls4mq4afMTqdIchaX ClFFJvzX1ARGpbyZ7YiqelIpLiVAqssZMBUjxviLbdD+rCPejPk47GmInwTOchKEEL2I XQq/aaOpL6qP9z1UYPigB0H9hL5iSx42aDHvtmWdcXqBEGYgIb3MfWY8lzPUyngY5R7T TroIShd6QuCN0lixcf3Rd1OMwLyvq7ETZ8aEIERmir9hiw7D37yVjxTTrpA29WCG0sm2 SRDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hXrfDEoF/55M53i73uGkMDRlja8di0X+2AypmqfVHNw=; b=bz+ogdBKq3eRvYBS+XApWOn4XC3MyH1iPH2LxP8BLPe+pfkbqVuybNR2oHOJY0vG21 uDl+BIUqJK0YyA5evsOkwkq5paXn9MAgqvddxxA9Q2EZiL3pxa/U/3EB+srVgwzlCYlm jsB7W6Pji+3WBNbbTcFHewpBP1k3GuvtnXQGg0D2+kdTXATf4XK1+b4s9UUtBAqA2rdF QBwBFseZ9khtn6h4RbqlNRsx2v0Mb1LSN2XuEsJoMGe1b+BlFeblZx6JbmIsAa41Ahf8 EFI/LKwNXd12e9+g8JMx9FzVYhXJxPfDvrMYDFffSKwrXqQqNI65Zlnpg2rZXf7F3bxt CXLw== X-Gm-Message-State: APjAAAUA2gX59+nI7RdGYXwAeA+T5SfOLYCLSohRJufgb67JIoEEMzdE smAFDkGGpiWT94hooRUtA/jXODB2GA== X-Google-Smtp-Source: APXvYqzgyrJme4y6dhoTeDlnTufFruTAUoP3Z3oZN38ufhtPFAUIr1r4mX1SDnHq92DBZtThmtGFyA== X-Received: by 2002:a9f:25e9:: with SMTP id 96mr22056402uaf.95.1560194070090; Mon, 10 Jun 2019 12:14:30 -0700 (PDT) Received: from kmo-pixel.hsd1.vt.comcast.net (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id t20sm4834014vkd.53.2019.06.10.12.14.28 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 10 Jun 2019 12:14:29 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 02/12] locking: SIX locks (shared/intent/exclusive) Date: Mon, 10 Jun 2019 15:14:10 -0400 Message-Id: <20190610191420.27007-3-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190610191420.27007-1-kent.overstreet@gmail.com> References: <20190610191420.27007-1-kent.overstreet@gmail.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP New lock for bcachefs, like read/write locks but with a third state, intent. Intent locks conflict with each other, but not with read locks; taking a write lock requires first holding an intent lock. Signed-off-by: Kent Overstreet --- include/linux/six.h | 192 +++++++++++++++ kernel/Kconfig.locks | 3 + kernel/locking/Makefile | 1 + kernel/locking/six.c | 512 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 708 insertions(+) create mode 100644 include/linux/six.h create mode 100644 kernel/locking/six.c diff --git a/include/linux/six.h b/include/linux/six.h new file mode 100644 index 0000000000..0fb1b2f493 --- /dev/null +++ b/include/linux/six.h @@ -0,0 +1,192 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _LINUX_SIX_H +#define _LINUX_SIX_H + +/* + * Shared/intent/exclusive locks: sleepable read/write locks, much like rw + * semaphores, except with a third intermediate state, intent. Basic operations + * are: + * + * six_lock_read(&foo->lock); + * six_unlock_read(&foo->lock); + * + * six_lock_intent(&foo->lock); + * six_unlock_intent(&foo->lock); + * + * six_lock_write(&foo->lock); + * six_unlock_write(&foo->lock); + * + * Intent locks block other intent locks, but do not block read locks, and you + * must have an intent lock held before taking a write lock, like so: + * + * six_lock_intent(&foo->lock); + * six_lock_write(&foo->lock); + * six_unlock_write(&foo->lock); + * six_unlock_intent(&foo->lock); + * + * Other operations: + * + * six_trylock_read() + * six_trylock_intent() + * six_trylock_write() + * + * six_lock_downgrade(): convert from intent to read + * six_lock_tryupgrade(): attempt to convert from read to intent + * + * Locks also embed a sequence number, which is incremented when the lock is + * locked or unlocked for write. The current sequence number can be grabbed + * while a lock is held from lock->state.seq; then, if you drop the lock you can + * use six_relock_(read|intent_write)(lock, seq) to attempt to retake the lock + * iff it hasn't been locked for write in the meantime. + * + * There are also operations that take the lock type as a parameter, where the + * type is one of SIX_LOCK_read, SIX_LOCK_intent, or SIX_LOCK_write: + * + * six_lock_type(lock, type) + * six_unlock_type(lock, type) + * six_relock(lock, type, seq) + * six_trylock_type(lock, type) + * six_trylock_convert(lock, from, to) + * + * A lock may be held multiple types by the same thread (for read or intent, + * not write). However, the six locks code does _not_ implement the actual + * recursive checks itself though - rather, if your code (e.g. btree iterator + * code) knows that the current thread already has a lock held, and for the + * correct type, six_lock_increment() may be used to bump up the counter for + * that type - the only effect is that one more call to unlock will be required + * before the lock is unlocked. + */ + +#include +#include +#include +#include + +#define SIX_LOCK_SEPARATE_LOCKFNS + +union six_lock_state { + struct { + atomic64_t counter; + }; + + struct { + u64 v; + }; + + struct { + /* for waitlist_bitnr() */ + unsigned long l; + }; + + struct { + unsigned read_lock:28; + unsigned intent_lock:1; + unsigned waiters:3; + /* + * seq works much like in seqlocks: it's incremented every time + * we lock and unlock for write. + * + * If it's odd write lock is held, even unlocked. + * + * Thus readers can unlock, and then lock again later iff it + * hasn't been modified in the meantime. + */ + u32 seq; + }; +}; + +enum six_lock_type { + SIX_LOCK_read, + SIX_LOCK_intent, + SIX_LOCK_write, +}; + +struct six_lock { + union six_lock_state state; + unsigned intent_lock_recurse; + struct task_struct *owner; + struct optimistic_spin_queue osq; + + raw_spinlock_t wait_lock; + struct list_head wait_list[2]; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; +#endif +}; + +static __always_inline void __six_lock_init(struct six_lock *lock, + const char *name, + struct lock_class_key *key) +{ + atomic64_set(&lock->state.counter, 0); + raw_spin_lock_init(&lock->wait_lock); + INIT_LIST_HEAD(&lock->wait_list[SIX_LOCK_read]); + INIT_LIST_HEAD(&lock->wait_list[SIX_LOCK_intent]); +#ifdef CONFIG_DEBUG_LOCK_ALLOC + debug_check_no_locks_freed((void *) lock, sizeof(*lock)); + lockdep_init_map(&lock->dep_map, name, key, 0); +#endif +} + +#define six_lock_init(lock) \ +do { \ + static struct lock_class_key __key; \ + \ + __six_lock_init((lock), #lock, &__key); \ +} while (0) + +#define __SIX_VAL(field, _v) (((union six_lock_state) { .field = _v }).v) + +#define __SIX_LOCK(type) \ +bool six_trylock_##type(struct six_lock *); \ +bool six_relock_##type(struct six_lock *, u32); \ +void six_lock_##type(struct six_lock *); \ +void six_unlock_##type(struct six_lock *); + +__SIX_LOCK(read) +__SIX_LOCK(intent) +__SIX_LOCK(write) +#undef __SIX_LOCK + +#define SIX_LOCK_DISPATCH(type, fn, ...) \ + switch (type) { \ + case SIX_LOCK_read: \ + return fn##_read(__VA_ARGS__); \ + case SIX_LOCK_intent: \ + return fn##_intent(__VA_ARGS__); \ + case SIX_LOCK_write: \ + return fn##_write(__VA_ARGS__); \ + default: \ + BUG(); \ + } + +static inline bool six_trylock_type(struct six_lock *lock, enum six_lock_type type) +{ + SIX_LOCK_DISPATCH(type, six_trylock, lock); +} + +static inline bool six_relock_type(struct six_lock *lock, enum six_lock_type type, + unsigned seq) +{ + SIX_LOCK_DISPATCH(type, six_relock, lock, seq); +} + +static inline void six_lock_type(struct six_lock *lock, enum six_lock_type type) +{ + SIX_LOCK_DISPATCH(type, six_lock, lock); +} + +static inline void six_unlock_type(struct six_lock *lock, enum six_lock_type type) +{ + SIX_LOCK_DISPATCH(type, six_unlock, lock); +} + +void six_lock_downgrade(struct six_lock *); +bool six_lock_tryupgrade(struct six_lock *); +bool six_trylock_convert(struct six_lock *, enum six_lock_type, + enum six_lock_type); + +void six_lock_increment(struct six_lock *, enum six_lock_type); + +#endif /* _LINUX_SIX_H */ diff --git a/kernel/Kconfig.locks b/kernel/Kconfig.locks index fbba478ae5..ff3e6121ae 100644 --- a/kernel/Kconfig.locks +++ b/kernel/Kconfig.locks @@ -251,3 +251,6 @@ config ARCH_USE_QUEUED_RWLOCKS config QUEUED_RWLOCKS def_bool y if ARCH_USE_QUEUED_RWLOCKS depends on SMP + +config SIXLOCKS + bool diff --git a/kernel/locking/Makefile b/kernel/locking/Makefile index 392c7f23af..9a73c8564e 100644 --- a/kernel/locking/Makefile +++ b/kernel/locking/Makefile @@ -30,3 +30,4 @@ obj-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem-xadd.o obj-$(CONFIG_QUEUED_RWLOCKS) += qrwlock.o obj-$(CONFIG_LOCK_TORTURE_TEST) += locktorture.o obj-$(CONFIG_WW_MUTEX_SELFTEST) += test-ww_mutex.o +obj-$(CONFIG_SIXLOCKS) += six.o diff --git a/kernel/locking/six.c b/kernel/locking/six.c new file mode 100644 index 0000000000..9fa58b6fad --- /dev/null +++ b/kernel/locking/six.c @@ -0,0 +1,512 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include +#include +#include +#include +#include + +#ifdef DEBUG +#define EBUG_ON(cond) BUG_ON(cond) +#else +#define EBUG_ON(cond) do {} while (0) +#endif + +#define six_acquire(l, t) lock_acquire(l, 0, t, 0, 0, NULL, _RET_IP_) +#define six_release(l) lock_release(l, 0, _RET_IP_) + +struct six_lock_vals { + /* Value we add to the lock in order to take the lock: */ + u64 lock_val; + + /* If the lock has this value (used as a mask), taking the lock fails: */ + u64 lock_fail; + + /* Value we add to the lock in order to release the lock: */ + u64 unlock_val; + + /* Mask that indicates lock is held for this type: */ + u64 held_mask; + + /* Waitlist we wakeup when releasing the lock: */ + enum six_lock_type unlock_wakeup; +}; + +#define __SIX_LOCK_HELD_read __SIX_VAL(read_lock, ~0) +#define __SIX_LOCK_HELD_intent __SIX_VAL(intent_lock, ~0) +#define __SIX_LOCK_HELD_write __SIX_VAL(seq, 1) + +#define LOCK_VALS { \ + [SIX_LOCK_read] = { \ + .lock_val = __SIX_VAL(read_lock, 1), \ + .lock_fail = __SIX_LOCK_HELD_write, \ + .unlock_val = -__SIX_VAL(read_lock, 1), \ + .held_mask = __SIX_LOCK_HELD_read, \ + .unlock_wakeup = SIX_LOCK_write, \ + }, \ + [SIX_LOCK_intent] = { \ + .lock_val = __SIX_VAL(intent_lock, 1), \ + .lock_fail = __SIX_LOCK_HELD_intent, \ + .unlock_val = -__SIX_VAL(intent_lock, 1), \ + .held_mask = __SIX_LOCK_HELD_intent, \ + .unlock_wakeup = SIX_LOCK_intent, \ + }, \ + [SIX_LOCK_write] = { \ + .lock_val = __SIX_VAL(seq, 1), \ + .lock_fail = __SIX_LOCK_HELD_read, \ + .unlock_val = __SIX_VAL(seq, 1), \ + .held_mask = __SIX_LOCK_HELD_write, \ + .unlock_wakeup = SIX_LOCK_read, \ + }, \ +} + +static inline void six_set_owner(struct six_lock *lock, enum six_lock_type type, + union six_lock_state old) +{ + if (type != SIX_LOCK_intent) + return; + + if (!old.intent_lock) { + EBUG_ON(lock->owner); + lock->owner = current; + } else { + EBUG_ON(lock->owner != current); + } +} + +static __always_inline bool do_six_trylock_type(struct six_lock *lock, + enum six_lock_type type) +{ + const struct six_lock_vals l[] = LOCK_VALS; + union six_lock_state old; + u64 v = READ_ONCE(lock->state.v); + + EBUG_ON(type == SIX_LOCK_write && lock->owner != current); + + do { + old.v = v; + + EBUG_ON(type == SIX_LOCK_write && + ((old.v & __SIX_LOCK_HELD_write) || + !(old.v & __SIX_LOCK_HELD_intent))); + + if (old.v & l[type].lock_fail) + return false; + } while ((v = atomic64_cmpxchg_acquire(&lock->state.counter, + old.v, + old.v + l[type].lock_val)) != old.v); + + six_set_owner(lock, type, old); + return true; +} + +__always_inline __flatten +static bool __six_trylock_type(struct six_lock *lock, enum six_lock_type type) +{ + if (!do_six_trylock_type(lock, type)) + return false; + + six_acquire(&lock->dep_map, 1); + return true; +} + +__always_inline __flatten +static bool __six_relock_type(struct six_lock *lock, enum six_lock_type type, + unsigned seq) +{ + const struct six_lock_vals l[] = LOCK_VALS; + union six_lock_state old; + u64 v = READ_ONCE(lock->state.v); + + do { + old.v = v; + + if (old.seq != seq || old.v & l[type].lock_fail) + return false; + } while ((v = atomic64_cmpxchg_acquire(&lock->state.counter, + old.v, + old.v + l[type].lock_val)) != old.v); + + six_set_owner(lock, type, old); + six_acquire(&lock->dep_map, 1); + return true; +} + +struct six_lock_waiter { + struct list_head list; + struct task_struct *task; +}; + +/* This is probably up there with the more evil things I've done */ +#define waitlist_bitnr(id) ilog2((((union six_lock_state) { .waiters = 1 << (id) }).l)) + +#ifdef CONFIG_LOCK_SPIN_ON_OWNER + +static inline int six_can_spin_on_owner(struct six_lock *lock) +{ + struct task_struct *owner; + int retval = 1; + + if (need_resched()) + return 0; + + rcu_read_lock(); + owner = READ_ONCE(lock->owner); + if (owner) + retval = owner->on_cpu; + rcu_read_unlock(); + /* + * if lock->owner is not set, the mutex owner may have just acquired + * it and not set the owner yet or the mutex has been released. + */ + return retval; +} + +static inline bool six_spin_on_owner(struct six_lock *lock, + struct task_struct *owner) +{ + bool ret = true; + + rcu_read_lock(); + while (lock->owner == owner) { + /* + * Ensure we emit the owner->on_cpu, dereference _after_ + * checking lock->owner still matches owner. If that fails, + * owner might point to freed memory. If it still matches, + * the rcu_read_lock() ensures the memory stays valid. + */ + barrier(); + + if (!owner->on_cpu || need_resched()) { + ret = false; + break; + } + + cpu_relax(); + } + rcu_read_unlock(); + + return ret; +} + +static inline bool six_optimistic_spin(struct six_lock *lock, enum six_lock_type type) +{ + struct task_struct *task = current; + + if (type == SIX_LOCK_write) + return false; + + preempt_disable(); + if (!six_can_spin_on_owner(lock)) + goto fail; + + if (!osq_lock(&lock->osq)) + goto fail; + + while (1) { + struct task_struct *owner; + + /* + * If there's an owner, wait for it to either + * release the lock or go to sleep. + */ + owner = READ_ONCE(lock->owner); + if (owner && !six_spin_on_owner(lock, owner)) + break; + + if (do_six_trylock_type(lock, type)) { + osq_unlock(&lock->osq); + preempt_enable(); + return true; + } + + /* + * When there's no owner, we might have preempted between the + * owner acquiring the lock and setting the owner field. If + * we're an RT task that will live-lock because we won't let + * the owner complete. + */ + if (!owner && (need_resched() || rt_task(task))) + break; + + /* + * The cpu_relax() call is a compiler barrier which forces + * everything in this loop to be re-loaded. We don't need + * memory barriers as we'll eventually observe the right + * values at the cost of a few extra spins. + */ + cpu_relax(); + } + + osq_unlock(&lock->osq); +fail: + preempt_enable(); + + /* + * If we fell out of the spin path because of need_resched(), + * reschedule now, before we try-lock again. This avoids getting + * scheduled out right after we obtained the lock. + */ + if (need_resched()) + schedule(); + + return false; +} + +#else /* CONFIG_LOCK_SPIN_ON_OWNER */ + +static inline bool six_optimistic_spin(struct six_lock *lock, enum six_lock_type type) +{ + return false; +} + +#endif + +noinline +static void __six_lock_type_slowpath(struct six_lock *lock, enum six_lock_type type) +{ + const struct six_lock_vals l[] = LOCK_VALS; + union six_lock_state old, new; + struct six_lock_waiter wait; + u64 v; + + if (six_optimistic_spin(lock, type)) + return; + + lock_contended(&lock->dep_map, _RET_IP_); + + INIT_LIST_HEAD(&wait.list); + wait.task = current; + + while (1) { + set_current_state(TASK_UNINTERRUPTIBLE); + if (type == SIX_LOCK_write) + EBUG_ON(lock->owner != current); + else if (list_empty_careful(&wait.list)) { + raw_spin_lock(&lock->wait_lock); + list_add_tail(&wait.list, &lock->wait_list[type]); + raw_spin_unlock(&lock->wait_lock); + } + + v = READ_ONCE(lock->state.v); + do { + new.v = old.v = v; + + if (!(old.v & l[type].lock_fail)) + new.v += l[type].lock_val; + else if (!(new.waiters & (1 << type))) + new.waiters |= 1 << type; + else + break; /* waiting bit already set */ + } while ((v = atomic64_cmpxchg_acquire(&lock->state.counter, + old.v, new.v)) != old.v); + + if (!(old.v & l[type].lock_fail)) + break; + + schedule(); + } + + six_set_owner(lock, type, old); + + __set_current_state(TASK_RUNNING); + + if (!list_empty_careful(&wait.list)) { + raw_spin_lock(&lock->wait_lock); + list_del_init(&wait.list); + raw_spin_unlock(&lock->wait_lock); + } +} + +__always_inline +static void __six_lock_type(struct six_lock *lock, enum six_lock_type type) +{ + six_acquire(&lock->dep_map, 0); + + if (!do_six_trylock_type(lock, type)) + __six_lock_type_slowpath(lock, type); + + lock_acquired(&lock->dep_map, _RET_IP_); +} + +static inline void six_lock_wakeup(struct six_lock *lock, + union six_lock_state state, + unsigned waitlist_id) +{ + struct list_head *wait_list = &lock->wait_list[waitlist_id]; + struct six_lock_waiter *w, *next; + + if (waitlist_id == SIX_LOCK_write && state.read_lock) + return; + + if (!(state.waiters & (1 << waitlist_id))) + return; + + clear_bit(waitlist_bitnr(waitlist_id), + (unsigned long *) &lock->state.v); + + if (waitlist_id == SIX_LOCK_write) { + struct task_struct *p = READ_ONCE(lock->owner); + + if (p) + wake_up_process(p); + return; + } + + raw_spin_lock(&lock->wait_lock); + + list_for_each_entry_safe(w, next, wait_list, list) { + list_del_init(&w->list); + + if (wake_up_process(w->task) && + waitlist_id != SIX_LOCK_read) { + if (!list_empty(wait_list)) + set_bit(waitlist_bitnr(waitlist_id), + (unsigned long *) &lock->state.v); + break; + } + } + + raw_spin_unlock(&lock->wait_lock); +} + +__always_inline __flatten +static void __six_unlock_type(struct six_lock *lock, enum six_lock_type type) +{ + const struct six_lock_vals l[] = LOCK_VALS; + union six_lock_state state; + + EBUG_ON(!(lock->state.v & l[type].held_mask)); + EBUG_ON(type == SIX_LOCK_write && + !(lock->state.v & __SIX_LOCK_HELD_intent)); + + six_release(&lock->dep_map); + + if (type == SIX_LOCK_intent) { + EBUG_ON(lock->owner != current); + + if (lock->intent_lock_recurse) { + --lock->intent_lock_recurse; + return; + } + + lock->owner = NULL; + } + + state.v = atomic64_add_return_release(l[type].unlock_val, + &lock->state.counter); + six_lock_wakeup(lock, state, l[type].unlock_wakeup); +} + +#define __SIX_LOCK(type) \ +bool six_trylock_##type(struct six_lock *lock) \ +{ \ + return __six_trylock_type(lock, SIX_LOCK_##type); \ +} \ +EXPORT_SYMBOL_GPL(six_trylock_##type); \ + \ +bool six_relock_##type(struct six_lock *lock, u32 seq) \ +{ \ + return __six_relock_type(lock, SIX_LOCK_##type, seq); \ +} \ +EXPORT_SYMBOL_GPL(six_relock_##type); \ + \ +void six_lock_##type(struct six_lock *lock) \ +{ \ + __six_lock_type(lock, SIX_LOCK_##type); \ +} \ +EXPORT_SYMBOL_GPL(six_lock_##type); \ + \ +void six_unlock_##type(struct six_lock *lock) \ +{ \ + __six_unlock_type(lock, SIX_LOCK_##type); \ +} \ +EXPORT_SYMBOL_GPL(six_unlock_##type); + +__SIX_LOCK(read) +__SIX_LOCK(intent) +__SIX_LOCK(write) + +#undef __SIX_LOCK + +/* Convert from intent to read: */ +void six_lock_downgrade(struct six_lock *lock) +{ + six_lock_increment(lock, SIX_LOCK_read); + six_unlock_intent(lock); +} +EXPORT_SYMBOL_GPL(six_lock_downgrade); + +bool six_lock_tryupgrade(struct six_lock *lock) +{ + const struct six_lock_vals l[] = LOCK_VALS; + union six_lock_state old, new; + u64 v = READ_ONCE(lock->state.v); + + do { + new.v = old.v = v; + + EBUG_ON(!(old.v & l[SIX_LOCK_read].held_mask)); + + new.v += l[SIX_LOCK_read].unlock_val; + + if (new.v & l[SIX_LOCK_intent].lock_fail) + return false; + + new.v += l[SIX_LOCK_intent].lock_val; + } while ((v = atomic64_cmpxchg_acquire(&lock->state.counter, + old.v, new.v)) != old.v); + + six_set_owner(lock, SIX_LOCK_intent, old); + six_lock_wakeup(lock, new, l[SIX_LOCK_read].unlock_wakeup); + + return true; +} +EXPORT_SYMBOL_GPL(six_lock_tryupgrade); + +bool six_trylock_convert(struct six_lock *lock, + enum six_lock_type from, + enum six_lock_type to) +{ + EBUG_ON(to == SIX_LOCK_write || from == SIX_LOCK_write); + + if (to == from) + return true; + + if (to == SIX_LOCK_read) { + six_lock_downgrade(lock); + return true; + } else { + return six_lock_tryupgrade(lock); + } +} +EXPORT_SYMBOL_GPL(six_trylock_convert); + +/* + * Increment read/intent lock count, assuming we already have it read or intent + * locked: + */ +void six_lock_increment(struct six_lock *lock, enum six_lock_type type) +{ + const struct six_lock_vals l[] = LOCK_VALS; + + EBUG_ON(type == SIX_LOCK_write); + six_acquire(&lock->dep_map, 0); + + /* XXX: assert already locked, and that we don't overflow: */ + + switch (type) { + case SIX_LOCK_read: + atomic64_add(l[type].lock_val, &lock->state.counter); + break; + case SIX_LOCK_intent: + lock->intent_lock_recurse++; + break; + case SIX_LOCK_write: + BUG(); + break; + } +} +EXPORT_SYMBOL_GPL(six_lock_increment); From patchwork Mon Jun 10 19:14:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 10985317 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4804913AD for ; Mon, 10 Jun 2019 19:14:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 368D928429 for ; Mon, 10 Jun 2019 19:14:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2906D283C3; Mon, 10 Jun 2019 19:14:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C3CEE283C3 for ; Mon, 10 Jun 2019 19:14:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389176AbfFJTOe (ORCPT ); Mon, 10 Jun 2019 15:14:34 -0400 Received: from mail-vs1-f68.google.com ([209.85.217.68]:45023 "EHLO mail-vs1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389167AbfFJTOd (ORCPT ); Mon, 10 Jun 2019 15:14:33 -0400 Received: by mail-vs1-f68.google.com with SMTP id v129so6191889vsb.11; Mon, 10 Jun 2019 12:14:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qVPQA1G03TZXmMhT1ReNlX8Dqt4hVw5sMuV447dMS9w=; b=BTH0LYCBiNfV2iYfhZtj+8EvTN2VZXZ9ofvyVlCb/xe8hZ60b75zs24bbUQx3dlxgo ldFA0kuEVfLr/SgpdkcJWGWH4zkEaJy1y0wB1lJECLqgrzO6t/5fJiMHzulB8cJBxJ0n O2fc534HbAhSmMLHS9/Uik8Uily6fOY9e+hP+Qhe9dqH96Ap80897eB67shJDQF8AKCQ Qav+fWUnKs92o6G9DoqJFDdOPy5pZ2DANGHnKad+r5+TH418hLTZ2WzsAm56IosibY+S OD+ttL58RuVwpTxNn5e6JxlvmRjGWsX7hKyGnDFQZaXTjPkFHjCeDImHeW5aLYf3K/wo lbPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qVPQA1G03TZXmMhT1ReNlX8Dqt4hVw5sMuV447dMS9w=; b=Hmn+zY84J9SmWJIwf3gCle2mbsxud0Bpzui7QmKaItE217UrTnpb4NY89g9Nbm9U+b zZR48m6d00xPUeEDgxf7XMgHuxpdp8LlNTWN2WcSxaY4rMZ5EMJpM/+DGaNQkMzf4ekn 0J2Y6IgYDUc+/HJimwOBTDR4X1MpZ9PyIHi/VQCz49jymi/jMV51OJI+m5zIUDEazRis PJPB6iuE7y/B2wmWiuU8vmxjuxnsCCITd7PUQkH2cV50Q8VSAvyIl3ejTCvQ69wJAPDE Cl3pP7fBFnzMA5j3pC1AjAv/hyi++W22ifiXAXxB+LKJ469aYBuZHQVD5fnNFRubIdNh 6reg== X-Gm-Message-State: APjAAAWqP1Z6zYsPCpuRAr4PrTOPX/TBZ4tKT5MwJPUP6otfTGA+pJUl QT2d9TH5I7AvpAn1gpavNEWhwix3fw== X-Google-Smtp-Source: APXvYqzUnGf9oPJeyunuLj6dNdswq27mhvEh0TasLZVBTzt24Ki+McvPeFyUsUWw5Rr9p3dYjhJepQ== X-Received: by 2002:a67:7cd0:: with SMTP id x199mr28882627vsc.233.1560194071634; Mon, 10 Jun 2019 12:14:31 -0700 (PDT) Received: from kmo-pixel.hsd1.vt.comcast.net (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id t20sm4834014vkd.53.2019.06.10.12.14.30 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 10 Jun 2019 12:14:30 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 03/12] mm: pagecache add lock Date: Mon, 10 Jun 2019 15:14:11 -0400 Message-Id: <20190610191420.27007-4-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190610191420.27007-1-kent.overstreet@gmail.com> References: <20190610191420.27007-1-kent.overstreet@gmail.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add a per address space lock around adding pages to the pagecache - making it possible for fallocate INSERT_RANGE/COLLAPSE_RANGE to work correctly, and also hopefully making truncate and dio a bit saner. Signed-off-by: Kent Overstreet --- fs/inode.c | 1 + include/linux/fs.h | 24 +++++++++++++ include/linux/sched.h | 4 +++ init/init_task.c | 1 + mm/filemap.c | 81 +++++++++++++++++++++++++++++++++++++++++-- 5 files changed, 108 insertions(+), 3 deletions(-) diff --git a/fs/inode.c b/fs/inode.c index 9a453f3637..8881dc551f 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -350,6 +350,7 @@ EXPORT_SYMBOL(inc_nlink); static void __address_space_init_once(struct address_space *mapping) { xa_init_flags(&mapping->i_pages, XA_FLAGS_LOCK_IRQ); + pagecache_lock_init(&mapping->add_lock); init_rwsem(&mapping->i_mmap_rwsem); INIT_LIST_HEAD(&mapping->private_list); spin_lock_init(&mapping->private_lock); diff --git a/include/linux/fs.h b/include/linux/fs.h index dd28e76790..a88d994751 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -418,6 +418,28 @@ int pagecache_write_end(struct file *, struct address_space *mapping, loff_t pos, unsigned len, unsigned copied, struct page *page, void *fsdata); +/* + * Two-state lock - can be taken for add or block - both states are shared, + * like read side of rwsem, but conflict with other state: + */ +struct pagecache_lock { + atomic_long_t v; + wait_queue_head_t wait; +}; + +static inline void pagecache_lock_init(struct pagecache_lock *lock) +{ + atomic_long_set(&lock->v, 0); + init_waitqueue_head(&lock->wait); +} + +void pagecache_add_put(struct pagecache_lock *); +void pagecache_add_get(struct pagecache_lock *); +void __pagecache_block_put(struct pagecache_lock *); +void __pagecache_block_get(struct pagecache_lock *); +void pagecache_block_put(struct pagecache_lock *); +void pagecache_block_get(struct pagecache_lock *); + /** * struct address_space - Contents of a cacheable, mappable object. * @host: Owner, either the inode or the block_device. @@ -452,6 +474,8 @@ struct address_space { spinlock_t private_lock; struct list_head private_list; void *private_data; + struct pagecache_lock add_lock + ____cacheline_aligned_in_smp; /* protects adding new pages */ } __attribute__((aligned(sizeof(long)))) __randomize_layout; /* * On most architectures that alignment is already the case; but diff --git a/include/linux/sched.h b/include/linux/sched.h index 1549584a15..a46baade99 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -43,6 +43,7 @@ struct io_context; struct mempolicy; struct nameidata; struct nsproxy; +struct pagecache_lock; struct perf_event_context; struct pid_namespace; struct pipe_inode_info; @@ -935,6 +936,9 @@ struct task_struct { unsigned int in_ubsan; #endif + /* currently held lock, for avoiding recursing in fault path: */ + struct pagecache_lock *pagecache_lock; + /* Journalling filesystem info: */ void *journal_info; diff --git a/init/init_task.c b/init/init_task.c index c70ef656d0..92bbb6e909 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -115,6 +115,7 @@ struct task_struct init_task }, .blocked = {{0}}, .alloc_lock = __SPIN_LOCK_UNLOCKED(init_task.alloc_lock), + .pagecache_lock = NULL, .journal_info = NULL, INIT_CPU_TIMERS(init_task) .pi_lock = __RAW_SPIN_LOCK_UNLOCKED(init_task.pi_lock), diff --git a/mm/filemap.c b/mm/filemap.c index d78f577bae..93d7e0e686 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -113,6 +113,73 @@ * ->tasklist_lock (memory_failure, collect_procs_ao) */ +static void __pagecache_lock_put(struct pagecache_lock *lock, long i) +{ + BUG_ON(atomic_long_read(&lock->v) == 0); + + if (atomic_long_sub_return_release(i, &lock->v) == 0) + wake_up_all(&lock->wait); +} + +static bool __pagecache_lock_tryget(struct pagecache_lock *lock, long i) +{ + long v = atomic_long_read(&lock->v), old; + + do { + old = v; + + if (i > 0 ? v < 0 : v > 0) + return false; + } while ((v = atomic_long_cmpxchg_acquire(&lock->v, + old, old + i)) != old); + return true; +} + +static void __pagecache_lock_get(struct pagecache_lock *lock, long i) +{ + wait_event(lock->wait, __pagecache_lock_tryget(lock, i)); +} + +void pagecache_add_put(struct pagecache_lock *lock) +{ + __pagecache_lock_put(lock, 1); +} +EXPORT_SYMBOL(pagecache_add_put); + +void pagecache_add_get(struct pagecache_lock *lock) +{ + __pagecache_lock_get(lock, 1); +} +EXPORT_SYMBOL(pagecache_add_get); + +void __pagecache_block_put(struct pagecache_lock *lock) +{ + __pagecache_lock_put(lock, -1); +} +EXPORT_SYMBOL(__pagecache_block_put); + +void __pagecache_block_get(struct pagecache_lock *lock) +{ + __pagecache_lock_get(lock, -1); +} +EXPORT_SYMBOL(__pagecache_block_get); + +void pagecache_block_put(struct pagecache_lock *lock) +{ + BUG_ON(current->pagecache_lock != lock); + current->pagecache_lock = NULL; + __pagecache_lock_put(lock, -1); +} +EXPORT_SYMBOL(pagecache_block_put); + +void pagecache_block_get(struct pagecache_lock *lock) +{ + __pagecache_lock_get(lock, -1); + BUG_ON(current->pagecache_lock); + current->pagecache_lock = lock; +} +EXPORT_SYMBOL(pagecache_block_get); + static void page_cache_delete(struct address_space *mapping, struct page *page, void *shadow) { @@ -829,11 +896,14 @@ static int __add_to_page_cache_locked(struct page *page, VM_BUG_ON_PAGE(PageSwapBacked(page), page); mapping_set_update(&xas, mapping); + if (current->pagecache_lock != &mapping->add_lock) + pagecache_add_get(&mapping->add_lock); + if (!huge) { error = mem_cgroup_try_charge(page, current->mm, gfp_mask, &memcg, false); if (error) - return error; + goto out; } get_page(page); @@ -869,14 +939,19 @@ static int __add_to_page_cache_locked(struct page *page, if (!huge) mem_cgroup_commit_charge(page, memcg, false, false); trace_mm_filemap_add_to_page_cache(page); - return 0; + error = 0; +out: + if (current->pagecache_lock != &mapping->add_lock) + pagecache_add_put(&mapping->add_lock); + return error; error: page->mapping = NULL; /* Leave page->index set: truncation relies upon it */ if (!huge) mem_cgroup_cancel_charge(page, memcg, false); put_page(page); - return xas_error(&xas); + error = xas_error(&xas); + goto out; } /** From patchwork Mon Jun 10 19:14:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 10985315 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0B8A013AF for ; Mon, 10 Jun 2019 19:14:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ED8AE28429 for ; Mon, 10 Jun 2019 19:14:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DEC752876E; Mon, 10 Jun 2019 19:14:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 38FEA28429 for ; Mon, 10 Jun 2019 19:14:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389212AbfFJTOf (ORCPT ); Mon, 10 Jun 2019 15:14:35 -0400 Received: from mail-vs1-f68.google.com ([209.85.217.68]:46015 "EHLO mail-vs1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389188AbfFJTOe (ORCPT ); Mon, 10 Jun 2019 15:14:34 -0400 Received: by mail-vs1-f68.google.com with SMTP id n21so6199403vsp.12; Mon, 10 Jun 2019 12:14:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ukvpJfRydaYPuTfjLdUYlRJkIRyXfryPz7hZQ9HXoow=; b=TumAj4TdvB74km6c9YeX0VRaQTSYfO74uvAcBv8rQ1GqyLhE5WllFwZE9P7cHvhdr3 YHdMY/ngPDORefWvUQj+mriprD70MyIfythZdNb/9rwIgWNb0vioc+XWh/L/Ln1yYtYc IA5t3nEqX/695bZCbCC15/a+pH4LrhfgWQ+QtjQtP6gU8JbiiQbviWbxnIFJW9tmlH5U cb3uTr1qzpz90tJAuvTy2QaK9w7cMlXqEFKmLX9XvzRn25SGomANMlqnAfZzazhbkX08 qh9GGIVFq7PdUkYq/Wz5we2FALX41OgavhksUmswN8k6TUtICmgX34eBAA5sJCF6M0su LU5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ukvpJfRydaYPuTfjLdUYlRJkIRyXfryPz7hZQ9HXoow=; b=ZnQB6bhTEL2wzbn7VRvv2zFPqJZC50XZPSKCn49W19CCXLf9lVM1X/mJCF5tFvoTVj LAWCJMthWmlGfw8kDX1PywMjQFW4Dr1ZFceGnJIwxJAjoMsSbSB6Tt88V2NnzjOhSR6X s2spyPPTdTeJIAEw50/WB2fZNoQXPUHAK5eQ/potG0T9HS7wPpSVAGZUPGrTGbWPBfhK EajyFHwO4G7W3HaPQC0S6mE2bVrd1w1g42CduFIf+TD15X09pql3ghjKbVILqwCN5V0m 8WEyVxIW2EG1eESxmvTg49DZ0zlf9ADatj/DQnDoHWMFmumi9pGq0Qk/Ob8C/GXmQWoH 23iA== X-Gm-Message-State: APjAAAUX9EAdr49exivOuY31wbHvvk4gCUtQwCL/zFAYLr7Zq44vDfu7 cRP/EKxyhYo4IzRhePA1PulkpY7oaw== X-Google-Smtp-Source: APXvYqxsHup84wBEkb0l4BUQGxeUm+1yBh79JR2Agcc/L3RBjwdqTTbsi3z7rNbZGTE03Pf/3whhwg== X-Received: by 2002:a67:63c2:: with SMTP id x185mr38240896vsb.166.1560194073159; Mon, 10 Jun 2019 12:14:33 -0700 (PDT) Received: from kmo-pixel.hsd1.vt.comcast.net (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id t20sm4834014vkd.53.2019.06.10.12.14.31 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 10 Jun 2019 12:14:32 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 04/12] mm: export find_get_pages() Date: Mon, 10 Jun 2019 15:14:12 -0400 Message-Id: <20190610191420.27007-5-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190610191420.27007-1-kent.overstreet@gmail.com> References: <20190610191420.27007-1-kent.overstreet@gmail.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Needed for bcachefs Signed-off-by: Kent Overstreet --- mm/filemap.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/filemap.c b/mm/filemap.c index 93d7e0e686..617168474e 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1899,6 +1899,7 @@ unsigned find_get_pages_range(struct address_space *mapping, pgoff_t *start, return ret; } +EXPORT_SYMBOL(find_get_pages_range); /** * find_get_pages_contig - gang contiguous pagecache lookup From patchwork Mon Jun 10 19:14:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 10985331 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2BF4113AD for ; Mon, 10 Jun 2019 19:15:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0F284206AF for ; Mon, 10 Jun 2019 19:15:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0097E27FE4; Mon, 10 Jun 2019 19:15:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9C4A5206AF for ; Mon, 10 Jun 2019 19:15:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389589AbfFJTP2 (ORCPT ); Mon, 10 Jun 2019 15:15:28 -0400 Received: from mail-vs1-f67.google.com ([209.85.217.67]:39098 "EHLO mail-vs1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389210AbfFJTOg (ORCPT ); Mon, 10 Jun 2019 15:14:36 -0400 Received: by mail-vs1-f67.google.com with SMTP id n2so6215954vso.6; Mon, 10 Jun 2019 12:14:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aR//uD4NhWFMVjpTld02Vmm29l8MHQS0/NqSGli70F8=; b=PGcOELLZdo4Kg9v3G6sVPoxRniXMINmFDyPMiPzN6isPtueXfeYgWgOyCx0uoSur6Y BXjfDo4sohWRnSKAPm3pnL13BdVoyTCx9qi/EqPsq03d/dFrAs4Dw0EAGhJO2s387BYa wkV4b6w1U/1tnm5eMOEcF6yk3uvrsTIBW7Bt6MF6DiKLaIrGx+8/XpDIebBQ0XvBKqA2 6UlHky1ZYmBRQDGoZ9N5kdM7QbeeK997QHERdJ2a1GzdhnDrpD/fmkI82eyHpQCrU58+ JOuVu272wM6Iye9GuJvp5FN9pnJlXOLfJUu6wCw5V/p9PXENnGZ2zaqXM5WkdrjhfesR FngA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aR//uD4NhWFMVjpTld02Vmm29l8MHQS0/NqSGli70F8=; b=qc+9CS/YgIyfnPGJWqynPr/2+8P20ILz1kbFGF7oSdFKVZ/YvN+aDXtcibXSzWqwUM MB+eo6C5JcQQgvUmCJcA4qN0n5aM4UIGR7wLH5AcVr5Fw3H/dEqAeOPDDKjwKvz7pwTb xsREjvaM5Jx2+btcH+Pd2QDxduCh+ilwOzKdz0KAQjw9giupxSJnlMD1KyHVF9H8kvuu Ssa0mZn5JlJJ9/rzUxIjQrzFQeZ/Tb5geElnSheC3TykChSPxcGbgzAT7hkxq4OhXiP/ HUFJE95HKPDvvBCxz29h1T+mNikgAPgGNNZbbRQKty5gjtT3VKEbccPx9gIoyFm6L4iY AaUQ== X-Gm-Message-State: APjAAAVtCqmFXJ/P7M9kFsLQSG5ZCFfrx+73frVe+MsNGMYTm/vQCUb+ r62gx9nap9VK/N/9PSxDuujpWdVXlQ== X-Google-Smtp-Source: APXvYqxaP8f1TN0DkPEILnhMA4UYfW7vT2pn5ycz2pBDc25trij2XZ0vvzlD2cfHHMKAgC0ktwlfvg== X-Received: by 2002:a67:f911:: with SMTP id t17mr3355111vsq.128.1560194074584; Mon, 10 Jun 2019 12:14:34 -0700 (PDT) Received: from kmo-pixel.hsd1.vt.comcast.net (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id t20sm4834014vkd.53.2019.06.10.12.14.33 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 10 Jun 2019 12:14:33 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 05/12] fs: insert_inode_locked2() Date: Mon, 10 Jun 2019 15:14:13 -0400 Message-Id: <20190610191420.27007-6-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190610191420.27007-1-kent.overstreet@gmail.com> References: <20190610191420.27007-1-kent.overstreet@gmail.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP New helper for bcachefs, so that when we race inserting an inode we can atomically grab a ref to the inode already in the inode cache. Signed-off-by: Kent Overstreet --- fs/inode.c | 40 ++++++++++++++++++++++++++++++++++++++++ include/linux/fs.h | 1 + 2 files changed, 41 insertions(+) diff --git a/fs/inode.c b/fs/inode.c index 8881dc551f..cc44f345e0 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -1479,6 +1479,46 @@ int insert_inode_locked(struct inode *inode) } EXPORT_SYMBOL(insert_inode_locked); +struct inode *insert_inode_locked2(struct inode *inode) +{ + struct super_block *sb = inode->i_sb; + ino_t ino = inode->i_ino; + struct hlist_head *head = inode_hashtable + hash(sb, ino); + + while (1) { + struct inode *old = NULL; + spin_lock(&inode_hash_lock); + hlist_for_each_entry(old, head, i_hash) { + if (old->i_ino != ino) + continue; + if (old->i_sb != sb) + continue; + spin_lock(&old->i_lock); + if (old->i_state & (I_FREEING|I_WILL_FREE)) { + spin_unlock(&old->i_lock); + continue; + } + break; + } + if (likely(!old)) { + spin_lock(&inode->i_lock); + inode->i_state |= I_NEW | I_CREATING; + hlist_add_head(&inode->i_hash, head); + spin_unlock(&inode->i_lock); + spin_unlock(&inode_hash_lock); + return NULL; + } + __iget(old); + spin_unlock(&old->i_lock); + spin_unlock(&inode_hash_lock); + wait_on_inode(old); + if (unlikely(!inode_unhashed(old))) + return old; + iput(old); + } +} +EXPORT_SYMBOL(insert_inode_locked2); + int insert_inode_locked4(struct inode *inode, unsigned long hashval, int (*test)(struct inode *, void *), void *data) { diff --git a/include/linux/fs.h b/include/linux/fs.h index a88d994751..d5d12d6981 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -3010,6 +3010,7 @@ extern struct inode *find_inode_nowait(struct super_block *, void *data); extern int insert_inode_locked4(struct inode *, unsigned long, int (*test)(struct inode *, void *), void *); extern int insert_inode_locked(struct inode *); +extern struct inode *insert_inode_locked2(struct inode *); #ifdef CONFIG_DEBUG_LOCK_ALLOC extern void lockdep_annotate_inode_mutex_key(struct inode *inode); #else From patchwork Mon Jun 10 19:14:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 10985329 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2E1F414DB for ; Mon, 10 Jun 2019 19:15:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1B8DE1FF7F for ; Mon, 10 Jun 2019 19:15:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0CD3527F7F; Mon, 10 Jun 2019 19:15:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AD14F1FF7F for ; Mon, 10 Jun 2019 19:15:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389530AbfFJTPS (ORCPT ); Mon, 10 Jun 2019 15:15:18 -0400 Received: from mail-ua1-f66.google.com ([209.85.222.66]:45774 "EHLO mail-ua1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389243AbfFJTOh (ORCPT ); Mon, 10 Jun 2019 15:14:37 -0400 Received: by mail-ua1-f66.google.com with SMTP id v18so3525024uad.12; Mon, 10 Jun 2019 12:14:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=03qXnIOs+9PMTRKALHPhtOzBUbY1LuDMI7042jLkIEM=; b=q+sYO+O25PAkVKSZpsKj1OfUzWZuizGq05GqQ//teUbVxoI0Gr+pudBTbJsrXVk0AK vp9lmmAuSKZtW2KfBHrQdTcGHhPT6OanBLFG/Fjd5/lE/G9PII8873awfmqFC8hpUhKg D2lY8xWKov4H/ovzgv+Jv2wNaGfUt7YJ8CYpYq9qiNNnHLFd3fNSRufWtnU8eWjDdDb/ fCCA0wei/JLshPe69z+yTS6HSU/mvkyKX2STQoHQVppctfDRW0SerJGmD7CdGkSc0YcH 5pFfkr/PmUXYwOdYEv0vwj5hTCy0OfvyRWzeApHLZaSmJjgBnkb23Rlz0i+dWupO6BEP JUbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=03qXnIOs+9PMTRKALHPhtOzBUbY1LuDMI7042jLkIEM=; b=jCu5XUgdOttHzL/7y+8mX5nWcKMClGE30JcXVsVO5yvd26X3cE9DR5O3y79IpQJjTm QWSSeEE3N471m1FZSDvuM8ldYa2FYwKobpAYexKZwaWVUfc4xEKMZaSCd8PkHqQEpFPZ 6g8RNdxMW17vDlqdP10G4ap82ehm4he29qvq5wdzJwZD09Rd1qf5iWOZuymnvbu3aarY 9MXtqqZKlT9Auh2hRE5xTXj4Ntlhdu4LT9xQn1glC3Vkb6LzfBMctVG+mwLU/JmMd295 G7UzXwZk5vIOsBhRbUZULMK4Vc8aByVQ793lsSG5cB2+8Kx72paneoYY2p86Z9h2vCho pcxw== X-Gm-Message-State: APjAAAU/50qk3yIzt/JoJcmQM9gj215yvl1SNGkF1bhC9KYW2cbnTymH DDJaSTcA9PHxqA02SQGwuYntaDKOHw== X-Google-Smtp-Source: APXvYqx0V5Lkfup6ZBapoJxNQzzjDcvcm9S6eEe2y+rfZAijhO/uAxre2kPXwhD/J/e4Lkqp0KgekA== X-Received: by 2002:ab0:1590:: with SMTP id i16mr9163111uae.141.1560194076068; Mon, 10 Jun 2019 12:14:36 -0700 (PDT) Received: from kmo-pixel.hsd1.vt.comcast.net (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id t20sm4834014vkd.53.2019.06.10.12.14.34 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 10 Jun 2019 12:14:35 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 06/12] fs: factor out d_mark_tmpfile() Date: Mon, 10 Jun 2019 15:14:14 -0400 Message-Id: <20190610191420.27007-7-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190610191420.27007-1-kent.overstreet@gmail.com> References: <20190610191420.27007-1-kent.overstreet@gmail.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP New helper for bcachefs - bcachefs doesn't want the inode_dec_link_count() call that d_tmpfile does, it handles i_nlink on its own atomically with other btree updates Signed-off-by: Kent Overstreet --- fs/dcache.c | 10 ++++++++-- include/linux/dcache.h | 1 + 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index aac41adf47..18edb4e5bc 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -3042,9 +3042,8 @@ void d_genocide(struct dentry *parent) EXPORT_SYMBOL(d_genocide); -void d_tmpfile(struct dentry *dentry, struct inode *inode) +void d_mark_tmpfile(struct dentry *dentry, struct inode *inode) { - inode_dec_link_count(inode); BUG_ON(dentry->d_name.name != dentry->d_iname || !hlist_unhashed(&dentry->d_u.d_alias) || !d_unlinked(dentry)); @@ -3054,6 +3053,13 @@ void d_tmpfile(struct dentry *dentry, struct inode *inode) (unsigned long long)inode->i_ino); spin_unlock(&dentry->d_lock); spin_unlock(&dentry->d_parent->d_lock); +} +EXPORT_SYMBOL(d_mark_tmpfile); + +void d_tmpfile(struct dentry *dentry, struct inode *inode) +{ + inode_dec_link_count(inode); + d_mark_tmpfile(dentry, inode); d_instantiate(dentry, inode); } EXPORT_SYMBOL(d_tmpfile); diff --git a/include/linux/dcache.h b/include/linux/dcache.h index 60996e64c5..e0fe330162 100644 --- a/include/linux/dcache.h +++ b/include/linux/dcache.h @@ -255,6 +255,7 @@ extern struct dentry * d_make_root(struct inode *); /* - the ramfs-type tree */ extern void d_genocide(struct dentry *); +extern void d_mark_tmpfile(struct dentry *, struct inode *); extern void d_tmpfile(struct dentry *, struct inode *); extern struct dentry *d_find_alias(struct inode *); From patchwork Mon Jun 10 19:14:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 10985319 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3962513AD for ; Mon, 10 Jun 2019 19:14:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 28CB3283C3 for ; Mon, 10 Jun 2019 19:14:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1D172286FE; Mon, 10 Jun 2019 19:14:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BC1FD283C3 for ; Mon, 10 Jun 2019 19:14:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389301AbfFJTOl (ORCPT ); Mon, 10 Jun 2019 15:14:41 -0400 Received: from mail-vs1-f67.google.com ([209.85.217.67]:39105 "EHLO mail-vs1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389188AbfFJTOk (ORCPT ); Mon, 10 Jun 2019 15:14:40 -0400 Received: by mail-vs1-f67.google.com with SMTP id n2so6216142vso.6; Mon, 10 Jun 2019 12:14:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tWvHhq8XxQsgdEEkuwdPbUPIjaH98CArgRPaOAub3ak=; b=UGM3lu2qIHghbQDu1EsRRuavlODvaP07lY9tARpBwIqOVzP5rUfP+otXPvOH45AbIn +q2gvV+wGnZYtNE2MViSJVHRWUI6IhueGI0T5o2nLX+gtRu0Rwf3H7Ky6Gdu7OKzQJmp TR9OY7pSkBYQeo3qJix4w/zCPD4tfmlQFHKdEpNokvXY3s8MIgQjcNyBsMxgHuA1Krv2 9+GYTMwk3XwFdKHF+LbNFHmTqw93AbXVxU972IouXyhmJ+1/c3vyUjn/PLC4IICaQzsR Brb3VMHqpYnTTyPFXbvg5yjV5/ctGZgH5oIrdlvkvZeCvWyiWF2IQBICWcH2GIbNOUA0 +45Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tWvHhq8XxQsgdEEkuwdPbUPIjaH98CArgRPaOAub3ak=; b=tYM6DCtYFl6iYpg+1je3DzC1W20V5wEzVBcam4PO2cLFZ8uRPfONyeHgM/Rj0s7X5+ 3bFhSaBTMAwLn9ERfWrMm/li+n6fQnqAWBj60P89Wod4i1vFXaR4CL/fVOqzd8ZPb1ef 9GgszwEax5P5pAbgMTX6aYynuWAe4c9o3cFyV8OER5ZWtstKCmVlIBznblySYBuDo83O y1tMDqIqD6fdYa6PSqfW9OabQG6dfwVOtP2Je/tetKAW5qY2z0uIbBAH81tD9vHEXiVu V83WEoGiq4I0avDEMNfO5+rbrc2rkzXMAoM0FHY8EixPpmEt/+pRsKQoKIsytTj6nLJv 4x2A== X-Gm-Message-State: APjAAAWuNPo9mhMmQSN59zCf7+4s9+9ECPGB9DkTmjxNbYEYjsuH7eti aSkWqWCE5sn1wSqoS+fmIcToO7Ikzg== X-Google-Smtp-Source: APXvYqyp9NIqoSDj8Nbh7rQIbCBMQwTNEntUK0tr2oYE/q/t7X0iHO7hy11ayPE3I91sFc8HgJ3JFw== X-Received: by 2002:a67:f911:: with SMTP id t17mr3355596vsq.128.1560194079062; Mon, 10 Jun 2019 12:14:39 -0700 (PDT) Received: from kmo-pixel.hsd1.vt.comcast.net (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id t20sm4834014vkd.53.2019.06.10.12.14.38 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 10 Jun 2019 12:14:38 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 08/12] block: Add some exports for bcachefs Date: Mon, 10 Jun 2019 15:14:16 -0400 Message-Id: <20190610191420.27007-9-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190610191420.27007-1-kent.overstreet@gmail.com> References: <20190610191420.27007-1-kent.overstreet@gmail.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Kent Overstreet --- block/bio.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/block/bio.c b/block/bio.c index 716510ecd7..a67aa6e0de 100644 --- a/block/bio.c +++ b/block/bio.c @@ -958,6 +958,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) return 0; } +EXPORT_SYMBOL_GPL(bio_iov_iter_get_pages); static void submit_bio_wait_endio(struct bio *bio) { @@ -1658,6 +1659,7 @@ void bio_set_pages_dirty(struct bio *bio) set_page_dirty_lock(bvec->bv_page); } } +EXPORT_SYMBOL_GPL(bio_set_pages_dirty); static void bio_release_pages(struct bio *bio) { @@ -1731,6 +1733,7 @@ void bio_check_pages_dirty(struct bio *bio) spin_unlock_irqrestore(&bio_dirty_lock, flags); schedule_work(&bio_dirty_work); } +EXPORT_SYMBOL_GPL(bio_check_pages_dirty); void update_io_ticks(struct hd_struct *part, unsigned long now) { From patchwork Mon Jun 10 19:14:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 10985327 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6D89D13AF for ; Mon, 10 Jun 2019 19:15:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5BF8D283C3 for ; Mon, 10 Jun 2019 19:15:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 508D3286FE; Mon, 10 Jun 2019 19:15:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 014EF283C3 for ; Mon, 10 Jun 2019 19:15:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389320AbfFJTOm (ORCPT ); Mon, 10 Jun 2019 15:14:42 -0400 Received: from mail-ua1-f65.google.com ([209.85.222.65]:41869 "EHLO mail-ua1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389290AbfFJTOl (ORCPT ); Mon, 10 Jun 2019 15:14:41 -0400 Received: by mail-ua1-f65.google.com with SMTP id n2so3525329uad.8; Mon, 10 Jun 2019 12:14:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3U3sA7JK4Wx4xJS71wfGXZ4KSvEZtwk1XyQ4suCC7B8=; b=BoFQLMwXN/UZRVJyQluQO+IOXl42mInR39LPWD30FsUVUbwEcOlNCwBRSs675vWfTp nDqqZC1kbJ5x4khJxDiawgPzJ5IcAGNh1tPH2eugTRlOKxMFBm3LjVwZKYadSI/l/qWa 0YP0ZfnQClFEIDjaqch6oLsjPXAWHQx3LovF86tLYwakeGuSfkV6h3h1DRFLr2qfNIWr HQigF/Ymtnn8lCCeTJXeLY/TAhQxJJ3VOPEallbf+zk3hBFBgj9N38eBPUVnb4C1sgkK APKy17dJTO2PC8mMUr8vpM34yPNj8ugLwG9vVxthcoEXlLID1am+Z6isAI3lhI6/lmFh pb1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3U3sA7JK4Wx4xJS71wfGXZ4KSvEZtwk1XyQ4suCC7B8=; b=cyXAi90wfIVRqHjQ/iVqGtjI8uY/cpGGUOGxsYC+EtVeI8qiU9IYoE3x1FLF7D7c4v ZBs+OAMejVeM4y6uLAlLqTEzeGRLdhA+shsC7Xbo7CCM5+sWNs2yMMmGh8E1iFLqzQpq T7CWpUsrt5FKw47K/xLvZXBmqKHsQOmPn0WFzm7WSb+WIB1O60LOPX4Kp40cg7aGpCph gu9jgNk2pIjmdtN2AsRNrn6DvIB4/X+p3ypYgxRGDsw/FzBFjBnTlWTWeffpKSOaQ98z 83PoXr7jGX9EbUFPnQYHDePfLMG8VGTh//dWpmNbidtpxNUX7U2xImwdBRPZUatKTZqa R18w== X-Gm-Message-State: APjAAAX995cWUtUnIWsmA4/b345sq7AXZwlLzl8MMM6HslJxoYKrvzXM tAZ7N9HBQIEetlfpFVvKf0/KBaj4rA== X-Google-Smtp-Source: APXvYqz+UI+IwkgjNqvo1opaGdx3Xa36p3YJ2cGddzfl+1CV+RI8VIKQc3zO5YmWny0ZbpULl+VSpQ== X-Received: by 2002:ab0:2c09:: with SMTP id l9mr5860205uar.6.1560194080267; Mon, 10 Jun 2019 12:14:40 -0700 (PDT) Received: from kmo-pixel.hsd1.vt.comcast.net (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id t20sm4834014vkd.53.2019.06.10.12.14.39 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 10 Jun 2019 12:14:39 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 09/12] bcache: optimize continue_at_nobarrier() Date: Mon, 10 Jun 2019 15:14:17 -0400 Message-Id: <20190610191420.27007-10-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190610191420.27007-1-kent.overstreet@gmail.com> References: <20190610191420.27007-1-kent.overstreet@gmail.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Kent Overstreet --- drivers/md/bcache/closure.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/drivers/md/bcache/closure.h b/drivers/md/bcache/closure.h index c88cdc4ae4..376c5e659c 100644 --- a/drivers/md/bcache/closure.h +++ b/drivers/md/bcache/closure.h @@ -245,7 +245,7 @@ static inline void closure_queue(struct closure *cl) != offsetof(struct work_struct, func)); if (wq) { INIT_WORK(&cl->work, cl->work.func); - BUG_ON(!queue_work(wq, &cl->work)); + queue_work(wq, &cl->work); } else cl->fn(cl); } @@ -340,8 +340,13 @@ do { \ */ #define continue_at_nobarrier(_cl, _fn, _wq) \ do { \ - set_closure_fn(_cl, _fn, _wq); \ - closure_queue(_cl); \ + closure_set_ip(_cl); \ + if (_wq) { \ + INIT_WORK(&(_cl)->work, (void *) _fn); \ + queue_work((_wq), &(_cl)->work); \ + } else { \ + (_fn)(_cl); \ + } \ } while (0) /** From patchwork Mon Jun 10 19:14:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 10985325 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6ECC413AD for ; Mon, 10 Jun 2019 19:15:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5CD2F283C3 for ; Mon, 10 Jun 2019 19:15:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5115B286FE; Mon, 10 Jun 2019 19:15:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 81479283C3 for ; Mon, 10 Jun 2019 19:15:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389492AbfFJTPG (ORCPT ); Mon, 10 Jun 2019 15:15:06 -0400 Received: from mail-ua1-f68.google.com ([209.85.222.68]:36362 "EHLO mail-ua1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389332AbfFJTOn (ORCPT ); Mon, 10 Jun 2019 15:14:43 -0400 Received: by mail-ua1-f68.google.com with SMTP id 94so3530246uam.3; Mon, 10 Jun 2019 12:14:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=txhvfUmx1NdKNULzWGRGEFnmNXgK3e8wqkbEtNoWTMc=; b=V6EcEqJtpqy371tJvh/wPOwK5W2ArSQ3IqmU9jsv9ThYsmRZchJ3fN6ZV5etD/hc1Y OwGJn2LmzUkdDK+pSFR3djL98iNsFYzPRmFUA5L8bnQOke3cCP++uU9s7rxp4qFzMUPV fbvdViaBXnQITGDktAD/OOjJZ0l3ScSlsrNt1FVzlBNoIY2DhmV7D4Uft6JqQcrazB+e GBg99lqBY5lSxm2EtbFAlPatXaHWYUimD1ehWkFrr6Apw7D8GQutKgPeR3kpN1DQeGNN K2sn0Lw78OkPpXqCb1zze+wyFj/iB7PtDQDt0Ky+NTGOt+3ev5/cudkk49xfOSNqPyBr DqZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=txhvfUmx1NdKNULzWGRGEFnmNXgK3e8wqkbEtNoWTMc=; b=Hd4fUM87amgU57/r2uUqYMi0iiKIfgVw1jVes+kj0zuY7h/NtADtIyr163q5AwfYzs XVva52fQjbsXk7slJzu8YTcAe7ouP/vp1TVhllPMDhC9lW4dJpJxM7M5/b8Ei231L+DX 6YtVhXzhMBh1tGjsuKx+qEzbRbvETlPNs8cE1IntLifrGwqaZhtrKgzuCTzsURJpGcd/ nf6ZzG9368h4lcANPD9il9E9R+3lFYUDnF+GFP80Y23wmSVD/zC5SrHU3IPbbEUEGcHi kKM0YYjHnfTwZ59NmxImV+MdsaygP4UeOM2WnIaNIlyCt0QqkHzVHoIeoH6iq8kbtKSa WcQg== X-Gm-Message-State: APjAAAVZyzUdVh4et7tb9srtY8GwaOzs8fIIZDI5X0zLsqt7DBo6cTKf Q9ST0leU2qiPpHgHs9w+5DY20j2zKg== X-Google-Smtp-Source: APXvYqylvfZ2f3SkuXYiaiG4oxbaoTGDsuGyK1Ash3Bs0XFgObtOIBB/DLf/WlpBS3yKqmjOqGyfkw== X-Received: by 2002:ab0:a1:: with SMTP id 30mr15306587uaj.29.1560194081899; Mon, 10 Jun 2019 12:14:41 -0700 (PDT) Received: from kmo-pixel.hsd1.vt.comcast.net (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id t20sm4834014vkd.53.2019.06.10.12.14.40 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 10 Jun 2019 12:14:40 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 10/12] bcache: move closures to lib/ Date: Mon, 10 Jun 2019 15:14:18 -0400 Message-Id: <20190610191420.27007-11-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190610191420.27007-1-kent.overstreet@gmail.com> References: <20190610191420.27007-1-kent.overstreet@gmail.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Prep work for bcachefs - being a fork of bcache it also uses closures Signed-off-by: Kent Overstreet Acked-by: Coly Li --- drivers/md/bcache/Kconfig | 10 +------ drivers/md/bcache/Makefile | 6 ++-- drivers/md/bcache/bcache.h | 2 +- drivers/md/bcache/super.c | 1 - drivers/md/bcache/util.h | 3 +- .../md/bcache => include/linux}/closure.h | 17 ++++++----- lib/Kconfig | 3 ++ lib/Kconfig.debug | 9 ++++++ lib/Makefile | 2 ++ {drivers/md/bcache => lib}/closure.c | 28 ++++++------------- 10 files changed, 37 insertions(+), 44 deletions(-) rename {drivers/md/bcache => include/linux}/closure.h (97%) rename {drivers/md/bcache => lib}/closure.c (89%) diff --git a/drivers/md/bcache/Kconfig b/drivers/md/bcache/Kconfig index f6e0a8b3a6..3dd1d48987 100644 --- a/drivers/md/bcache/Kconfig +++ b/drivers/md/bcache/Kconfig @@ -2,6 +2,7 @@ config BCACHE tristate "Block device as cache" select CRC64 + select CLOSURES help Allows a block device to be used as cache for other devices; uses a btree for indexing and the layout is optimized for SSDs. @@ -16,12 +17,3 @@ config BCACHE_DEBUG Enables extra debugging tools, allows expensive runtime checks to be turned on. - -config BCACHE_CLOSURES_DEBUG - bool "Debug closures" - depends on BCACHE - select DEBUG_FS - help - Keeps all active closures in a linked list and provides a debugfs - interface to list them, which makes it possible to see asynchronous - operations that get stuck. diff --git a/drivers/md/bcache/Makefile b/drivers/md/bcache/Makefile index d26b351958..2b790fb813 100644 --- a/drivers/md/bcache/Makefile +++ b/drivers/md/bcache/Makefile @@ -2,8 +2,8 @@ obj-$(CONFIG_BCACHE) += bcache.o -bcache-y := alloc.o bset.o btree.o closure.o debug.o extents.o\ - io.o journal.o movinggc.o request.o stats.o super.o sysfs.o trace.o\ - util.o writeback.o +bcache-y := alloc.o bset.o btree.o debug.o extents.o io.o\ + journal.o movinggc.o request.o stats.o super.o sysfs.o trace.o util.o\ + writeback.o CFLAGS_request.o += -Iblock diff --git a/drivers/md/bcache/bcache.h b/drivers/md/bcache/bcache.h index fdf75352e1..ced9f1526c 100644 --- a/drivers/md/bcache/bcache.h +++ b/drivers/md/bcache/bcache.h @@ -180,6 +180,7 @@ #include #include +#include #include #include #include @@ -192,7 +193,6 @@ #include "bset.h" #include "util.h" -#include "closure.h" struct bucket { atomic_t pin; diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c index a697a3a923..da6803f280 100644 --- a/drivers/md/bcache/super.c +++ b/drivers/md/bcache/super.c @@ -2487,7 +2487,6 @@ static int __init bcache_init(void) goto err; bch_debug_init(); - closure_debug_init(); return 0; err: diff --git a/drivers/md/bcache/util.h b/drivers/md/bcache/util.h index 00aab6abcf..8a75100c0b 100644 --- a/drivers/md/bcache/util.h +++ b/drivers/md/bcache/util.h @@ -4,6 +4,7 @@ #define _BCACHE_UTIL_H #include +#include #include #include #include @@ -13,8 +14,6 @@ #include #include -#include "closure.h" - #define PAGE_SECTORS (PAGE_SIZE / 512) struct closure; diff --git a/drivers/md/bcache/closure.h b/include/linux/closure.h similarity index 97% rename from drivers/md/bcache/closure.h rename to include/linux/closure.h index 376c5e659c..308e38028c 100644 --- a/drivers/md/bcache/closure.h +++ b/include/linux/closure.h @@ -155,7 +155,7 @@ struct closure { atomic_t remaining; -#ifdef CONFIG_BCACHE_CLOSURES_DEBUG +#ifdef CONFIG_DEBUG_CLOSURES #define CLOSURE_MAGIC_DEAD 0xc054dead #define CLOSURE_MAGIC_ALIVE 0xc054a11e @@ -184,15 +184,13 @@ static inline void closure_sync(struct closure *cl) __closure_sync(cl); } -#ifdef CONFIG_BCACHE_CLOSURES_DEBUG +#ifdef CONFIG_DEBUG_CLOSURES -void closure_debug_init(void); void closure_debug_create(struct closure *cl); void closure_debug_destroy(struct closure *cl); #else -static inline void closure_debug_init(void) {} static inline void closure_debug_create(struct closure *cl) {} static inline void closure_debug_destroy(struct closure *cl) {} @@ -200,21 +198,21 @@ static inline void closure_debug_destroy(struct closure *cl) {} static inline void closure_set_ip(struct closure *cl) { -#ifdef CONFIG_BCACHE_CLOSURES_DEBUG +#ifdef CONFIG_DEBUG_CLOSURES cl->ip = _THIS_IP_; #endif } static inline void closure_set_ret_ip(struct closure *cl) { -#ifdef CONFIG_BCACHE_CLOSURES_DEBUG +#ifdef CONFIG_DEBUG_CLOSURES cl->ip = _RET_IP_; #endif } static inline void closure_set_waiting(struct closure *cl, unsigned long f) { -#ifdef CONFIG_BCACHE_CLOSURES_DEBUG +#ifdef CONFIG_DEBUG_CLOSURES cl->waiting_on = f; #endif } @@ -243,6 +241,7 @@ static inline void closure_queue(struct closure *cl) */ BUILD_BUG_ON(offsetof(struct closure, fn) != offsetof(struct work_struct, func)); + if (wq) { INIT_WORK(&cl->work, cl->work.func); queue_work(wq, &cl->work); @@ -255,7 +254,7 @@ static inline void closure_queue(struct closure *cl) */ static inline void closure_get(struct closure *cl) { -#ifdef CONFIG_BCACHE_CLOSURES_DEBUG +#ifdef CONFIG_DEBUG_CLOSURES BUG_ON((atomic_inc_return(&cl->remaining) & CLOSURE_REMAINING_MASK) <= 1); #else @@ -271,7 +270,7 @@ static inline void closure_get(struct closure *cl) */ static inline void closure_init(struct closure *cl, struct closure *parent) { - memset(cl, 0, sizeof(struct closure)); + cl->fn = NULL; cl->parent = parent; if (parent) closure_get(parent); diff --git a/lib/Kconfig b/lib/Kconfig index a9e56539bd..09a25af0d0 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -427,6 +427,9 @@ config ASSOCIATIVE_ARRAY for more information. +config CLOSURES + bool + config HAS_IOMEM bool depends on !NO_IOMEM diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index d5a4a4036d..6d97985e7e 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -1397,6 +1397,15 @@ config DEBUG_CREDENTIALS source "kernel/rcu/Kconfig.debug" +config DEBUG_CLOSURES + bool "Debug closures (bcache async widgits)" + depends on CLOSURES + select DEBUG_FS + help + Keeps all active closures in a linked list and provides a debugfs + interface to list them, which makes it possible to see asynchronous + operations that get stuck. + config DEBUG_WQ_FORCE_RR_CPU bool "Force round-robin CPU selection for unbound work items" depends on DEBUG_KERNEL diff --git a/lib/Makefile b/lib/Makefile index 18c2be516a..2003eda127 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -193,6 +193,8 @@ obj-$(CONFIG_ATOMIC64_SELFTEST) += atomic64_test.o obj-$(CONFIG_CPU_RMAP) += cpu_rmap.o +obj-$(CONFIG_CLOSURES) += closure.o + obj-$(CONFIG_CORDIC) += cordic.o obj-$(CONFIG_DQL) += dynamic_queue_limits.o diff --git a/drivers/md/bcache/closure.c b/lib/closure.c similarity index 89% rename from drivers/md/bcache/closure.c rename to lib/closure.c index 73f5319295..46cfe4c382 100644 --- a/drivers/md/bcache/closure.c +++ b/lib/closure.c @@ -6,13 +6,12 @@ * Copyright 2012 Google, Inc. */ +#include #include -#include +#include #include #include -#include "closure.h" - static inline void closure_put_after_sub(struct closure *cl, int flags) { int r = flags & CLOSURE_REMAINING_MASK; @@ -127,7 +126,7 @@ void __sched __closure_sync(struct closure *cl) } EXPORT_SYMBOL(__closure_sync); -#ifdef CONFIG_BCACHE_CLOSURES_DEBUG +#ifdef CONFIG_DEBUG_CLOSURES static LIST_HEAD(closure_list); static DEFINE_SPINLOCK(closure_list_lock); @@ -158,8 +157,6 @@ void closure_debug_destroy(struct closure *cl) } EXPORT_SYMBOL(closure_debug_destroy); -static struct dentry *closure_debug; - static int debug_seq_show(struct seq_file *f, void *data) { struct closure *cl; @@ -182,7 +179,7 @@ static int debug_seq_show(struct seq_file *f, void *data) seq_printf(f, " W %pS\n", (void *) cl->waiting_on); - seq_printf(f, "\n"); + seq_puts(f, "\n"); } spin_unlock_irq(&closure_list_lock); @@ -201,18 +198,11 @@ static const struct file_operations debug_ops = { .release = single_release }; -void __init closure_debug_init(void) +static int __init closure_debug_init(void) { - if (!IS_ERR_OR_NULL(bcache_debug)) - /* - * it is unnecessary to check return value of - * debugfs_create_file(), we should not care - * about this. - */ - closure_debug = debugfs_create_file( - "closures", 0400, bcache_debug, NULL, &debug_ops); + debugfs_create_file("closures", 0400, NULL, NULL, &debug_ops); + return 0; } -#endif +late_initcall(closure_debug_init) -MODULE_AUTHOR("Kent Overstreet "); -MODULE_LICENSE("GPL"); +#endif From patchwork Mon Jun 10 19:14:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 10985323 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AD52C13AD for ; Mon, 10 Jun 2019 19:15:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 900F9283C3 for ; Mon, 10 Jun 2019 19:15:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 81D0E2873B; Mon, 10 Jun 2019 19:15:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 36B44283C3 for ; Mon, 10 Jun 2019 19:15:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389435AbfFJTO4 (ORCPT ); Mon, 10 Jun 2019 15:14:56 -0400 Received: from mail-ua1-f65.google.com ([209.85.222.65]:38645 "EHLO mail-ua1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389339AbfFJTOo (ORCPT ); Mon, 10 Jun 2019 15:14:44 -0400 Received: by mail-ua1-f65.google.com with SMTP id j2so3529399uaq.5; Mon, 10 Jun 2019 12:14:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ckddRdKuY3n99DQ30I8Ucr4dD4i/wMlRy1QE1lcEpe0=; b=BdyAt9kKhE85l9YnxOxtZEV5ZNunxZzUaVFn/32oWTDCrze5pjJ7U9L53aXthrLGOy Lk51+I5TuZ5DmVk8+LXBzrN7h8fzZ9zKOMcL/d0LUc6aIvI2+FHKN+g+5//Rjh6BF2tK ckVeJfDgPIfikX7h64HDovPUq5ypZm7x6dqRUqYkuGOz53QhmE2KwSCvTWaWdpHzYc8b hMs213eb/Vi6vF+XJwc7hKWvHAYWIMZeEFnI+3vzVhRtr2DhnR0YBtnaNmhKJimfggnQ FL6xVr8XOdlQHZfcGMJD9hlzW43eULL3dLlXhl8f0gYn7bDonk8+U2D1vWIceEHLb5PR 1dyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ckddRdKuY3n99DQ30I8Ucr4dD4i/wMlRy1QE1lcEpe0=; b=ZdEnkOtNlY9hWRvoz1W8ZXG6qvBPW23pbJ+9stAXZPJb2YNZSL12zNTtfEc5qSflXn SQyRiOr04glsZgZpiPyB3GZOUsANkIT31pbq/5Du5HNRt5cqajS/BjAOLD6jCVDJ+KME bvAt29noj87VDBevtOoRUhgkOV5my0Uhe+3bjfpFXZsFzyqbEE2Chg2dQNn2V2CSA3nP 4OD622uQAiwm8di4Z+eD+2NNOOwQqRNfI+Z/UMEctGlCcfIbFaIK/Gl00CybYwr8QdBL gf4QGdtWmXTWB2VnOBzt8DwwGbjcNhdGge08A2WS6DFDT/2bjRoKiStCGmh7KpGTUl/S uaMA== X-Gm-Message-State: APjAAAWl2Ju5duHBERS2XIVMa6NxlOyQfkUnlgKA97RyVzZUrwsVEpkz l6uHeOaDbJ9lA1eSHNHazrg9H0JigQ== X-Google-Smtp-Source: APXvYqy9Ha5IMQid3VUAn6+bt6stjC+0pzI9SwfLubGHSWXn5aqbz4Jmg0Z0iQimvpw2aRpxt808Jg== X-Received: by 2002:a9f:21e7:: with SMTP id 94mr7564002uac.137.1560194083552; Mon, 10 Jun 2019 12:14:43 -0700 (PDT) Received: from kmo-pixel.hsd1.vt.comcast.net (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id t20sm4834014vkd.53.2019.06.10.12.14.42 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 10 Jun 2019 12:14:42 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 11/12] closures: closure_wait_event() Date: Mon, 10 Jun 2019 15:14:19 -0400 Message-Id: <20190610191420.27007-12-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190610191420.27007-1-kent.overstreet@gmail.com> References: <20190610191420.27007-1-kent.overstreet@gmail.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Kent Overstreet Acked-by: Coly Li --- include/linux/closure.h | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/include/linux/closure.h b/include/linux/closure.h index 308e38028c..abacb91c35 100644 --- a/include/linux/closure.h +++ b/include/linux/closure.h @@ -379,4 +379,26 @@ static inline void closure_call(struct closure *cl, closure_fn fn, continue_at_nobarrier(cl, fn, wq); } +#define __closure_wait_event(waitlist, _cond) \ +do { \ + struct closure cl; \ + \ + closure_init_stack(&cl); \ + \ + while (1) { \ + closure_wait(waitlist, &cl); \ + if (_cond) \ + break; \ + closure_sync(&cl); \ + } \ + closure_wake_up(waitlist); \ + closure_sync(&cl); \ +} while (0) + +#define closure_wait_event(waitlist, _cond) \ +do { \ + if (!(_cond)) \ + __closure_wait_event(waitlist, _cond); \ +} while (0) + #endif /* _LINUX_CLOSURE_H */ From patchwork Mon Jun 10 19:14:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 10985321 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6020B13AF for ; Mon, 10 Jun 2019 19:14:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4FA4F283C3 for ; Mon, 10 Jun 2019 19:14:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4420F2873B; Mon, 10 Jun 2019 19:14:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F22FB283C3 for ; Mon, 10 Jun 2019 19:14:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389366AbfFJTOs (ORCPT ); Mon, 10 Jun 2019 15:14:48 -0400 Received: from mail-vs1-f68.google.com ([209.85.217.68]:34602 "EHLO mail-vs1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389353AbfFJTOq (ORCPT ); Mon, 10 Jun 2019 15:14:46 -0400 Received: by mail-vs1-f68.google.com with SMTP id q64so6227258vsd.1; Mon, 10 Jun 2019 12:14:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=M+OkbCIxuFLegTVpH06BqaBwHcYDWj2efUoxZ29HmLU=; b=W1SjbKamopitAaoclsyBsK7m8S7AzERGRKlTfb+yFZ3adAWG2rzZr2hfUwnov+P87B vw18tsCdYvV8bx6u1lOYDktyzZbiKLEcNhxpFOiq/u53g3rN9bLIAzGy83lfCqbLtR4B IsoDXAhb8whDoMpnKFF9Cp7dWDFsMWwJ6+dj354s5d5Bq118PwSWtaOPd5+JB4agqV8A lRXpFX+waaRVdsPbZJS9Hfvb8/kaYk0hNBGs/IRL/gL09al/thBKjIc7RGjYrKYrX8Sb f6phYca2bykOr7RMkMFIYQHOgnxR4RL9AidVs34lmo20OitiRtakSp113Fksx4pfd3sr +FpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=M+OkbCIxuFLegTVpH06BqaBwHcYDWj2efUoxZ29HmLU=; b=IswXtzt08rMYe6SHcgqsnOx5+r7v/f4jyrOV/kX/2EjTiEZzoGT6KbzJQmXo7u1YQW 3YLSgLoyqcld74iE4m1X3UT8ZJtKCofifnzawOjgF9/6lW9634/oqL6NIay7qcfQJQOe fkJwZSc8aCv5nbtQWBUKyOVuME4Nqtq4jsDTsxukKnWpn9lzdr5Xixfjqrataopohbyu gg6xccLQloXF0ymCLFyIz1h6M0H8HSvpfFS7C+mvuHQljCzmigUbo7JswAc+sn/BIInG FKzAVKQ6l8ax4dnUFACiN9R30/xNjQRZySGRNajXJTLhan9yEW8N4AKJBXpFuooWRFTW YLKg== X-Gm-Message-State: APjAAAUGRf2c/gwWBcldYX7AIBmPJrKm+gjytQr/oBcEjJztNOUC8vYV yLmk142iriVYP5jYk0Lviv0MdwG4ew== X-Google-Smtp-Source: APXvYqxMjxT/TOdBkrDkVRcyvJVVBlm14OQBM2ULXJus1NpZtycPr2ZumLGbzRGxbtqTqldyNuKg9w== X-Received: by 2002:a67:f759:: with SMTP id w25mr18307516vso.235.1560194084934; Mon, 10 Jun 2019 12:14:44 -0700 (PDT) Received: from kmo-pixel.hsd1.vt.comcast.net (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id t20sm4834014vkd.53.2019.06.10.12.14.43 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 10 Jun 2019 12:14:44 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcache@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 12/12] closures: fix a race on wakeup from closure_sync Date: Mon, 10 Jun 2019 15:14:20 -0400 Message-Id: <20190610191420.27007-13-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190610191420.27007-1-kent.overstreet@gmail.com> References: <20190610191420.27007-1-kent.overstreet@gmail.com> MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Kent Overstreet Acked-by: Coly Li --- lib/closure.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/lib/closure.c b/lib/closure.c index 46cfe4c382..3e6366c262 100644 --- a/lib/closure.c +++ b/lib/closure.c @@ -104,8 +104,14 @@ struct closure_syncer { static void closure_sync_fn(struct closure *cl) { - cl->s->done = 1; - wake_up_process(cl->s->task); + struct closure_syncer *s = cl->s; + struct task_struct *p; + + rcu_read_lock(); + p = READ_ONCE(s->task); + s->done = 1; + wake_up_process(p); + rcu_read_unlock(); } void __sched __closure_sync(struct closure *cl)