From patchwork Wed Dec 20 17:21:54 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Matias_Bj=C3=B8rling?= X-Patchwork-Id: 10125981 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7625060638 for ; Wed, 20 Dec 2017 17:22:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5CF692851B for ; Wed, 20 Dec 2017 17:22:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 51BB72871C; Wed, 20 Dec 2017 17:22:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C26532851B for ; Wed, 20 Dec 2017 17:22:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755863AbdLTRWp (ORCPT ); Wed, 20 Dec 2017 12:22:45 -0500 Received: from mail-wm0-f68.google.com ([74.125.82.68]:34211 "EHLO mail-wm0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755983AbdLTRW3 (ORCPT ); Wed, 20 Dec 2017 12:22:29 -0500 Received: by mail-wm0-f68.google.com with SMTP id y82so16621404wmg.1 for ; Wed, 20 Dec 2017 09:22:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bjorling.me; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JNJztgZypty3w3FH+gAUTJ566sS6bLWw1ATg+i1gesA=; b=FaHbm2U7lyyemYUfgDhv9R7SESYKfoi2W+4AqNqzp3t75m4QH+eMhU1WyQ2SvqJSWq SJI75o/LrRxXRWLG+bTQZTzDx8+/6De7I0PnxD9cugAeoED3fMTlggbGOXoHYsac9bRp zfks6+/MJo48Pvife94EfuNU2GffWbs9FlCzA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JNJztgZypty3w3FH+gAUTJ566sS6bLWw1ATg+i1gesA=; b=ojedN0Ji45K/Z+z18ARLkxVdWcUAX9fE2Y5Qd5TQqe2FMb+ubjCG7DNcfj7O50beU5 xM/fwSg925ylP+9zD8JJZD+ad1ZUUjaoZpHeWAr6GEnracTEXpW9X8jonBmmuE39CRtq XBEVPo8FVL4oYIs/jTLZW31/DN+yXR04cyUnmeMKSz/wveq0LbviBXfa9QBpwLzGEi5h xzh7+ghB0ONVXMqK3Vqd8CLZUFwsrFJIQitMEJCYQPglnH5zx35mf8yVj53Zr71cRzLX ipM+vL7glt5X2ETqT78syA44l4iReJYFjwuVp0tIyEHvxXZDLwOvlD2QrS4sPOv9bMN/ uJyA== X-Gm-Message-State: AKGB3mIs44ZuTESOB5L/ZHQT5Kt2LLWFOCJG9YhpNdhDXUyEMw/3jtrA Hqb0Jp6AdJBGVVD2ag2GLkE6E1Pm X-Google-Smtp-Source: ACJfBovQ1RcysSGIMJOo9715z4LKCFRI3EnOlSwvqtLm8JdV6x2DnDxy2L560mEC76LvDvVQnGFjPQ== X-Received: by 10.80.169.123 with SMTP id m56mr6356279edc.126.1513790547825; Wed, 20 Dec 2017 09:22:27 -0800 (PST) Received: from skyninja.cnexlabs.com (6164211-cl69.boa.fiberby.dk. [193.106.164.211]) by smtp.gmail.com with ESMTPSA id 26sm15339120eds.67.2017.12.20.09.22.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 20 Dec 2017 09:22:27 -0800 (PST) From: =?UTF-8?q?Matias=20Bj=C3=B8rling?= To: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Hans Holmberg , =?UTF-8?q?Javier=20Gonz=C3=A1lez?= , =?UTF-8?q?Matias=20Bj=C3=B8rling?= Subject: [PATCH 14/25] lightnvm: pblk: clear flush point on completed writes Date: Wed, 20 Dec 2017 18:21:54 +0100 Message-Id: <20171220172205.26464-15-m@bjorling.me> X-Mailer: git-send-email 2.9.3 In-Reply-To: <20171220172205.26464-1-m@bjorling.me> References: <20171220172205.26464-1-m@bjorling.me> MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Hans Holmberg Move completion of syncs and clearing of flush points to the write completion path - this ensures that the data has been committed to the media before completing bios containing syncs. Signed-off-by: Hans Holmberg Signed-off-by: Javier González Signed-off-by: Matias Bjørling --- drivers/lightnvm/pblk-rb.c | 58 +++++++++++++++++++++---------------------- drivers/lightnvm/pblk-write.c | 17 ++++++++++++- 2 files changed, 44 insertions(+), 31 deletions(-) diff --git a/drivers/lightnvm/pblk-rb.c b/drivers/lightnvm/pblk-rb.c index 941842e..9a4100c 100644 --- a/drivers/lightnvm/pblk-rb.c +++ b/drivers/lightnvm/pblk-rb.c @@ -353,17 +353,17 @@ static int pblk_rb_flush_point_set(struct pblk_rb *rb, struct bio *bio, unsigned int pos) { struct pblk_rb_entry *entry; - unsigned int subm, flush_point; + unsigned int sync, flush_point; - subm = READ_ONCE(rb->subm); + sync = READ_ONCE(rb->sync); + + if (pos == sync) + return 0; #ifdef CONFIG_NVM_DEBUG atomic_inc(&rb->inflight_flush_point); #endif - if (pos == subm) - return 0; - flush_point = (pos == 0) ? (rb->nr_entries - 1) : (pos - 1); entry = &rb->entries[flush_point]; @@ -606,22 +606,6 @@ unsigned int pblk_rb_read_to_bio(struct pblk_rb *rb, struct nvm_rq *rqd, return NVM_IO_ERR; } - if (flags & PBLK_FLUSH_ENTRY) { - unsigned int flush_point; - - flush_point = READ_ONCE(rb->flush_point); - if (flush_point == pos) { - /* Protect flush points */ - smp_store_release(&rb->flush_point, - EMPTY_ENTRY); - } - - flags &= ~PBLK_FLUSH_ENTRY; -#ifdef CONFIG_NVM_DEBUG - atomic_dec(&rb->inflight_flush_point); -#endif - } - flags &= ~PBLK_WRITTEN_DATA; flags |= PBLK_SUBMITTED_ENTRY; @@ -731,15 +715,24 @@ void pblk_rb_sync_end(struct pblk_rb *rb, unsigned long *flags) unsigned int pblk_rb_sync_advance(struct pblk_rb *rb, unsigned int nr_entries) { - unsigned int sync; - unsigned int i; - + unsigned int sync, flush_point; lockdep_assert_held(&rb->s_lock); sync = READ_ONCE(rb->sync); + flush_point = READ_ONCE(rb->flush_point); - for (i = 0; i < nr_entries; i++) - sync = (sync + 1) & (rb->nr_entries - 1); + if (flush_point != EMPTY_ENTRY) { + unsigned int secs_to_flush; + + secs_to_flush = pblk_rb_ring_count(flush_point, sync, + rb->nr_entries); + if (secs_to_flush <= nr_entries) { + /* Protect flush points */ + smp_store_release(&rb->flush_point, EMPTY_ENTRY); + } + } + + sync = (sync + nr_entries) & (rb->nr_entries - 1); /* Protect from counts */ smp_store_release(&rb->sync, sync); @@ -747,22 +740,27 @@ unsigned int pblk_rb_sync_advance(struct pblk_rb *rb, unsigned int nr_entries) return sync; } +/* Calculate how many sectors to submit up to the current flush point. */ unsigned int pblk_rb_flush_point_count(struct pblk_rb *rb) { - unsigned int subm, flush_point; - unsigned int count; + unsigned int subm, sync, flush_point; + unsigned int submitted, to_flush; /* Protect flush points */ flush_point = smp_load_acquire(&rb->flush_point); if (flush_point == EMPTY_ENTRY) return 0; + /* Protect syncs */ + sync = smp_load_acquire(&rb->sync); + subm = READ_ONCE(rb->subm); + submitted = pblk_rb_ring_count(subm, sync, rb->nr_entries); /* The sync point itself counts as a sector to sync */ - count = pblk_rb_ring_count(flush_point, subm, rb->nr_entries) + 1; + to_flush = pblk_rb_ring_count(flush_point, sync, rb->nr_entries) + 1; - return count; + return (submitted < to_flush) ? (to_flush - submitted) : 0; } /* diff --git a/drivers/lightnvm/pblk-write.c b/drivers/lightnvm/pblk-write.c index 018af87..aae86ed 100644 --- a/drivers/lightnvm/pblk-write.c +++ b/drivers/lightnvm/pblk-write.c @@ -21,13 +21,28 @@ static unsigned long pblk_end_w_bio(struct pblk *pblk, struct nvm_rq *rqd, struct pblk_c_ctx *c_ctx) { struct bio *original_bio; + struct pblk_rb *rwb = &pblk->rwb; unsigned long ret; int i; for (i = 0; i < c_ctx->nr_valid; i++) { struct pblk_w_ctx *w_ctx; + int pos = c_ctx->sentry + i; + int flags; + + w_ctx = pblk_rb_w_ctx(rwb, pos); + flags = READ_ONCE(w_ctx->flags); + + if (flags & PBLK_FLUSH_ENTRY) { + flags &= ~PBLK_FLUSH_ENTRY; + /* Release flags on context. Protect from writes */ + smp_store_release(&w_ctx->flags, flags); + +#ifdef CONFIG_NVM_DEBUG + atomic_dec(&rwb->inflight_flush_point); +#endif + } - w_ctx = pblk_rb_w_ctx(&pblk->rwb, c_ctx->sentry + i); while ((original_bio = bio_list_pop(&w_ctx->bios))) bio_endio(original_bio); }