From patchwork Mon Sep 17 05:29:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiner Litz X-Patchwork-Id: 10602077 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 76BA0112B for ; Mon, 17 Sep 2018 05:29:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6558429585 for ; Mon, 17 Sep 2018 05:29:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 59B2D2958D; Mon, 17 Sep 2018 05:29:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D90B229585 for ; Mon, 17 Sep 2018 05:29:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727456AbeIQKzi (ORCPT ); Mon, 17 Sep 2018 06:55:38 -0400 Received: from mail-pl1-f196.google.com ([209.85.214.196]:35178 "EHLO mail-pl1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726073AbeIQKzi (ORCPT ); Mon, 17 Sep 2018 06:55:38 -0400 Received: by mail-pl1-f196.google.com with SMTP id g2-v6so6853681plo.2 for ; Sun, 16 Sep 2018 22:29:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PjzKlMAPQlWSj1cqXFEQrvy/8NUScWJcuVih85m6ffw=; b=hSSUjKxrkzgqEMfKgDPucc+Wwf0utkm/s8Uq/jXvNN/maOv9g7g25+MCmf0EziGM/S gBSSkQw9U+M+0h8T3/ybEXMUEvoSnY71x+2Jh4pEb1wbDVkRUkAhGnqnQHT5x54UWUVL herTMtuvEg9xg1nX4MrrQnFaSUYSPoyahszHA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=PjzKlMAPQlWSj1cqXFEQrvy/8NUScWJcuVih85m6ffw=; b=R30O1jmH72MPMqlLmM8l6N4m/fRAd04i703TTkQdyzkDRqELS3sLw8hH1GCQCr+rP5 MRcKgNe8z2GaDnmvQAZDV6eqsV0CVgKhQ5tRXalv8wyS5q/tWHdl7cUPf5IeOxwMYzO2 Dehhi4EWhp6sS8EXsIM+SM8gbZYFRPt5zLktkjs0o1zNv4JxmxnNFQx0iJULLW2gkY5B Lb9bCleso9AyHfYoAMdLh984+VRHEF8KEXGtNHE1PA3Rd6i9AFEf4HaAnXWr45wgyOJI sADzM0FsysxF1C8Yw7oupdgqpO6AEegCGrbWdGi5BGh6eZFrNRacMA94U9o3/j0WcQ7g e5pA== X-Gm-Message-State: APzg51BjSYW9/MzcCM9yVYicRjme5gBxqF4DFm2jOhRRYExafbk2IBid Kye4ZHJiC+bFIKmbYa7yXNjq/uO1w28Lwg== X-Google-Smtp-Source: ANB0VdZsvlhsWO1XJKN5lvGZwrisfYgIrVuzz+weYeSF9tvdzywqAL3poIL9Io0RvXH5eMMcg7L4fQ== X-Received: by 2002:a17:902:a504:: with SMTP id s4-v6mr23769413plq.101.1537162192108; Sun, 16 Sep 2018 22:29:52 -0700 (PDT) Received: from bohr1.soe.ucsc.edu (bohr1.soe.ucsc.edu. [128.114.52.184]) by smtp.gmail.com with ESMTPSA id u11-v6sm22197795pfd.117.2018.09.16.22.29.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 16 Sep 2018 22:29:51 -0700 (PDT) From: Heiner Litz To: linux-block@vger.kernel.org, hlitz@ucsc.edu Cc: javier@cnexlabs.com, mb@lightnvm.io, igor.j.konopko@intel.com, marcin.dziegielewski@intel.com Subject: [RFC PATCH 1/6] lightnvm: pblk: refactor read and write APIs Date: Sun, 16 Sep 2018 22:29:34 -0700 Message-Id: <20180917052939.4776-2-hlitz@ucsc.edu> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180917052939.4776-1-hlitz@ucsc.edu> References: <20180917052939.4776-1-hlitz@ucsc.edu> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In prepartion of supporting RAIL, expose read and write APIs so their functionality can be leveraged by RAIL. Signed-off-by: Heiner Litz --- drivers/lightnvm/pblk-read.c | 8 +++----- drivers/lightnvm/pblk-write.c | 4 ++-- drivers/lightnvm/pblk.h | 7 +++++++ 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/drivers/lightnvm/pblk-read.c b/drivers/lightnvm/pblk-read.c index 6d13763f2f6a..67d44caefff4 100644 --- a/drivers/lightnvm/pblk-read.c +++ b/drivers/lightnvm/pblk-read.c @@ -170,8 +170,7 @@ static void pblk_end_user_read(struct bio *bio) bio_endio(bio); } -static void __pblk_end_io_read(struct pblk *pblk, struct nvm_rq *rqd, - bool put_line) +void __pblk_end_io_read(struct pblk *pblk, struct nvm_rq *rqd, bool put_line) { struct nvm_tgt_dev *dev = pblk->dev; struct pblk_g_ctx *r_ctx = nvm_rq_to_pdu(rqd); @@ -285,10 +284,9 @@ static void pblk_end_partial_read(struct nvm_rq *rqd) __pblk_end_io_read(pblk, rqd, false); } -static int pblk_setup_partial_read(struct pblk *pblk, struct nvm_rq *rqd, +int pblk_setup_partial_read(struct pblk *pblk, struct nvm_rq *rqd, unsigned int bio_init_idx, - unsigned long *read_bitmap, - int nr_holes) + unsigned long *read_bitmap, int nr_holes) { struct pblk_sec_meta *meta_list = rqd->meta_list; struct pblk_g_ctx *r_ctx = nvm_rq_to_pdu(rqd); diff --git a/drivers/lightnvm/pblk-write.c b/drivers/lightnvm/pblk-write.c index 9554febee480..1ce03d7c873b 100644 --- a/drivers/lightnvm/pblk-write.c +++ b/drivers/lightnvm/pblk-write.c @@ -217,7 +217,7 @@ static void pblk_submit_rec(struct work_struct *work) } -static void pblk_end_w_fail(struct pblk *pblk, struct nvm_rq *rqd) +void pblk_end_w_fail(struct pblk *pblk, struct nvm_rq *rqd) { struct pblk_rec_ctx *recovery; @@ -500,7 +500,7 @@ static struct pblk_line *pblk_should_submit_meta_io(struct pblk *pblk, return meta_line; } -static int pblk_submit_io_set(struct pblk *pblk, struct nvm_rq *rqd) +int pblk_submit_io_set(struct pblk *pblk, struct nvm_rq *rqd) { struct ppa_addr erase_ppa; struct pblk_line *meta_line; diff --git a/drivers/lightnvm/pblk.h b/drivers/lightnvm/pblk.h index 3596043332f2..eab50df70ae6 100644 --- a/drivers/lightnvm/pblk.h +++ b/drivers/lightnvm/pblk.h @@ -861,6 +861,8 @@ void pblk_lookup_l2p_seq(struct pblk *pblk, struct ppa_addr *ppas, int pblk_write_to_cache(struct pblk *pblk, struct bio *bio, unsigned long flags); int pblk_write_gc_to_cache(struct pblk *pblk, struct pblk_gc_rq *gc_rq); +void pblk_end_w_fail(struct pblk *pblk, struct nvm_rq *rqd); +int pblk_submit_io_set(struct pblk *pblk, struct nvm_rq *rqd); /* * pblk map @@ -886,6 +888,11 @@ void pblk_write_kick(struct pblk *pblk); extern struct bio_set pblk_bio_set; int pblk_submit_read(struct pblk *pblk, struct bio *bio); int pblk_submit_read_gc(struct pblk *pblk, struct pblk_gc_rq *gc_rq); +void __pblk_end_io_read(struct pblk *pblk, struct nvm_rq *rqd, bool put_line); +int pblk_setup_partial_read(struct pblk *pblk, struct nvm_rq *rqd, + unsigned int bio_init_idx, + unsigned long *read_bitmap, int nr_holes); + /* * pblk recovery */ From patchwork Mon Sep 17 05:29:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiner Litz X-Patchwork-Id: 10602079 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DCDC8112B for ; Mon, 17 Sep 2018 05:29:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CD86629585 for ; Mon, 17 Sep 2018 05:29:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C22262958D; Mon, 17 Sep 2018 05:29:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5D2F529585 for ; Mon, 17 Sep 2018 05:29:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726073AbeIQKzl (ORCPT ); Mon, 17 Sep 2018 06:55:41 -0400 Received: from mail-pf1-f195.google.com ([209.85.210.195]:36161 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727835AbeIQKzl (ORCPT ); Mon, 17 Sep 2018 06:55:41 -0400 Received: by mail-pf1-f195.google.com with SMTP id b11-v6so7001762pfo.3 for ; Sun, 16 Sep 2018 22:29:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=hpmWOpWwCRFvxA7Ic9DYW9Srabix9GqnpgXvi125Y6U=; b=cJmwGpzoqCbNqy1FnL1f+1HH0YnyMocsZjPTP1tDUMUb27IawrCOLm0o4TAsKF3xOM eLtRf3iOJiIab+9pEtrtv6kiZJ9eSODm+N9AkKbcvFk/n84rH9SZLrhrudBixf2paXu8 Dxyw3KZmVA7Z0brvEPmAHFt+Yyn1C3NwIitOo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=hpmWOpWwCRFvxA7Ic9DYW9Srabix9GqnpgXvi125Y6U=; b=XccFsDoyOEkPrE1zDLRRYWv+4URAq+OaG5PSLCzHrtQDTi2NSmfaDFKbQj3LPcb1hH eWyRZCPND6XY2gKTWfiLfjRMwGebIxsxwWufHCoFXftc1iFcCguPHRMJXH8UCwjN56zY k5H13eGvq7//4TZKXDeeco167HSZSNNDK/Ruz2cRdyJdKLIR+KDCis/NwFPy1Kq3uxEG CWVwNBqbpn1eT/thvWKrFvLxKjI2QOh7Jv2Alpb2Ug3680+d7pnbrJ2IuQ6TjaP8++Zr SaqrTyvDT7y7IfgZNgU0XJZq06L+zeOt0LiO4q9/S2dnnSBCzuoqXfKDyakfFpyikXH2 Di+w== X-Gm-Message-State: APzg51DU7MAGFM1LQ7WFRuOR6M7O1EHJE36wnRggImHxCLa52CIiYvJU 5gOrv2hy0w9qIeCRvOyEcgskyiPrFS6gGg== X-Google-Smtp-Source: ANB0Vdaiq5+rdkFeRpgEUXNQ6+jOgDy402ZsjEXnY8s+VLBZkY98C4OnMG9nzwJFQKumqA6m8HHGmA== X-Received: by 2002:a62:5cc1:: with SMTP id q184-v6mr24518864pfb.241.1537162194359; Sun, 16 Sep 2018 22:29:54 -0700 (PDT) Received: from bohr1.soe.ucsc.edu (bohr1.soe.ucsc.edu. [128.114.52.184]) by smtp.gmail.com with ESMTPSA id u11-v6sm22197795pfd.117.2018.09.16.22.29.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 16 Sep 2018 22:29:53 -0700 (PDT) From: Heiner Litz To: linux-block@vger.kernel.org, hlitz@ucsc.edu Cc: javier@cnexlabs.com, mb@lightnvm.io, igor.j.konopko@intel.com, marcin.dziegielewski@intel.com Subject: [RFC PATCH 2/6] lightnvm: pblk: Add configurable mapping function Date: Sun, 16 Sep 2018 22:29:35 -0700 Message-Id: <20180917052939.4776-3-hlitz@ucsc.edu> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180917052939.4776-1-hlitz@ucsc.edu> References: <20180917052939.4776-1-hlitz@ucsc.edu> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In prepartion of supporting RAIL, introduce a new function pointer so that different mapping functions can be used to determine sector placement. Signed-off-by: Heiner Litz --- drivers/lightnvm/pblk-init.c | 2 ++ drivers/lightnvm/pblk-map.c | 18 +++++++++--------- drivers/lightnvm/pblk.h | 13 +++++++++++++ 3 files changed, 24 insertions(+), 9 deletions(-) diff --git a/drivers/lightnvm/pblk-init.c b/drivers/lightnvm/pblk-init.c index fb66bc84d5ca..2b9c6ebd9fac 100644 --- a/drivers/lightnvm/pblk-init.c +++ b/drivers/lightnvm/pblk-init.c @@ -411,6 +411,8 @@ static int pblk_core_init(struct pblk *pblk) pblk->pad_rst_wa = 0; pblk->gc_rst_wa = 0; + pblk->map_page = pblk_map_page_data; + atomic64_set(&pblk->nr_flush, 0); pblk->nr_flush_rst = 0; diff --git a/drivers/lightnvm/pblk-map.c b/drivers/lightnvm/pblk-map.c index ff677ca6e4e1..9490601de3a5 100644 --- a/drivers/lightnvm/pblk-map.c +++ b/drivers/lightnvm/pblk-map.c @@ -18,11 +18,11 @@ #include "pblk.h" -static int pblk_map_page_data(struct pblk *pblk, unsigned int sentry, - struct ppa_addr *ppa_list, - unsigned long *lun_bitmap, - struct pblk_sec_meta *meta_list, - unsigned int valid_secs) +int pblk_map_page_data(struct pblk *pblk, unsigned int sentry, + struct ppa_addr *ppa_list, + unsigned long *lun_bitmap, + struct pblk_sec_meta *meta_list, + unsigned int valid_secs) { struct pblk_line *line = pblk_line_get_data(pblk); struct pblk_emeta *emeta; @@ -95,8 +95,8 @@ void pblk_map_rq(struct pblk *pblk, struct nvm_rq *rqd, unsigned int sentry, for (i = off; i < rqd->nr_ppas; i += min) { map_secs = (i + min > valid_secs) ? (valid_secs % min) : min; - if (pblk_map_page_data(pblk, sentry + i, &ppa_list[i], - lun_bitmap, &meta_list[i], map_secs)) { + if (pblk->map_page(pblk, sentry + i, &ppa_list[i], lun_bitmap, + &meta_list[i], map_secs)) { bio_put(rqd->bio); pblk_free_rqd(pblk, rqd, PBLK_WRITE); pblk_pipeline_stop(pblk); @@ -121,8 +121,8 @@ void pblk_map_erase_rq(struct pblk *pblk, struct nvm_rq *rqd, for (i = 0; i < rqd->nr_ppas; i += min) { map_secs = (i + min > valid_secs) ? (valid_secs % min) : min; - if (pblk_map_page_data(pblk, sentry + i, &ppa_list[i], - lun_bitmap, &meta_list[i], map_secs)) { + if (pblk->map_page(pblk, sentry + i, &ppa_list[i], lun_bitmap, + &meta_list[i], map_secs)) { bio_put(rqd->bio); pblk_free_rqd(pblk, rqd, PBLK_WRITE); pblk_pipeline_stop(pblk); diff --git a/drivers/lightnvm/pblk.h b/drivers/lightnvm/pblk.h index eab50df70ae6..87dc24772dad 100644 --- a/drivers/lightnvm/pblk.h +++ b/drivers/lightnvm/pblk.h @@ -604,6 +604,12 @@ struct pblk_addrf { int sec_ws_stripe; }; +typedef int (pblk_map_page_fn)(struct pblk *pblk, unsigned int sentry, + struct ppa_addr *ppa_list, + unsigned long *lun_bitmap, + struct pblk_sec_meta *meta_list, + unsigned int valid_secs); + struct pblk { struct nvm_tgt_dev *dev; struct gendisk *disk; @@ -709,6 +715,8 @@ struct pblk { struct timer_list wtimer; struct pblk_gc gc; + + pblk_map_page_fn *map_page; }; struct pblk_line_ws { @@ -873,6 +881,11 @@ void pblk_map_erase_rq(struct pblk *pblk, struct nvm_rq *rqd, void pblk_map_rq(struct pblk *pblk, struct nvm_rq *rqd, unsigned int sentry, unsigned long *lun_bitmap, unsigned int valid_secs, unsigned int off); +int pblk_map_page_data(struct pblk *pblk, unsigned int sentry, + struct ppa_addr *ppa_list, + unsigned long *lun_bitmap, + struct pblk_sec_meta *meta_list, + unsigned int valid_secs); /* * pblk write thread From patchwork Mon Sep 17 05:29:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiner Litz X-Patchwork-Id: 10602081 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 85FFD112B for ; Mon, 17 Sep 2018 05:29:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7476029585 for ; Mon, 17 Sep 2018 05:29:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 68BB22958D; Mon, 17 Sep 2018 05:29:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5C5F229585 for ; Mon, 17 Sep 2018 05:29:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727851AbeIQKzn (ORCPT ); Mon, 17 Sep 2018 06:55:43 -0400 Received: from mail-pf1-f196.google.com ([209.85.210.196]:45986 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727835AbeIQKzn (ORCPT ); Mon, 17 Sep 2018 06:55:43 -0400 Received: by mail-pf1-f196.google.com with SMTP id i26-v6so6976895pfo.12 for ; Sun, 16 Sep 2018 22:29:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Q6PGyVozigerN4HKa8YCaKUMjLvrFGRNvsHtvzk7Qt4=; b=fFYRGvhTJieCXiAfO5lONXZ1O1rVA9tO4oI0rdomHgddK30s5iUFRldgT78PFaiOxr UOq/slbKg5jjk7MX9Qdb8FZEWKbEjyM0TOoQzO2AW/cg/m1tquG+XE7vpsTblC7f7T1b 1F108KJF/2jVMKz1gyxt93CD4GTVKIFku8Bew= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Q6PGyVozigerN4HKa8YCaKUMjLvrFGRNvsHtvzk7Qt4=; b=asIWbkwI/rOd77421Yq9JLEEMy7BvdLyLypnwpA/FMIpmn9vmEWjpyhHlLj2FzU/WO luO6m7fuyFcOfqjIgT5p4vyJ8usng1vOg2iAfWlAa5n505MVSmWq3elWgbE2X8lXuLbo o9fI/S+YhZrSRPQCK6kzO2yt+YMgrIO0F1NSyNdwfnQFqMVn6HSiMEZ5d+veaYMX/H1i WeVtE62jwbbFJyZvtZSxaSaLKMy8pE9OlQhLE8n1Ut6UKBOjQ0xOQKvKq6izvTmCyx6m XAGHoMVjHc1MoseIpQcGVUgO0jHtzLRUHxU43qmwTXeyovVUIxwiNN/DwzVqDH+ZmAc9 30Mw== X-Gm-Message-State: APzg51CSIdfAX/jVBdlIbt2+w3uO07YegYhGG149QlS0KnhU8pE3zPqr nLZd3+cEK1z6xOeGhpjR8VywnPo0aNYs4g== X-Google-Smtp-Source: ANB0Vda21kKRnu/y++pDN8moFXsz/gm7ZUtxN8TX3WYqGm4XzHeW5pnFXLXKfWfQUWTYcVC9Hjzm+g== X-Received: by 2002:a63:6b86:: with SMTP id g128-v6mr21688977pgc.344.1537162196320; Sun, 16 Sep 2018 22:29:56 -0700 (PDT) Received: from bohr1.soe.ucsc.edu (bohr1.soe.ucsc.edu. [128.114.52.184]) by smtp.gmail.com with ESMTPSA id u11-v6sm22197795pfd.117.2018.09.16.22.29.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 16 Sep 2018 22:29:55 -0700 (PDT) From: Heiner Litz To: linux-block@vger.kernel.org, hlitz@ucsc.edu Cc: javier@cnexlabs.com, mb@lightnvm.io, igor.j.konopko@intel.com, marcin.dziegielewski@intel.com Subject: [RFC PATCH 3/6] lightnvm: pblk: Refactor end_io function in pblk_submit_io_set Date: Sun, 16 Sep 2018 22:29:36 -0700 Message-Id: <20180917052939.4776-4-hlitz@ucsc.edu> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180917052939.4776-1-hlitz@ucsc.edu> References: <20180917052939.4776-1-hlitz@ucsc.edu> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In preparation of supporting RAIL, refactor pblk_submit_io_set in the write path so that the end_io function can be specified when setting up the request. Signed-off-by: Heiner Litz --- drivers/lightnvm/pblk-write.c | 11 ++++++----- drivers/lightnvm/pblk.h | 3 ++- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/drivers/lightnvm/pblk-write.c b/drivers/lightnvm/pblk-write.c index 1ce03d7c873b..6eba38b83acd 100644 --- a/drivers/lightnvm/pblk-write.c +++ b/drivers/lightnvm/pblk-write.c @@ -309,7 +309,7 @@ static int pblk_alloc_w_rq(struct pblk *pblk, struct nvm_rq *rqd, } static int pblk_setup_w_rq(struct pblk *pblk, struct nvm_rq *rqd, - struct ppa_addr *erase_ppa) + struct ppa_addr *erase_ppa, nvm_end_io_fn(*end_io)) { struct pblk_line_meta *lm = &pblk->lm; struct pblk_line *e_line = pblk_line_get_erase(pblk); @@ -325,7 +325,7 @@ static int pblk_setup_w_rq(struct pblk *pblk, struct nvm_rq *rqd, return -ENOMEM; c_ctx->lun_bitmap = lun_bitmap; - ret = pblk_alloc_w_rq(pblk, rqd, nr_secs, pblk_end_io_write); + ret = pblk_alloc_w_rq(pblk, rqd, nr_secs, end_io); if (ret) { kfree(lun_bitmap); return ret; @@ -500,7 +500,8 @@ static struct pblk_line *pblk_should_submit_meta_io(struct pblk *pblk, return meta_line; } -int pblk_submit_io_set(struct pblk *pblk, struct nvm_rq *rqd) +int pblk_submit_io_set(struct pblk *pblk, struct nvm_rq *rqd, + nvm_end_io_fn(*end_io)) { struct ppa_addr erase_ppa; struct pblk_line *meta_line; @@ -509,7 +510,7 @@ int pblk_submit_io_set(struct pblk *pblk, struct nvm_rq *rqd) pblk_ppa_set_empty(&erase_ppa); /* Assign lbas to ppas and populate request structure */ - err = pblk_setup_w_rq(pblk, rqd, &erase_ppa); + err = pblk_setup_w_rq(pblk, rqd, &erase_ppa, end_io); if (err) { pblk_err(pblk, "could not setup write request: %d\n", err); return NVM_IO_ERR; @@ -631,7 +632,7 @@ static int pblk_submit_write(struct pblk *pblk) goto fail_put_bio; } - if (pblk_submit_io_set(pblk, rqd)) + if (pblk_submit_io_set(pblk, rqd, pblk_end_io_write)) goto fail_free_bio; #ifdef CONFIG_NVM_PBLK_DEBUG diff --git a/drivers/lightnvm/pblk.h b/drivers/lightnvm/pblk.h index 87dc24772dad..64d9c206ec52 100644 --- a/drivers/lightnvm/pblk.h +++ b/drivers/lightnvm/pblk.h @@ -870,7 +870,8 @@ int pblk_write_to_cache(struct pblk *pblk, struct bio *bio, unsigned long flags); int pblk_write_gc_to_cache(struct pblk *pblk, struct pblk_gc_rq *gc_rq); void pblk_end_w_fail(struct pblk *pblk, struct nvm_rq *rqd); -int pblk_submit_io_set(struct pblk *pblk, struct nvm_rq *rqd); +int pblk_submit_io_set(struct pblk *pblk, struct nvm_rq *rqd, + nvm_end_io_fn(*end_io)); /* * pblk map From patchwork Mon Sep 17 05:29:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiner Litz X-Patchwork-Id: 10602083 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 89622112B for ; Mon, 17 Sep 2018 05:30:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7A19929585 for ; Mon, 17 Sep 2018 05:30:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6CF452958D; Mon, 17 Sep 2018 05:30:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1AB5B29585 for ; Mon, 17 Sep 2018 05:30:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727874AbeIQKzp (ORCPT ); Mon, 17 Sep 2018 06:55:45 -0400 Received: from mail-pg1-f193.google.com ([209.85.215.193]:36602 "EHLO mail-pg1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727835AbeIQKzp (ORCPT ); Mon, 17 Sep 2018 06:55:45 -0400 Received: by mail-pg1-f193.google.com with SMTP id d1-v6so7096983pgo.3 for ; Sun, 16 Sep 2018 22:29:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Aci+kam7hng1Lfb4VLItJ/OAVhVqPKcNKxIFHEK4lNI=; b=ZP/kFytfj5FwcDOT+RCR4g2gGQtRwR+F3ueudZKbAIX+G9jeimGNyzGkpnxnZOXObZ RldfrCR3/dJAHJF4V+uk+hegxOT7UvNvMcdBLR83gkIKf7cCDVmrO1SgRvbVDOLxbSPt TTqeaA6Z5mNQb5IhgilISiF0wLK+x6xnrMtYg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Aci+kam7hng1Lfb4VLItJ/OAVhVqPKcNKxIFHEK4lNI=; b=QGTOdiXZ/xObTRaaslVXwITdnGHQJeTzYYrEttVAdZGAmyR5zT5QE3NzbPgk7mfuK5 coVfKG7m60CEVtjdiIQ/d6kuuigZfblobanm1KEF98/S178DDfAKkKJWbjW89B9bH9bd jxebM69UtJDnGSMY6l5HH3/8KzjxVNUca+QggfdJg1auYeaL69NkVQ9dMmZKDdY40tsP MJDh1Y3cSyI5aA9QT2+cWU+n3nPxVu7k8YkCxImqY7Vkx/EMO9a0nT3TAaUEVEenV7I3 Aq2/BDNkXeeE1ADwvb1ZdO0nRLnJsBdDTwEWwEbU6l9ww80CjzL/MsFMEiitgH6JHTRK 1FSw== X-Gm-Message-State: APzg51BvD8yX/2oXG70hzk5S9FPxjOso/AdYdJQGLm0XywGpVsRth41I Z80lnJ7WmwR6ABFbhwPq8DaUrSLjzI9D6Q== X-Google-Smtp-Source: ANB0VdaMiUxd0YZLgVLvvS0RKB8FJFxp52mvUR/md7jetfiDpB3hgii46rEFugpehGQO3WjSUCeBkg== X-Received: by 2002:a62:da0b:: with SMTP id c11-v6mr24258128pfh.68.1537162198215; Sun, 16 Sep 2018 22:29:58 -0700 (PDT) Received: from bohr1.soe.ucsc.edu (bohr1.soe.ucsc.edu. [128.114.52.184]) by smtp.gmail.com with ESMTPSA id u11-v6sm22197795pfd.117.2018.09.16.22.29.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 16 Sep 2018 22:29:57 -0700 (PDT) From: Heiner Litz To: linux-block@vger.kernel.org, hlitz@ucsc.edu Cc: javier@cnexlabs.com, mb@lightnvm.io, igor.j.konopko@intel.com, marcin.dziegielewski@intel.com Subject: [RFC PATCH 4/6] lightnvm: pblk: Add pblk_submit_io_sem Date: Sun, 16 Sep 2018 22:29:37 -0700 Message-Id: <20180917052939.4776-5-hlitz@ucsc.edu> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180917052939.4776-1-hlitz@ucsc.edu> References: <20180917052939.4776-1-hlitz@ucsc.edu> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In preparation of supporting RAIL, add a new API pblk_submit_io_sem which takes the lun semaphore before submitting the asynchronous request. Signed-off-by: Heiner Litz --- drivers/lightnvm/pblk-core.c | 11 +++++++++++ drivers/lightnvm/pblk.h | 1 + 2 files changed, 12 insertions(+) diff --git a/drivers/lightnvm/pblk-core.c b/drivers/lightnvm/pblk-core.c index 2e40666fdf80..a31bf359f905 100644 --- a/drivers/lightnvm/pblk-core.c +++ b/drivers/lightnvm/pblk-core.c @@ -492,6 +492,17 @@ int pblk_submit_io(struct pblk *pblk, struct nvm_rq *rqd) return nvm_submit_io(dev, rqd); } +int pblk_submit_io_sem(struct pblk *pblk, struct nvm_rq *rqd) +{ + struct ppa_addr *ppa_list = nvm_rq_to_ppa_list(rqd); + int ret; + + pblk_down_chunk(pblk, ppa_list[0]); + ret = pblk_submit_io(pblk, rqd); + + return ret; +} + void pblk_check_chunk_state_update(struct pblk *pblk, struct nvm_rq *rqd) { struct ppa_addr *ppa_list = nvm_rq_to_ppa_list(rqd); diff --git a/drivers/lightnvm/pblk.h b/drivers/lightnvm/pblk.h index 64d9c206ec52..bd88784e51d9 100644 --- a/drivers/lightnvm/pblk.h +++ b/drivers/lightnvm/pblk.h @@ -797,6 +797,7 @@ struct nvm_chk_meta *pblk_chunk_get_off(struct pblk *pblk, void pblk_log_write_err(struct pblk *pblk, struct nvm_rq *rqd); void pblk_log_read_err(struct pblk *pblk, struct nvm_rq *rqd); int pblk_submit_io(struct pblk *pblk, struct nvm_rq *rqd); +int pblk_submit_io_sem(struct pblk *pblk, struct nvm_rq *rqd); int pblk_submit_io_sync(struct pblk *pblk, struct nvm_rq *rqd); int pblk_submit_io_sync_sem(struct pblk *pblk, struct nvm_rq *rqd); int pblk_submit_meta_io(struct pblk *pblk, struct pblk_line *meta_line); From patchwork Mon Sep 17 05:29:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiner Litz X-Patchwork-Id: 10602085 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 97637112B for ; Mon, 17 Sep 2018 05:30:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 848292958D for ; Mon, 17 Sep 2018 05:30:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 77B5F295A0; Mon, 17 Sep 2018 05:30:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D8FC82958D for ; Mon, 17 Sep 2018 05:30:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727944AbeIQKzs (ORCPT ); Mon, 17 Sep 2018 06:55:48 -0400 Received: from mail-pf1-f196.google.com ([209.85.210.196]:43542 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727835AbeIQKzr (ORCPT ); Mon, 17 Sep 2018 06:55:47 -0400 Received: by mail-pf1-f196.google.com with SMTP id j26-v6so6977918pfi.10 for ; Sun, 16 Sep 2018 22:30:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=iOw0kYcbNHlnIO77NDIK6HmR6m51SkdCYCWzJ81JgeU=; b=Ti7Ddhn1Rz2cYwKY4ZMrAWr+YQxAh1yGN5ClT5iJJdZF6SOFJCY2sRG8/xO9NeHEpM uK5icxLcToAzpIH54/11kqK9JzPJgWKy8Ykj3vt0DWG1fhZfHlT7PSgrCc73a/Y5ndY2 +l2J73KXb7g1R1nm+CcPtLVgpL6Vytf0P2dco= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=iOw0kYcbNHlnIO77NDIK6HmR6m51SkdCYCWzJ81JgeU=; b=BlN2K3MoKxiiCo8HZobnfdHm2JpyU6b9NprplNHDujxElaszoy733EVrrixdR/v4SC em3SoFODJBUf8nwZn3FCnoegE9KKiFbP+TqIL9K5ssxwFRLqfrq2BU46VQxhy2XZG09C YvbVHqQQy16py1OiklxssYZnYIpWBllvvsHM/qkAhWo1igKxO94D1NGRSC6fhnypT28t uAT9hDt1zO2dyO1tuuGT+oap59YojrRMmhNvpEREdJCKc7ZwzjmU8EYLxZ4o17d7j9xe aW0nDy9itFoh1Eaiis5ZMejovXff68Go7wQM0mSZnqhi40BT+UPReIilu23VE9kpOie2 pTzA== X-Gm-Message-State: APzg51DTPhWLDcgazCBoXAEZ7d+79woeMw0miGCbblZB1uB7SrKjHZq0 MbTSf4Xym62IV70YwqXQqzlWJDm0pJd1Ag== X-Google-Smtp-Source: ANB0VdbjfdnuuJfXTnoIpgNCthoUnZ8gX2mDeWA5MYvz27DaWheciLgvo0SEfWE7+o0WW4KGqNO8FA== X-Received: by 2002:a63:bd41:: with SMTP id d1-v6mr22043583pgp.309.1537162200201; Sun, 16 Sep 2018 22:30:00 -0700 (PDT) Received: from bohr1.soe.ucsc.edu (bohr1.soe.ucsc.edu. [128.114.52.184]) by smtp.gmail.com with ESMTPSA id u11-v6sm22197795pfd.117.2018.09.16.22.29.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 16 Sep 2018 22:29:59 -0700 (PDT) From: Heiner Litz To: linux-block@vger.kernel.org, hlitz@ucsc.edu Cc: javier@cnexlabs.com, mb@lightnvm.io, igor.j.konopko@intel.com, marcin.dziegielewski@intel.com Subject: [RFC PATCH 5/6] lightnvm: pblk: Add RAIL interface Date: Sun, 16 Sep 2018 22:29:38 -0700 Message-Id: <20180917052939.4776-6-hlitz@ucsc.edu> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180917052939.4776-1-hlitz@ucsc.edu> References: <20180917052939.4776-1-hlitz@ucsc.edu> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In prepartion of supporting RAIL, add the RAIL API. Signed-off-by: Heiner Litz --- drivers/lightnvm/pblk-rail.c | 808 +++++++++++++++++++++++++++++++++++ drivers/lightnvm/pblk.h | 63 ++- 2 files changed, 870 insertions(+), 1 deletion(-) create mode 100644 drivers/lightnvm/pblk-rail.c diff --git a/drivers/lightnvm/pblk-rail.c b/drivers/lightnvm/pblk-rail.c new file mode 100644 index 000000000000..a48ed31a0ba9 --- /dev/null +++ b/drivers/lightnvm/pblk-rail.c @@ -0,0 +1,808 @@ +/* + * Copyright (C) 2018 Heiner Litz + * Initial release: Heiner Litz + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License version + * 2 as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + * + * pblk-rail.c - pblk's RAIL path + */ + +#include "pblk.h" + +#define PBLK_RAIL_EMPTY ~0x0 +#define PBLK_RAIL_PARITY_WRITE 0x8000 + +/* RAIL auxiliary functions */ +static unsigned int pblk_rail_nr_parity_luns(struct pblk *pblk) +{ + struct pblk_line_meta *lm = &pblk->lm; + + return lm->blk_per_line / PBLK_RAIL_STRIDE_WIDTH; +} + +static unsigned int pblk_rail_nr_data_luns(struct pblk *pblk) +{ + struct pblk_line_meta *lm = &pblk->lm; + + return lm->blk_per_line - pblk_rail_nr_parity_luns(pblk); +} + +static unsigned int pblk_rail_sec_per_stripe(struct pblk *pblk) +{ + struct pblk_line_meta *lm = &pblk->lm; + + return lm->blk_per_line * pblk->min_write_pgs; +} + +static unsigned int pblk_rail_psec_per_stripe(struct pblk *pblk) +{ + return pblk_rail_nr_parity_luns(pblk) * pblk->min_write_pgs; +} + +static unsigned int pblk_rail_dsec_per_stripe(struct pblk *pblk) +{ + return pblk_rail_sec_per_stripe(pblk) - pblk_rail_psec_per_stripe(pblk); +} + +static unsigned int pblk_rail_wrap_lun(struct pblk *pblk, unsigned int lun) +{ + struct pblk_line_meta *lm = &pblk->lm; + + return (lun & (lm->blk_per_line - 1)); +} + +bool pblk_rail_meta_distance(struct pblk_line *data_line) +{ + return (data_line->meta_distance % PBLK_RAIL_STRIDE_WIDTH) == 0; +} + +/* Notify readers that LUN is serving high latency operation */ +static void pblk_rail_notify_reader_down(struct pblk *pblk, int lun) +{ + WARN_ON(test_and_set_bit(lun, pblk->rail.busy_bitmap)); + /* Make sure that busy bit is seen by reader before proceeding */ + smp_mb__after_atomic(); +} + +static void pblk_rail_notify_reader_up(struct pblk *pblk, int lun) +{ + /* Make sure that write is completed before releasing busy bit */ + smp_mb__before_atomic(); + WARN_ON(!test_and_clear_bit(lun, pblk->rail.busy_bitmap)); +} + +int pblk_rail_lun_busy(struct pblk *pblk, struct ppa_addr ppa) +{ + struct nvm_tgt_dev *dev = pblk->dev; + struct nvm_geo *geo = &dev->geo; + int lun_pos = pblk_ppa_to_pos(geo, ppa); + + return test_bit(lun_pos, pblk->rail.busy_bitmap); +} + +/* Enforces one writer per stride */ +int pblk_rail_down_stride(struct pblk *pblk, int lun_pos, int timeout) +{ + struct pblk_lun *rlun; + int strides = pblk_rail_nr_parity_luns(pblk); + int stride = lun_pos % strides; + int ret; + + rlun = &pblk->luns[stride]; + ret = down_timeout(&rlun->wr_sem, timeout); + pblk_rail_notify_reader_down(pblk, lun_pos); + + return ret; +} + +void pblk_rail_up_stride(struct pblk *pblk, int lun_pos) +{ + struct pblk_lun *rlun; + int strides = pblk_rail_nr_parity_luns(pblk); + int stride = lun_pos % strides; + + pblk_rail_notify_reader_up(pblk, lun_pos); + rlun = &pblk->luns[stride]; + up(&rlun->wr_sem); +} + +/* Determine whether a sector holds data, meta or is bad*/ +bool pblk_rail_valid_sector(struct pblk *pblk, struct pblk_line *line, int pos) +{ + struct pblk_line_meta *lm = &pblk->lm; + struct nvm_tgt_dev *dev = pblk->dev; + struct nvm_geo *geo = &dev->geo; + struct ppa_addr ppa; + int lun; + + if (pos >= line->smeta_ssec && pos < (line->smeta_ssec + lm->smeta_sec)) + return false; + + if (pos >= line->emeta_ssec && + pos < (line->emeta_ssec + lm->emeta_sec[0])) + return false; + + ppa = addr_to_gen_ppa(pblk, pos, line->id); + lun = pblk_ppa_to_pos(geo, ppa); + + return !test_bit(lun, line->blk_bitmap); +} + +/* Delay rb overwrite until whole stride has been written */ +int pblk_rail_rb_delay(struct pblk_rb *rb) +{ + struct pblk *pblk = container_of(rb, struct pblk, rwb); + + return pblk_rail_sec_per_stripe(pblk); +} + +static unsigned int pblk_rail_sec_to_stride(struct pblk *pblk, unsigned int sec) +{ + unsigned int sec_in_stripe = sec % pblk_rail_sec_per_stripe(pblk); + int page = sec_in_stripe / pblk->min_write_pgs; + + return page % pblk_rail_nr_parity_luns(pblk); +} + +static unsigned int pblk_rail_sec_to_idx(struct pblk *pblk, unsigned int sec) +{ + unsigned int sec_in_stripe = sec % pblk_rail_sec_per_stripe(pblk); + + return sec_in_stripe / pblk_rail_psec_per_stripe(pblk); +} + +static void pblk_rail_data_parity(void *dest, void *src) +{ + unsigned int i; + + for (i = 0; i < PBLK_EXPOSED_PAGE_SIZE / sizeof(unsigned long); i++) + ((unsigned long *)dest)[i] ^= ((unsigned long *)src)[i]; +} + +static void pblk_rail_lba_parity(u64 *dest, u64 *src) +{ + *dest ^= *src; +} + +/* Tracks where a sector is located in the rwb */ +void pblk_rail_track_sec(struct pblk *pblk, struct pblk_line *line, int cur_sec, + int sentry, int nr_valid) +{ + int stride, idx, pos; + + stride = pblk_rail_sec_to_stride(pblk, cur_sec); + idx = pblk_rail_sec_to_idx(pblk, cur_sec); + pos = pblk_rb_wrap_pos(&pblk->rwb, sentry); + pblk->rail.p2b[stride][idx].pos = pos; + pblk->rail.p2b[stride][idx].nr_valid = nr_valid; +} + +/* RAIL's sector mapping function */ +static void pblk_rail_map_sec(struct pblk *pblk, struct pblk_line *line, + int sentry, struct pblk_sec_meta *meta_list, + __le64 *lba_list, struct ppa_addr ppa) +{ + struct pblk_w_ctx *w_ctx; + __le64 addr_empty = cpu_to_le64(ADDR_EMPTY); + + kref_get(&line->ref); + + if (sentry & PBLK_RAIL_PARITY_WRITE) { + u64 *lba; + + sentry &= ~PBLK_RAIL_PARITY_WRITE; + lba = &pblk->rail.lba[sentry]; + meta_list->lba = cpu_to_le64(*lba); + *lba_list = cpu_to_le64(*lba); + line->nr_valid_lbas++; + } else { + w_ctx = pblk_rb_w_ctx(&pblk->rwb, sentry); + w_ctx->ppa = ppa; + meta_list->lba = cpu_to_le64(w_ctx->lba); + *lba_list = cpu_to_le64(w_ctx->lba); + + if (*lba_list != addr_empty) + line->nr_valid_lbas++; + else + atomic64_inc(&pblk->pad_wa); + } +} + +int pblk_rail_map_page_data(struct pblk *pblk, unsigned int sentry, + struct ppa_addr *ppa_list, + unsigned long *lun_bitmap, + struct pblk_sec_meta *meta_list, + unsigned int valid_secs) +{ + struct pblk_line *line = pblk_line_get_data(pblk); + struct pblk_emeta *emeta; + __le64 *lba_list; + u64 paddr; + int nr_secs = pblk->min_write_pgs; + int i; + + if (pblk_line_is_full(line)) { + struct pblk_line *prev_line = line; + + /* If we cannot allocate a new line, make sure to store metadata + * on current line and then fail + */ + line = pblk_line_replace_data(pblk); + pblk_line_close_meta(pblk, prev_line); + + if (!line) + return -EINTR; + } + + emeta = line->emeta; + lba_list = emeta_to_lbas(pblk, emeta->buf); + + paddr = pblk_alloc_page(pblk, line, nr_secs); + + pblk_rail_track_sec(pblk, line, paddr, sentry, valid_secs); + + for (i = 0; i < nr_secs; i++, paddr++) { + __le64 addr_empty = cpu_to_le64(ADDR_EMPTY); + + /* ppa to be sent to the device */ + ppa_list[i] = addr_to_gen_ppa(pblk, paddr, line->id); + + /* Write context for target bio completion on write buffer. Note + * that the write buffer is protected by the sync backpointer, + * and a single writer thread have access to each specific entry + * at a time. Thus, it is safe to modify the context for the + * entry we are setting up for submission without taking any + * lock or memory barrier. + */ + if (i < valid_secs) { + pblk_rail_map_sec(pblk, line, sentry + i, &meta_list[i], + &lba_list[paddr], ppa_list[i]); + } else { + lba_list[paddr] = meta_list[i].lba = addr_empty; + __pblk_map_invalidate(pblk, line, paddr); + } + } + + pblk_down_rq(pblk, ppa_list[0], lun_bitmap); + return 0; +} + +/* RAIL Initialization and tear down */ +int pblk_rail_init(struct pblk *pblk) +{ + struct pblk_line_meta *lm = &pblk->lm; + int i, p2be; + unsigned int nr_strides; + unsigned int psecs; + void *kaddr; + + if (!PBLK_RAIL_STRIDE_WIDTH) + return 0; + + if (((lm->blk_per_line % PBLK_RAIL_STRIDE_WIDTH) != 0) || + (lm->blk_per_line < PBLK_RAIL_STRIDE_WIDTH)) { + pr_err("pblk: unsupported RAIL stride %i\n", lm->blk_per_line); + return -EINVAL; + } + + psecs = pblk_rail_psec_per_stripe(pblk); + nr_strides = pblk_rail_sec_per_stripe(pblk) / PBLK_RAIL_STRIDE_WIDTH; + + pblk->rail.p2b = kmalloc_array(nr_strides, sizeof(struct p2b_entry *), + GFP_KERNEL); + if (!pblk->rail.p2b) + return -ENOMEM; + + for (p2be = 0; p2be < nr_strides; p2be++) { + pblk->rail.p2b[p2be] = kmalloc_array(PBLK_RAIL_STRIDE_WIDTH - 1, + sizeof(struct p2b_entry), + GFP_KERNEL); + if (!pblk->rail.p2b[p2be]) + goto free_p2b_entries; + } + + pblk->rail.data = kmalloc(psecs * sizeof(void *), GFP_KERNEL); + if (!pblk->rail.data) + goto free_p2b_entries; + + pblk->rail.pages = alloc_pages(GFP_KERNEL, get_count_order(psecs)); + if (!pblk->rail.pages) + goto free_data; + + kaddr = page_address(pblk->rail.pages); + for (i = 0; i < psecs; i++) + pblk->rail.data[i] = kaddr + i * PBLK_EXPOSED_PAGE_SIZE; + + pblk->rail.lba = kmalloc_array(psecs, sizeof(u64 *), GFP_KERNEL); + if (!pblk->rail.lba) + goto free_pages; + + /* Subtract parity bits from device capacity */ + pblk->capacity = pblk->capacity * (PBLK_RAIL_STRIDE_WIDTH - 1) / + PBLK_RAIL_STRIDE_WIDTH; + + pblk->map_page = pblk_rail_map_page_data; + + return 0; + +free_pages: + free_pages((unsigned long)page_address(pblk->rail.pages), + get_count_order(psecs)); +free_data: + kfree(pblk->rail.data); +free_p2b_entries: + for (p2be = p2be - 1; p2be >= 0; p2be--) + kfree(pblk->rail.p2b[p2be]); + kfree(pblk->rail.p2b); + + return -ENOMEM; +} + +void pblk_rail_free(struct pblk *pblk) +{ + unsigned int i; + unsigned int nr_strides; + unsigned int psecs; + + psecs = pblk_rail_psec_per_stripe(pblk); + nr_strides = pblk_rail_sec_per_stripe(pblk) / PBLK_RAIL_STRIDE_WIDTH; + + kfree(pblk->rail.lba); + free_pages((unsigned long)page_address(pblk->rail.pages), + get_count_order(psecs)); + kfree(pblk->rail.data); + for (i = 0; i < nr_strides; i++) + kfree(pblk->rail.p2b[i]); + kfree(pblk->rail.p2b); +} + +/* PBLK supports 64 ppas max. By performing RAIL reads, a sector is read using + * multiple ppas which can lead to violation of the 64 ppa limit. In this case, + * split the bio + */ +static void pblk_rail_bio_split(struct pblk *pblk, struct bio **bio, int sec) +{ + struct nvm_tgt_dev *dev = pblk->dev; + struct bio *split; + + sec *= (dev->geo.csecs >> 9); + + split = bio_split(*bio, sec, GFP_KERNEL, &pblk_bio_set); + /* there isn't chance to merge the split bio */ + split->bi_opf |= REQ_NOMERGE; + bio_set_flag(*bio, BIO_QUEUE_ENTERED); + bio_chain(split, *bio); + generic_make_request(*bio); + *bio = split; +} + +/* RAIL's Write Path */ +static int pblk_rail_sched_parity(struct pblk *pblk) +{ + struct pblk_line *line = pblk_line_get_data(pblk); + unsigned int sec_in_stripe; + + while (1) { + sec_in_stripe = line->cur_sec % pblk_rail_sec_per_stripe(pblk); + + /* Schedule parity write at end of data section */ + if (sec_in_stripe >= pblk_rail_dsec_per_stripe(pblk)) + return 1; + + /* Skip bad blocks and meta sectors until we find a valid sec */ + if (test_bit(line->cur_sec, line->map_bitmap)) + line->cur_sec += pblk->min_write_pgs; + else + break; + } + + return 0; +} + +/* Mark RAIL parity sectors as invalid sectors so they will be gc'ed */ +void pblk_rail_line_close(struct pblk *pblk, struct pblk_line *line) +{ + int off, bit; + + for (off = pblk_rail_dsec_per_stripe(pblk); + off < pblk->lm.sec_per_line; + off += pblk_rail_sec_per_stripe(pblk)) { + for (bit = 0; bit < pblk_rail_psec_per_stripe(pblk); bit++) + set_bit(off + bit, line->invalid_bitmap); + } +} + +void pblk_rail_end_io_write(struct nvm_rq *rqd) +{ + struct pblk *pblk = rqd->private; + struct pblk_c_ctx *c_ctx = nvm_rq_to_pdu(rqd); + + if (rqd->error) { + pblk_log_write_err(pblk, rqd); + return pblk_end_w_fail(pblk, rqd); + } +#ifdef CONFIG_NVM_DEBUG + else + WARN_ONCE(rqd->bio->bi_status, "pblk: corrupted write error\n"); +#endif + + pblk_up_rq(pblk, c_ctx->lun_bitmap); + + pblk_rq_to_line_put(pblk, rqd); + bio_put(rqd->bio); + pblk_free_rqd(pblk, rqd, PBLK_WRITE); + + atomic_dec(&pblk->inflight_io); +} + +static int pblk_rail_read_to_bio(struct pblk *pblk, struct nvm_rq *rqd, + struct bio *bio, unsigned int stride, + unsigned int nr_secs, unsigned int paddr) +{ + struct pblk_c_ctx *c_ctx = nvm_rq_to_pdu(rqd); + int sec, i; + int nr_data = PBLK_RAIL_STRIDE_WIDTH - 1; + struct pblk_line *line = pblk_line_get_data(pblk); + + c_ctx->nr_valid = nr_secs; + /* sentry indexes rail page buffer, instead of rwb */ + c_ctx->sentry = stride * pblk->min_write_pgs; + c_ctx->sentry |= PBLK_RAIL_PARITY_WRITE; + + for (sec = 0; sec < pblk->min_write_pgs; sec++) { + void *pg_addr; + struct page *page; + u64 *lba; + + lba = &pblk->rail.lba[stride * pblk->min_write_pgs + sec]; + pg_addr = pblk->rail.data[stride * pblk->min_write_pgs + sec]; + page = virt_to_page(pg_addr); + + if (!page) { + pr_err("pblk: could not allocate RAIL bio page %p\n", + pg_addr); + return -NVM_IO_ERR; + } + + if (bio_add_page(bio, page, pblk->rwb.seg_size, 0) != + pblk->rwb.seg_size) { + pr_err("pblk: could not add page to RAIL bio\n"); + return -NVM_IO_ERR; + } + + *lba = 0; + memset(pg_addr, 0, PBLK_EXPOSED_PAGE_SIZE); + + for (i = 0; i < nr_data; i++) { + struct pblk_rb_entry *entry; + struct pblk_w_ctx *w_ctx; + u64 lba_src; + unsigned int pos; + unsigned int cur; + int distance = pblk_rail_psec_per_stripe(pblk); + + cur = paddr - distance * (nr_data - i) + sec; + + if (!pblk_rail_valid_sector(pblk, line, cur)) + continue; + + pos = pblk->rail.p2b[stride][i].pos; + pos = pblk_rb_wrap_pos(&pblk->rwb, pos + sec); + entry = &pblk->rwb.entries[pos]; + w_ctx = &entry->w_ctx; + lba_src = w_ctx->lba; + + if (sec < pblk->rail.p2b[stride][i].nr_valid && + lba_src != ADDR_EMPTY) { + pblk_rail_data_parity(pg_addr, entry->data); + pblk_rail_lba_parity(lba, &lba_src); + } + } + } + + return 0; +} + +int pblk_rail_submit_write(struct pblk *pblk) +{ + int i; + struct nvm_rq *rqd; + struct bio *bio; + struct pblk_line *line = pblk_line_get_data(pblk); + int start, end, bb_offset; + unsigned int stride = 0; + + if (!pblk_rail_sched_parity(pblk)) + return 0; + + start = line->cur_sec; + bb_offset = start % pblk_rail_sec_per_stripe(pblk); + end = start + pblk_rail_sec_per_stripe(pblk) - bb_offset; + + for (i = start; i < end; i += pblk->min_write_pgs, stride++) { + /* Do not generate parity in this slot if the sec is bad + * or reserved for meta. + * We check on the read path and perform a conventional + * read, to avoid reading parity from the bad block + */ + if (!pblk_rail_valid_sector(pblk, line, i)) + continue; + + rqd = pblk_alloc_rqd(pblk, PBLK_WRITE); + if (IS_ERR(rqd)) { + pr_err("pblk: cannot allocate parity write req.\n"); + return -ENOMEM; + } + + bio = bio_alloc(GFP_KERNEL, pblk->min_write_pgs); + if (!bio) { + pr_err("pblk: cannot allocate parity write bio\n"); + pblk_free_rqd(pblk, rqd, PBLK_WRITE); + return -ENOMEM; + } + + bio->bi_iter.bi_sector = 0; /* internal bio */ + bio_set_op_attrs(bio, REQ_OP_WRITE, 0); + rqd->bio = bio; + + pblk_rail_read_to_bio(pblk, rqd, bio, stride, + pblk->min_write_pgs, i); + + if (pblk_submit_io_set(pblk, rqd, pblk_rail_end_io_write)) { + bio_put(rqd->bio); + pblk_free_rqd(pblk, rqd, PBLK_WRITE); + + return -NVM_IO_ERR; + } + } + + return 0; +} + +/* RAIL's Read Path */ +static void pblk_rail_end_io_read(struct nvm_rq *rqd) +{ + struct pblk *pblk = rqd->private; + struct pblk_g_ctx *r_ctx = nvm_rq_to_pdu(rqd); + struct pblk_pr_ctx *pr_ctx = r_ctx->private; + struct bio *new_bio = rqd->bio; + struct bio *bio = pr_ctx->orig_bio; + struct bio_vec src_bv, dst_bv; + struct pblk_sec_meta *meta_list = rqd->meta_list; + int bio_init_idx = pr_ctx->bio_init_idx; + int nr_secs = pr_ctx->orig_nr_secs; + __le64 *lba_list_mem, *lba_list_media; + __le64 addr_empty = cpu_to_le64(ADDR_EMPTY); + void *src_p, *dst_p; + int i, r, rail_ppa = 0; + unsigned char valid; + + if (unlikely(rqd->nr_ppas == 1)) { + struct ppa_addr ppa; + + ppa = rqd->ppa_addr; + rqd->ppa_list = pr_ctx->ppa_ptr; + rqd->dma_ppa_list = pr_ctx->dma_ppa_list; + rqd->ppa_list[0] = ppa; + } + + /* Re-use allocated memory for intermediate lbas */ + lba_list_mem = (((void *)rqd->ppa_list) + pblk_dma_ppa_size); + lba_list_media = (((void *)rqd->ppa_list) + 2 * pblk_dma_ppa_size); + + for (i = 0; i < rqd->nr_ppas; i++) + lba_list_media[i] = meta_list[i].lba; + for (i = 0; i < nr_secs; i++) + meta_list[i].lba = lba_list_mem[i]; + + for (i = 0; i < nr_secs; i++) { + struct pblk_line *line; + u64 meta_lba = 0x0UL, mlba; + + line = pblk_ppa_to_line(pblk, rqd->ppa_list[rail_ppa]); + + valid = bitmap_weight(pr_ctx->bitmap, PBLK_RAIL_STRIDE_WIDTH); + bitmap_shift_right(pr_ctx->bitmap, pr_ctx->bitmap, + PBLK_RAIL_STRIDE_WIDTH, PR_BITMAP_SIZE); + + if (valid == 0) /* Skip cached reads */ + continue; + + kref_put(&line->ref, pblk_line_put); + + dst_bv = bio->bi_io_vec[bio_init_idx + i]; + dst_p = kmap_atomic(dst_bv.bv_page); + + memset(dst_p + dst_bv.bv_offset, 0, PBLK_EXPOSED_PAGE_SIZE); + meta_list[i].lba = cpu_to_le64(0x0UL); + + for (r = 0; r < valid; r++, rail_ppa++) { + src_bv = new_bio->bi_io_vec[rail_ppa]; + + if (lba_list_media[rail_ppa] != addr_empty) { + src_p = kmap_atomic(src_bv.bv_page); + pblk_rail_data_parity(dst_p + dst_bv.bv_offset, + src_p + src_bv.bv_offset); + mlba = le64_to_cpu(lba_list_media[rail_ppa]); + pblk_rail_lba_parity(&meta_lba, &mlba); + kunmap_atomic(src_p); + } + + mempool_free(src_bv.bv_page, &pblk->page_bio_pool); + } + meta_list[i].lba = cpu_to_le64(meta_lba); + kunmap_atomic(dst_p); + } + + bio_put(new_bio); + rqd->nr_ppas = pr_ctx->orig_nr_secs; + kfree(pr_ctx); + rqd->bio = NULL; + + bio_endio(bio); + __pblk_end_io_read(pblk, rqd, false); +} + +/* Converts original ppa into ppa list of RAIL reads */ +static int pblk_rail_setup_ppas(struct pblk *pblk, struct ppa_addr ppa, + struct ppa_addr *rail_ppas, + unsigned char *pvalid, int *nr_rail_ppas, + int *rail_reads) +{ + struct nvm_tgt_dev *dev = pblk->dev; + struct nvm_geo *geo = &dev->geo; + struct ppa_addr rail_ppa = ppa; + unsigned int lun_pos = pblk_ppa_to_pos(geo, ppa); + unsigned int strides = pblk_rail_nr_parity_luns(pblk); + struct pblk_line *line; + unsigned int i; + int ppas = *nr_rail_ppas; + int valid = 0; + + for (i = 1; i < PBLK_RAIL_STRIDE_WIDTH; i++) { + unsigned int neighbor, lun, chnl; + int laddr; + + neighbor = pblk_rail_wrap_lun(pblk, lun_pos + i * strides); + + lun = pblk_pos_to_lun(geo, neighbor); + chnl = pblk_pos_to_chnl(geo, neighbor); + pblk_dev_ppa_set_lun(&rail_ppa, lun); + pblk_dev_ppa_set_chnl(&rail_ppa, chnl); + + line = pblk_ppa_to_line(pblk, rail_ppa); + laddr = pblk_dev_ppa_to_line_addr(pblk, rail_ppa); + + /* Do not read from bad blocks */ + if (!pblk_rail_valid_sector(pblk, line, laddr)) { + /* Perform regular read if parity sector is bad */ + if (neighbor >= pblk_rail_nr_data_luns(pblk)) + return 0; + + /* If any other neighbor is bad we can just skip it */ + continue; + } + + rail_ppas[ppas++] = rail_ppa; + valid++; + } + + if (valid == 1) + return 0; + + *pvalid = valid; + *nr_rail_ppas = ppas; + (*rail_reads)++; + return 1; +} + +static void pblk_rail_set_bitmap(struct pblk *pblk, struct ppa_addr *ppa_list, + int ppa, struct ppa_addr *rail_ppa_list, + int *nr_rail_ppas, unsigned long *read_bitmap, + unsigned long *pvalid, int *rail_reads) +{ + unsigned char valid; + + if (test_bit(ppa, read_bitmap)) + return; + + if (pblk_rail_lun_busy(pblk, ppa_list[ppa]) && + pblk_rail_setup_ppas(pblk, ppa_list[ppa], + rail_ppa_list, &valid, + nr_rail_ppas, rail_reads)) { + WARN_ON(test_and_set_bit(ppa, read_bitmap)); + bitmap_set(pvalid, ppa * PBLK_RAIL_STRIDE_WIDTH, valid); + } else { + rail_ppa_list[(*nr_rail_ppas)++] = ppa_list[ppa]; + bitmap_set(pvalid, ppa * PBLK_RAIL_STRIDE_WIDTH, 1); + } +} + +int pblk_rail_read_bio(struct pblk *pblk, struct nvm_rq *rqd, int blba, + unsigned long *read_bitmap, int bio_init_idx, + struct bio **bio) +{ + struct pblk_g_ctx *r_ctx = nvm_rq_to_pdu(rqd); + struct pblk_pr_ctx *pr_ctx; + struct ppa_addr rail_ppa_list[NVM_MAX_VLBA]; + DECLARE_BITMAP(pvalid, PR_BITMAP_SIZE); + int nr_secs = rqd->nr_ppas; + bool read_empty = bitmap_empty(read_bitmap, nr_secs); + int nr_rail_ppas = 0, rail_reads = 0; + int i; + int ret; + + /* Fully cached reads should not enter this path */ + WARN_ON(bitmap_full(read_bitmap, nr_secs)); + + bitmap_zero(pvalid, PR_BITMAP_SIZE); + if (rqd->nr_ppas == 1) { + pblk_rail_set_bitmap(pblk, &rqd->ppa_addr, 0, rail_ppa_list, + &nr_rail_ppas, read_bitmap, pvalid, + &rail_reads); + + if (nr_rail_ppas == 1) { + memcpy(&rqd->ppa_addr, rail_ppa_list, + nr_rail_ppas * sizeof(struct ppa_addr)); + } else { + rqd->ppa_list = rqd->meta_list + pblk_dma_meta_size; + rqd->dma_ppa_list = rqd->dma_meta_list + + pblk_dma_meta_size; + memcpy(rqd->ppa_list, rail_ppa_list, + nr_rail_ppas * sizeof(struct ppa_addr)); + } + } else { + for (i = 0; i < rqd->nr_ppas; i++) { + pblk_rail_set_bitmap(pblk, rqd->ppa_list, i, + rail_ppa_list, &nr_rail_ppas, + read_bitmap, pvalid, &rail_reads); + + /* Don't split if this it the last ppa of the rqd */ + if (((nr_rail_ppas + PBLK_RAIL_STRIDE_WIDTH) >= + NVM_MAX_VLBA) && (i + 1 < rqd->nr_ppas)) { + struct pblk_g_ctx *r_ctx = nvm_rq_to_pdu(rqd); + + pblk_rail_bio_split(pblk, bio, i + 1); + rqd->nr_ppas = pblk_get_secs(*bio); + r_ctx->private = *bio; + break; + } + } + memcpy(rqd->ppa_list, rail_ppa_list, + nr_rail_ppas * sizeof(struct ppa_addr)); + } + + if (bitmap_empty(read_bitmap, rqd->nr_ppas)) + return NVM_IO_REQUEUE; + + if (read_empty && !bitmap_empty(read_bitmap, rqd->nr_ppas)) + bio_advance(*bio, (rqd->nr_ppas) * PBLK_EXPOSED_PAGE_SIZE); + + if (pblk_setup_partial_read(pblk, rqd, bio_init_idx, read_bitmap, + nr_rail_ppas)) + return NVM_IO_ERR; + + rqd->end_io = pblk_rail_end_io_read; + pr_ctx = r_ctx->private; + bitmap_copy(pr_ctx->bitmap, pvalid, PR_BITMAP_SIZE); + + ret = pblk_submit_io(pblk, rqd); + if (ret) { + bio_put(rqd->bio); + pr_err("pblk: partial RAIL read IO submission failed\n"); + /* Free allocated pages in new bio */ + pblk_bio_free_pages(pblk, rqd->bio, 0, rqd->bio->bi_vcnt); + kfree(pr_ctx); + __pblk_end_io_read(pblk, rqd, false); + return NVM_IO_ERR; + } + + return NVM_IO_OK; +} diff --git a/drivers/lightnvm/pblk.h b/drivers/lightnvm/pblk.h index bd88784e51d9..01fe4362b27e 100644 --- a/drivers/lightnvm/pblk.h +++ b/drivers/lightnvm/pblk.h @@ -28,6 +28,7 @@ #include #include #include +#include #include @@ -45,7 +46,7 @@ #define PBLK_COMMAND_TIMEOUT_MS 30000 /* Max 512 LUNs per device */ -#define PBLK_MAX_LUNS_BITMAP (4) +#define PBLK_MAX_LUNS_BITMAP (512) #define NR_PHY_IN_LOG (PBLK_EXPOSED_PAGE_SIZE / PBLK_SECTOR) @@ -123,6 +124,13 @@ struct pblk_g_ctx { u64 lba; }; +#ifdef CONFIG_NVM_PBLK_RAIL +#define PBLK_RAIL_STRIDE_WIDTH 4 +#define PR_BITMAP_SIZE (NVM_MAX_VLBA * PBLK_RAIL_STRIDE_WIDTH) +#else +#define PR_BITMAP_SIZE NVM_MAX_VLBA +#endif + /* partial read context */ struct pblk_pr_ctx { struct bio *orig_bio; @@ -604,6 +612,39 @@ struct pblk_addrf { int sec_ws_stripe; }; +#ifdef CONFIG_NVM_PBLK_RAIL + +struct p2b_entry { + int pos; + int nr_valid; +}; + +struct pblk_rail { + struct p2b_entry **p2b; /* Maps RAIL sectors to rb pos */ + struct page *pages; /* Pages to hold parity writes */ + void **data; /* Buffer that holds parity pages */ + DECLARE_BITMAP(busy_bitmap, PBLK_MAX_LUNS_BITMAP); + u64 *lba; /* Buffer to compute LBA parity */ +}; + +/* Initialize and tear down RAIL */ +int pblk_rail_init(struct pblk *pblk); +void pblk_rail_free(struct pblk *pblk); +/* Adjust some system parameters */ +bool pblk_rail_meta_distance(struct pblk_line *data_line); +int pblk_rail_rb_delay(struct pblk_rb *rb); +/* Core */ +void pblk_rail_line_close(struct pblk *pblk, struct pblk_line *line); +int pblk_rail_down_stride(struct pblk *pblk, int lun, int timeout); +void pblk_rail_up_stride(struct pblk *pblk, int lun); +/* Write path */ +int pblk_rail_submit_write(struct pblk *pblk); +/* Read Path */ +int pblk_rail_read_bio(struct pblk *pblk, struct nvm_rq *rqd, int blba, + unsigned long *read_bitmap, int bio_init_idx, + struct bio **bio); +#endif /* CONFIG_NVM_PBLK_RAIL */ + typedef int (pblk_map_page_fn)(struct pblk *pblk, unsigned int sentry, struct ppa_addr *ppa_list, unsigned long *lun_bitmap, @@ -1115,6 +1156,26 @@ static inline u64 pblk_dev_ppa_to_line_addr(struct pblk *pblk, return paddr; } +static inline int pblk_pos_to_lun(struct nvm_geo *geo, int pos) +{ + return pos >> ilog2(geo->num_ch); +} + +static inline int pblk_pos_to_chnl(struct nvm_geo *geo, int pos) +{ + return pos % geo->num_ch; +} + +static inline void pblk_dev_ppa_set_lun(struct ppa_addr *p, int lun) +{ + p->a.lun = lun; +} + +static inline void pblk_dev_ppa_set_chnl(struct ppa_addr *p, int chnl) +{ + p->a.ch = chnl; +} + static inline struct ppa_addr pblk_ppa32_to_ppa64(struct pblk *pblk, u32 ppa32) { struct nvm_tgt_dev *dev = pblk->dev; From patchwork Mon Sep 17 05:29:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiner Litz X-Patchwork-Id: 10602087 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B3D86112B for ; Mon, 17 Sep 2018 05:30:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A41712958D for ; Mon, 17 Sep 2018 05:30:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 989F029592; Mon, 17 Sep 2018 05:30:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D8B002958F for ; Mon, 17 Sep 2018 05:30:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728079AbeIQKzu (ORCPT ); Mon, 17 Sep 2018 06:55:50 -0400 Received: from mail-pg1-f196.google.com ([209.85.215.196]:36609 "EHLO mail-pg1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727835AbeIQKzt (ORCPT ); Mon, 17 Sep 2018 06:55:49 -0400 Received: by mail-pg1-f196.google.com with SMTP id d1-v6so7097051pgo.3 for ; Sun, 16 Sep 2018 22:30:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=zChuJFyjjqgVYT0Vsw/wvIMyY54pLxvLz1BpOSRIhQA=; b=dV91q2P8VhWbssJXvJabH22Tn6aTzWqIzYB2p+caTOQRIY8Q4wTdgcrCG5g3/z7H21 JaJKtoqwRXTlLcG3jYmKmibR0kQrK9iPmnH+uguZsVYaNWNqEZtSsgn4kgNkm9cqHJs+ 0xPn/xPwwlqsopCkLhAVxNdTDOFpuPoDBHbDw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=zChuJFyjjqgVYT0Vsw/wvIMyY54pLxvLz1BpOSRIhQA=; b=cd5yOImh+CtZNzYxDANJCo5OOK7Cr4WGHZKutCNWl1wEwP7INroYl26Ik+4kGGx8ZB wmwO/cfptrgcq80YzXbAZZO63aywbufTIgspeqqjE+e4THAWuDBwgfVBy/K2goSAHTuN V3Ld1u8gSnH3DRhu54z4SjEiPRXxRve5eLE/a0OnvwM4kwgT3lbUvf5rf+GxsLS9gRN3 rQbxW9IDDt5+V397nnKA3XLnOLyon3a2Quf+S9KUcpP86TdVW3XNJRdzfEbZIZjhxwb/ VZt4sJAcKkcS/5kSsCD3BfhXUTVeOz3TKkJ+6qhCgJFnz3mECDB+6jOL/KoiAHactK4t PpOA== X-Gm-Message-State: APzg51C8zpJrdc9uYzoFBCHGEuiyBexpox0BzgElPTRrl4uOwQEy8qbt JbAgFZm22Aw5JHAZwZPcUe5Kvq5JzlXzsQ== X-Google-Smtp-Source: ANB0VdaC/vwI5H9l9Vz6Sat0iV5Rp7iWxJfVB1yij0BufwDC8eh8MsszwKybgawUgYDI8OoliwTUuQ== X-Received: by 2002:a63:ac54:: with SMTP id z20-v6mr21237624pgn.74.1537162202415; Sun, 16 Sep 2018 22:30:02 -0700 (PDT) Received: from bohr1.soe.ucsc.edu (bohr1.soe.ucsc.edu. [128.114.52.184]) by smtp.gmail.com with ESMTPSA id u11-v6sm22197795pfd.117.2018.09.16.22.30.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 16 Sep 2018 22:30:01 -0700 (PDT) From: Heiner Litz To: linux-block@vger.kernel.org, hlitz@ucsc.edu Cc: javier@cnexlabs.com, mb@lightnvm.io, igor.j.konopko@intel.com, marcin.dziegielewski@intel.com Subject: [RFC PATCH 6/6] lightnvm: pblk: Integrate RAIL Date: Sun, 16 Sep 2018 22:29:39 -0700 Message-Id: <20180917052939.4776-7-hlitz@ucsc.edu> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180917052939.4776-1-hlitz@ucsc.edu> References: <20180917052939.4776-1-hlitz@ucsc.edu> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Integrate Redundant Array of Independent Luns (RAIL) into lightnvm. RAIL enforces low tail read latency by guaranteeing that reads are never serialized behind writes and erases to the same LUN. Whenever LUNs serve a high latency operation, reads are performed by recomputing the original utilizing redundant parity information. Rail trades-off read latency for capacity (redundancy) which, however, can be leveraged for fault tolerance. On FIO, with the kyber scheduler set to a target read latency of 500us, RAIL reduces tail latency percentiles (us) as follows: Avg 90% 99% 99.9% 99.95% 99.99% pblk 90 1000 2200 3000 6000 RAIL 85 100 250 400 500 Signed-off-by: Heiner Litz --- drivers/lightnvm/Kconfig | 10 ++++++++++ drivers/lightnvm/Makefile | 1 + drivers/lightnvm/pblk-core.c | 36 ++++++++++++++++++++++++++++++++++- drivers/lightnvm/pblk-init.c | 17 +++++++++++++++++ drivers/lightnvm/pblk-rail.c | 1 + drivers/lightnvm/pblk-rb.c | 6 ++++++ drivers/lightnvm/pblk-read.c | 9 +++++++++ drivers/lightnvm/pblk-write.c | 9 +++++++++ drivers/lightnvm/pblk.h | 5 +++++ 9 files changed, 93 insertions(+), 1 deletion(-) diff --git a/drivers/lightnvm/Kconfig b/drivers/lightnvm/Kconfig index a872cd720967..165d5a29acc3 100644 --- a/drivers/lightnvm/Kconfig +++ b/drivers/lightnvm/Kconfig @@ -35,6 +35,16 @@ config NVM_PBLK_DEBUG vocal error messages, and extra tracking fields in the pblk sysfs entries. +config NVM_PBLK_RAIL + bool "Pblk RAIL Support" + default n + help + Enables RAIL for pblk. RAIL enforces tail read latency guarantees by + eliminiating reads being serialized behind writes to the same LUN. + RAIL partitions LUNs into strides and enforces that only one LUN per + stride is written at a time. Reads can bypass busy LUNs by recompting + requested data using parity redundancy. + endif # NVM_PBLK_DEBUG endif # NVM diff --git a/drivers/lightnvm/Makefile b/drivers/lightnvm/Makefile index 97d9d7c71550..92f4376428cc 100644 --- a/drivers/lightnvm/Makefile +++ b/drivers/lightnvm/Makefile @@ -5,6 +5,7 @@ obj-$(CONFIG_NVM) := core.o obj-$(CONFIG_NVM_PBLK) += pblk.o +obj-$(CONFIG_NVM_PBLK_RAIL) += pblk-rail.o pblk-y := pblk-init.o pblk-core.o pblk-rb.o \ pblk-write.o pblk-cache.o pblk-read.o \ pblk-gc.o pblk-recovery.o pblk-map.o \ diff --git a/drivers/lightnvm/pblk-core.c b/drivers/lightnvm/pblk-core.c index a31bf359f905..ca74d7763fa9 100644 --- a/drivers/lightnvm/pblk-core.c +++ b/drivers/lightnvm/pblk-core.c @@ -113,6 +113,12 @@ static void pblk_end_io_erase(struct nvm_rq *rqd) { struct pblk *pblk = rqd->private; +#ifdef CONFIG_NVM_PBLK_RAIL + struct ppa_addr *ppa_list = nvm_rq_to_ppa_list(rqd); + + pblk_up_chunk(pblk, ppa_list[0]); +#endif + __pblk_end_io_erase(pblk, rqd); mempool_free(rqd, &pblk->e_rq_pool); } @@ -940,7 +946,11 @@ static int pblk_blk_erase_sync(struct pblk *pblk, struct ppa_addr ppa) /* The write thread schedules erases so that it minimizes disturbances * with writes. Thus, there is no need to take the LUN semaphore. */ +#ifdef CONFIG_NVM_PBLK_RAIL + ret = pblk_submit_io_sync_sem(pblk, &rqd); +#else ret = pblk_submit_io_sync(pblk, &rqd); +#endif rqd.private = pblk; __pblk_end_io_erase(pblk, &rqd); @@ -1754,7 +1764,11 @@ int pblk_blk_erase_async(struct pblk *pblk, struct ppa_addr ppa) /* The write thread schedules erases so that it minimizes disturbances * with writes. Thus, there is no need to take the LUN semaphore. */ +#ifdef CONFIG_NVM_PBLK_RAIL + err = pblk_submit_io_sem(pblk, rqd); +#else err = pblk_submit_io(pblk, rqd); +#endif if (err) { struct nvm_tgt_dev *dev = pblk->dev; struct nvm_geo *geo = &dev->geo; @@ -1909,6 +1923,10 @@ void pblk_line_close_ws(struct work_struct *work) if (w_err_gc->has_write_err) pblk_save_lba_list(pblk, line); +#ifdef CONFIG_NVM_PBLK_RAIL + pblk_rail_line_close(pblk, line); +#endif + pblk_line_close(pblk, line); mempool_free(line_ws, &pblk->gen_ws_pool); } @@ -1938,8 +1956,12 @@ static void __pblk_down_chunk(struct pblk *pblk, int pos) * Only send one inflight I/O per LUN. Since we map at a page * granurality, all ppas in the I/O will map to the same LUN */ - +#ifdef CONFIG_NVM_PBLK_RAIL + (void)rlun; + ret = pblk_rail_down_stride(pblk, pos, msecs_to_jiffies(30000)); +#else ret = down_timeout(&rlun->wr_sem, msecs_to_jiffies(30000)); +#endif if (ret == -ETIME || ret == -EINTR) pblk_err(pblk, "taking lun semaphore timed out: err %d\n", -ret); @@ -1978,7 +2000,13 @@ void pblk_up_chunk(struct pblk *pblk, struct ppa_addr ppa) int pos = pblk_ppa_to_pos(geo, ppa); rlun = &pblk->luns[pos]; + +#ifdef CONFIG_NVM_PBLK_RAIL + pblk_rail_up_stride(pblk, pos); +#else up(&rlun->wr_sem); +#endif + } void pblk_up_rq(struct pblk *pblk, unsigned long *lun_bitmap) @@ -1991,7 +2019,13 @@ void pblk_up_rq(struct pblk *pblk, unsigned long *lun_bitmap) while ((bit = find_next_bit(lun_bitmap, num_lun, bit + 1)) < num_lun) { rlun = &pblk->luns[bit]; + +#ifdef CONFIG_NVM_PBLK_RAIL + pblk_rail_up_stride(pblk, bit); +#else up(&rlun->wr_sem); +#endif + } } diff --git a/drivers/lightnvm/pblk-init.c b/drivers/lightnvm/pblk-init.c index 2b9c6ebd9fac..3e8255c8873f 100644 --- a/drivers/lightnvm/pblk-init.c +++ b/drivers/lightnvm/pblk-init.c @@ -1050,6 +1050,7 @@ static int pblk_lines_init(struct pblk *pblk) kfree(pblk->lines); fail_free_chunk_meta: kfree(chunk_meta); + fail_free_luns: kfree(pblk->luns); fail_free_meta: @@ -1108,6 +1109,11 @@ static void pblk_tear_down(struct pblk *pblk, bool graceful) __pblk_pipeline_flush(pblk); __pblk_pipeline_stop(pblk); pblk_writer_stop(pblk); + +#ifdef CONFIG_NVM_PBLK_RAIL + pblk_rail_free(pblk); +#endif + pblk_rb_sync_l2p(&pblk->rwb); pblk_rl_free(&pblk->rl); @@ -1226,6 +1232,12 @@ static void *pblk_init(struct nvm_tgt_dev *dev, struct gendisk *tdisk, goto fail_stop_writer; } +#ifdef CONFIG_NVM_PBLK_RAIL + ret = pblk_rail_init(pblk); + if (ret) + goto fail_free_gc; +#endif + /* inherit the size from the underlying device */ blk_queue_logical_block_size(tqueue, queue_physical_block_size(bqueue)); blk_queue_max_hw_sectors(tqueue, queue_max_hw_sectors(bqueue)); @@ -1249,6 +1261,11 @@ static void *pblk_init(struct nvm_tgt_dev *dev, struct gendisk *tdisk, return pblk; +#ifdef CONFIG_NVM_PBLK_RAIL +fail_free_gc: + pblk_gc_exit(pblk, false); +#endif + fail_stop_writer: pblk_writer_stop(pblk); fail_free_l2p: diff --git a/drivers/lightnvm/pblk-rail.c b/drivers/lightnvm/pblk-rail.c index a48ed31a0ba9..619ff9689d29 100644 --- a/drivers/lightnvm/pblk-rail.c +++ b/drivers/lightnvm/pblk-rail.c @@ -1,3 +1,4 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* * Copyright (C) 2018 Heiner Litz * Initial release: Heiner Litz diff --git a/drivers/lightnvm/pblk-rb.c b/drivers/lightnvm/pblk-rb.c index a7648e12f54f..b04462479fe3 100644 --- a/drivers/lightnvm/pblk-rb.c +++ b/drivers/lightnvm/pblk-rb.c @@ -389,8 +389,14 @@ static int __pblk_rb_may_write(struct pblk_rb *rb, unsigned int nr_entries, sync = READ_ONCE(rb->sync); mem = READ_ONCE(rb->mem); +#ifdef CONFIG_NVM_PBLK_RAIL + if (pblk_rb_ring_space(rb, mem, sync, rb->nr_entries) < + nr_entries + pblk_rail_rb_delay(rb)) + return 0; +#else if (pblk_rb_ring_space(rb, mem, sync, rb->nr_entries) < nr_entries) return 0; +#endif if (pblk_rb_update_l2p(rb, nr_entries, mem, sync)) return 0; diff --git a/drivers/lightnvm/pblk-read.c b/drivers/lightnvm/pblk-read.c index 67d44caefff4..a3f33503f60c 100644 --- a/drivers/lightnvm/pblk-read.c +++ b/drivers/lightnvm/pblk-read.c @@ -472,6 +472,15 @@ int pblk_submit_read(struct pblk *pblk, struct bio *bio) return NVM_IO_DONE; } +#ifdef CONFIG_NVM_PBLK_RAIL + ret = pblk_rail_read_bio(pblk, rqd, blba, read_bitmap, bio_init_idx, + &bio); + if (ret == NVM_IO_OK) + return ret; + if (ret == NVM_IO_ERR) + goto fail_end_io; +#endif + /* All sectors are to be read from the device */ if (bitmap_empty(read_bitmap, rqd->nr_ppas)) { struct bio *int_bio = NULL; diff --git a/drivers/lightnvm/pblk-write.c b/drivers/lightnvm/pblk-write.c index 6eba38b83acd..db42184cfba3 100644 --- a/drivers/lightnvm/pblk-write.c +++ b/drivers/lightnvm/pblk-write.c @@ -469,6 +469,11 @@ static inline bool pblk_valid_meta_ppa(struct pblk *pblk, test_bit(pos_opt, data_line->blk_bitmap)) return true; +#ifdef CONFIG_NVM_PBLK_RAIL + if (unlikely(pblk_rail_meta_distance(data_line))) + data_line->meta_distance--; +#endif + if (unlikely(pblk_ppa_comp(ppa_opt, ppa))) data_line->meta_distance--; @@ -571,6 +576,10 @@ static int pblk_submit_write(struct pblk *pblk) unsigned long pos; unsigned int resubmit; +#ifdef CONFIG_NVM_PBLK_RAIL + pblk_rail_submit_write(pblk); +#endif + spin_lock(&pblk->resubmit_lock); resubmit = !list_empty(&pblk->resubmit_list); spin_unlock(&pblk->resubmit_lock); diff --git a/drivers/lightnvm/pblk.h b/drivers/lightnvm/pblk.h index 01fe4362b27e..9742524f74ea 100644 --- a/drivers/lightnvm/pblk.h +++ b/drivers/lightnvm/pblk.h @@ -758,6 +758,11 @@ struct pblk { struct pblk_gc gc; pblk_map_page_fn *map_page; + +#ifdef CONFIG_NVM_PBLK_RAIL + struct pblk_rail rail; +#endif + }; struct pblk_line_ws {