From patchwork Tue Sep 11 18:41:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 10596173 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4406913B8 for ; Tue, 11 Sep 2018 18:43:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 338D4285F2 for ; Tue, 11 Sep 2018 18:43:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 260D02863F; Tue, 11 Sep 2018 18:43:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 531BC28608 for ; Tue, 11 Sep 2018 18:43:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728032AbeIKXmU (ORCPT ); Tue, 11 Sep 2018 19:42:20 -0400 Received: from mail-yw1-f68.google.com ([209.85.161.68]:47079 "EHLO mail-yw1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726863AbeIKXmT (ORCPT ); Tue, 11 Sep 2018 19:42:19 -0400 Received: by mail-yw1-f68.google.com with SMTP id j131-v6so9596490ywc.13; Tue, 11 Sep 2018 11:41:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TOgagscrlT9jC7QBTZg6gKdyqWBEv1N3Mq6bLuVx1Cc=; b=tOANTxBN2ph6yUNSyope2AvZccq5ScvmJBZ4f0VSOirYQkPxnNWaSnmx9+Y2gQUYpm qqnDOgjPQ2iukgzQbH0Q8WcpTi0HLh+F/LIq7fVgh4LX56CK8X7YheFARVo7HA7vqW2g fjUlZr9cqBzd5t7Vs9c0G9Lxq9CqQ3jNTWFy64wM3ZBr20hqdfzwoevv2w5V+/FWVACS IrZ7Xnt6dt3iEhqc+6KtcSqMa4RzgNO8uCRGnjHcULc7wLSjDVDVzqXXk26Par/SHwVi HGToZr2BKyzBBI0aD8kGjkOnrgPqRAeTyq4LeSjGqrUXEbHkteTbtG0UwmvhajW7YINE c+/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=TOgagscrlT9jC7QBTZg6gKdyqWBEv1N3Mq6bLuVx1Cc=; b=Hp8s8zrGM8MHHBJ0uhhU5qeqOJMDqSmGKM4G8g27M4FNArh7mJij0qUoVVBbjtiZJN 8L0G2Ir/cOfoZwOFlQHhVmGZY4iK+20EwiBSBzOgCLFKAKF2ZHeTkPw1Q+lHs3kgPput SKG70g/iN3o0pQeMJLf5cr8rhpcXrjcz9UCyCROEsl0D1A9zC72yrJ8TQUSYHabdfB8T ID4eiuItKijiKgZ0pThxuO/VWIIQ7eHPN6siL/UZC7awhVbfv2MTd9fR8+PQ6QU9GeB8 m5SmKrMQ3CjzaeiIWZ4PSDs58Zy1S2CyyIb0OuB1E7cg+tbbD/LWMw1uIwjpER1YS7vc oC3A== X-Gm-Message-State: APzg51AlKqu0iihUH+zdP77GiS24vnxQwJxL2z3iS/LVXH/A6XKr95Om RpyONo5WxLVyAd77Cqs7UPs= X-Google-Smtp-Source: ANB0VdZGCVFnuVOdZJi7yDj2yFTL3E+d8enKCDVVd2YrFMw+FkXQL0Lj2+LGvQkO5m2NcJ7NqgMUjw== X-Received: by 2002:a0d:e1cb:: with SMTP id k194-v6mr12758161ywe.314.1536691303339; Tue, 11 Sep 2018 11:41:43 -0700 (PDT) Received: from dennisz-mbp.thefacebook.com ([199.201.65.129]) by smtp.gmail.com with ESMTPSA id i123-v6sm7902875ywe.14.2018.09.11.11.41.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 11:41:42 -0700 (PDT) From: Dennis Zhou To: Jens Axboe , Tejun Heo , Johannes Weiner , Josef Bacik Cc: kernel-team@fb.com, linux-block@vger.kernel.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, "Dennis Zhou (Facebook)" Subject: [PATCH 01/12] blkcg: fix ref count issue with bio_blkcg using task_css Date: Tue, 11 Sep 2018 14:41:26 -0400 Message-Id: <20180911184137.35897-2-dennisszhou@gmail.com> X-Mailer: git-send-email 2.13.5 In-Reply-To: <20180911184137.35897-1-dennisszhou@gmail.com> References: <20180911184137.35897-1-dennisszhou@gmail.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: "Dennis Zhou (Facebook)" The accessor function bio_blkcg either returns the blkcg associated with the bio or finds one in the current context. This can cause an issue when trying to associate a bio with a blkcg. Particularly, it's the third case that is problematic: return css_to_blkcg(task_css(current, io_cgrp_id)); As the above may race against task migration and the cgroup exiting, it is not always ok to take a reference on the blkcg returned from bio_blkcg. This patch adds association ahead of calling bio_blkcg rather than after. This makes association a required and explicit step along the code paths for calling bio_blkcg. blk_get_rl is modified as well to get a reference to the blkcg it may use and blk_put_rl will always put the reference back. Association is also moved above the bio_blkcg call to ensure it will not return NULL in blk-iolatency. BFQ and CFQ utilize this flaw, but due to the complexity, I do not want to address this in this series. I've created a private version of the function with notes not to use it describing the flaw. Hopefully soon, that code can be cleaned up. Signed-off-by: Dennis Zhou Acked-by: Tejun Heo --- block/bfq-cgroup.c | 4 +- block/bfq-iosched.c | 2 +- block/bio.c | 10 +++- block/blk-iolatency.c | 2 +- block/cfq-iosched.c | 4 +- include/linux/blk-cgroup.h | 101 ++++++++++++++++++++++++++++++++++--- 6 files changed, 107 insertions(+), 16 deletions(-) diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index 9fe5952d117d..d9a7916ff0ab 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -642,7 +642,7 @@ void bfq_bic_update_cgroup(struct bfq_io_cq *bic, struct bio *bio) uint64_t serial_nr; rcu_read_lock(); - serial_nr = bio_blkcg(bio)->css.serial_nr; + serial_nr = __bio_blkcg(bio)->css.serial_nr; /* * Check whether blkcg has changed. The condition may trigger @@ -651,7 +651,7 @@ void bfq_bic_update_cgroup(struct bfq_io_cq *bic, struct bio *bio) if (unlikely(!bfqd) || likely(bic->blkcg_serial_nr == serial_nr)) goto out; - bfqg = __bfq_bic_change_cgroup(bfqd, bic, bio_blkcg(bio)); + bfqg = __bfq_bic_change_cgroup(bfqd, bic, __bio_blkcg(bio)); /* * Update blkg_path for bfq_log_* functions. We cache this * path, and update it here, for the following diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 653100fb719e..6647355c901c 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -4297,7 +4297,7 @@ static struct bfq_queue *bfq_get_queue(struct bfq_data *bfqd, rcu_read_lock(); - bfqg = bfq_find_set_group(bfqd, bio_blkcg(bio)); + bfqg = bfq_find_set_group(bfqd, __bio_blkcg(bio)); if (!bfqg) { bfqq = &bfqd->oom_bfqq; goto out; diff --git a/block/bio.c b/block/bio.c index f685e762809d..6ca4dda481ca 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1989,13 +1989,19 @@ int bio_associate_blkcg_from_page(struct bio *bio, struct page *page) * * This function takes an extra reference of @blkcg_css which will be put * when @bio is released. The caller must own @bio and is responsible for - * synchronizing calls to this function. + * synchronizing calls to this function. If @blkcg_css is NULL, a call to + * blkcg_get_css finds the current css from the kthread or task. */ int bio_associate_blkcg(struct bio *bio, struct cgroup_subsys_state *blkcg_css) { if (unlikely(bio->bi_css)) return -EBUSY; - css_get(blkcg_css); + + if (blkcg_css) + css_get(blkcg_css); + else + blkcg_css = blkcg_get_css(); + bio->bi_css = blkcg_css; return 0; } diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index 19923f8a029d..62fdd9002c29 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -404,8 +404,8 @@ static void blkcg_iolatency_throttle(struct rq_qos *rqos, struct bio *bio, return; rcu_read_lock(); + bio_associate_blkcg(bio, NULL); blkcg = bio_blkcg(bio); - bio_associate_blkcg(bio, &blkcg->css); blkg = blkg_lookup(blkcg, q); if (unlikely(!blkg)) { if (!lock) diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c index 2eb87444b157..d219e9a1af65 100644 --- a/block/cfq-iosched.c +++ b/block/cfq-iosched.c @@ -3753,7 +3753,7 @@ static void check_blkcg_changed(struct cfq_io_cq *cic, struct bio *bio) uint64_t serial_nr; rcu_read_lock(); - serial_nr = bio_blkcg(bio)->css.serial_nr; + serial_nr = __bio_blkcg(bio)->css.serial_nr; rcu_read_unlock(); /* @@ -3818,7 +3818,7 @@ cfq_get_queue(struct cfq_data *cfqd, bool is_sync, struct cfq_io_cq *cic, struct cfq_group *cfqg; rcu_read_lock(); - cfqg = cfq_lookup_cfqg(cfqd, bio_blkcg(bio)); + cfqg = cfq_lookup_cfqg(cfqd, __bio_blkcg(bio)); if (!cfqg) { cfqq = &cfqd->oom_cfqq; goto out; diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index 6d766a19f2bb..24067a1f8b36 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -230,22 +230,100 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, char *input, struct blkg_conf_ctx *ctx); void blkg_conf_finish(struct blkg_conf_ctx *ctx); +/** + * blkcg_css - find the current css + * + * Find the css associated with either the kthread or the current task. + * This may return a dying css, so it is up to the caller to use tryget logic + * to confirm it is alive and well. + */ +static inline struct cgroup_subsys_state *blkcg_css(void) +{ + struct cgroup_subsys_state *css; + + css = kthread_blkcg(); + if (css) + return css; + return task_css(current, io_cgrp_id); +} + +/** + * blkcg_get_css - find and get a reference to the css + * + * Find the css associated with either the kthread or the current task. + * This takes a reference on the blkcg which will need to be managed by the + * caller. + */ +static inline struct cgroup_subsys_state *blkcg_get_css(void) +{ + struct cgroup_subsys_state *css; + + rcu_read_lock(); + + css = kthread_blkcg(); + if (css) { + css_get(css); + } else { + /* + * This is a bit complicated. It is possible task_css is seeing + * an old css pointer here. This is caused by the current + * thread migrating away from this cgroup and this cgroup dying. + * css_tryget() will fail when trying to take a ref on a cgroup + * that's ref count has hit 0. + * + * Therefore, if it does fail, this means current must have + * been swapped away already and this is waiting for it to + * propagate on the polling cpu. Hence the use of cpu_relax(). + */ + while (true) { + css = task_css(current, io_cgrp_id); + if (likely(css_tryget(css))) + break; + cpu_relax(); + } + } + + rcu_read_unlock(); + + return css; +} static inline struct blkcg *css_to_blkcg(struct cgroup_subsys_state *css) { return css ? container_of(css, struct blkcg, css) : NULL; } -static inline struct blkcg *bio_blkcg(struct bio *bio) +/** + * __bio_blkcg - internal version of bio_blkcg for bfq and cfq + * + * DO NOT USE. + * There is a flaw using this version of the function. In particular, this was + * used in a broken paradigm where association was called on the given css. It + * is possible though that the returned css from task_css() is in the process + * of dying due to migration of the current task. So it is improper to assume + * *_get() is going to succeed. Both BFQ and CFQ rely on this logic and will + * take additional work to handle more gracefully. + */ +static inline struct blkcg *__bio_blkcg(struct bio *bio) { - struct cgroup_subsys_state *css; + if (bio && bio->bi_css) + return css_to_blkcg(bio->bi_css); + return css_to_blkcg(blkcg_css()); +} +/** + * bio_blkcg - grab the blkcg associated with a bio + * @bio: target bio + * + * This returns the blkcg associated with a bio, NULL if not associated. + * Callers are expected to either handle NULL or know association has been + * done prior to calling this. + */ +static inline struct blkcg *bio_blkcg(struct bio *bio) +{ if (bio && bio->bi_css) return css_to_blkcg(bio->bi_css); - css = kthread_blkcg(); - if (css) - return css_to_blkcg(css); - return css_to_blkcg(task_css(current, io_cgrp_id)); + return NULL; } static inline bool blk_cgroup_congested(void) @@ -534,6 +612,10 @@ static inline struct request_list *blk_get_rl(struct request_queue *q, rcu_read_lock(); blkcg = bio_blkcg(bio); + if (blkcg) + css_get(&blkcg->css); + else + blkcg = css_to_blkcg(blkcg_get_css()); /* bypass blkg lookup and use @q->root_rl directly for root */ if (blkcg == &blkcg_root) @@ -565,6 +647,8 @@ static inline struct request_list *blk_get_rl(struct request_queue *q, */ static inline void blk_put_rl(struct request_list *rl) { + /* an additional ref is always taken for rl */ + css_put(&rl->blkg->blkcg->css); if (rl->blkg->blkcg != &blkcg_root) blkg_put(rl->blkg); } @@ -805,10 +889,10 @@ static inline bool blkcg_bio_issue_check(struct request_queue *q, bool throtl = false; rcu_read_lock(); - blkcg = bio_blkcg(bio); /* associate blkcg if bio hasn't attached one */ - bio_associate_blkcg(bio, &blkcg->css); + bio_associate_blkcg(bio, NULL); + blkcg = bio_blkcg(bio); blkg = blkg_lookup(blkcg, q); if (unlikely(!blkg)) { @@ -930,6 +1014,7 @@ static inline int blkcg_activate_policy(struct request_queue *q, static inline void blkcg_deactivate_policy(struct request_queue *q, const struct blkcg_policy *pol) { } +static inline struct blkcg *__bio_blkcg(struct bio *bio) { return NULL; } static inline struct blkcg *bio_blkcg(struct bio *bio) { return NULL; } static inline struct blkg_policy_data *blkg_to_pd(struct blkcg_gq *blkg, From patchwork Tue Sep 11 18:41:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 10596171 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2298613B8 for ; Tue, 11 Sep 2018 18:43:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1448E284E8 for ; Tue, 11 Sep 2018 18:43:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 08887285F2; Tue, 11 Sep 2018 18:43:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 814F5284E8 for ; Tue, 11 Sep 2018 18:43:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728135AbeIKXmV (ORCPT ); Tue, 11 Sep 2018 19:42:21 -0400 Received: from mail-yw1-f68.google.com ([209.85.161.68]:37531 "EHLO mail-yw1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727838AbeIKXmV (ORCPT ); Tue, 11 Sep 2018 19:42:21 -0400 Received: by mail-yw1-f68.google.com with SMTP id x83-v6so9626404ywd.4; Tue, 11 Sep 2018 11:41:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Eiy8jfjFxZHKC78nIxR0T2ImKwTJro98JlpbbKeKMRI=; b=HJ798DPPWplZmJZHKHsOW0+OYclb38wIF2w+ETObMy2xRpsvJJV77fOn7PNoon9lTJ FFQyi197w65ow/gFMbpVIlEYlKl74C9zN9RFNL3LSaSmru9twJVejPcPhoPDRLWjiLAy ZokjYpjiESf+hZyWYkCYFeTj/ZL9RcRfb8vjjqBx5iNoXxKFiPn28DIxejIP+4l+DVQF O+Jh8WB59ucN4WgaNC0f9Y5UUFIVH/U1eJOl7BwwHBFDR0veFhJww4WhUP3X4EusF05q K6TCWZSdiaK1MbJG7ytOSTduro16l74zwAo/lCpyZLSUKP4xStZFt1PPZNuSJ96KAtd7 JEPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Eiy8jfjFxZHKC78nIxR0T2ImKwTJro98JlpbbKeKMRI=; b=KMPigE63GWPScyBaWdinZrIJvZ48p29Y+XUr/VelS1RmeT32lLYX8Dzl7B1sI9RsK2 T8a3tBi1gzbMpoyaTcuhUwAKtazboliYp9J12vTxNtQuRwWhi21yhMbmxT41wheR0Cfh 7rFQ6hYa+SbdL6K8Mh49oWOBIuWMrtVv4V1KQdV2I30c+aUzienni/BUtw2ch48XLR5e BtE7gP60tN4h0SqwSk6y+w5uFltTTtemGH2KNj0GT9t+JB+5V9HjYw57IEk3M8v3QqaL SpRt8IOqpCvOUxBGmLZ6hF8K8j0pGFng/jjV6Rwhzl6RRgB6/C6EDSLVJdR6AAx2h7G0 gcBA== X-Gm-Message-State: APzg51DevicAzoLweGVY2jeviJhNmzx0crKs86lVwKHXVOFGwtKT6ORW Xb/CP9thfRt2gvvlRiXTyds= X-Google-Smtp-Source: ANB0Vdb/7npWEgKReVeB4e7g1GJwW4RbwTE/AjF+rh2C1Bz3DYdqTBn8tutH+a6WpZ1Dt7fQlsolwQ== X-Received: by 2002:a81:7a4b:: with SMTP id v72-v6mr12980077ywc.438.1536691304663; Tue, 11 Sep 2018 11:41:44 -0700 (PDT) Received: from dennisz-mbp.thefacebook.com ([199.201.65.129]) by smtp.gmail.com with ESMTPSA id i123-v6sm7902875ywe.14.2018.09.11.11.41.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 11:41:43 -0700 (PDT) From: Dennis Zhou To: Jens Axboe , Tejun Heo , Johannes Weiner , Josef Bacik Cc: kernel-team@fb.com, linux-block@vger.kernel.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, "Dennis Zhou (Facebook)" Subject: [PATCH 02/12] blkcg: update blkg_lookup_create to do locking Date: Tue, 11 Sep 2018 14:41:27 -0400 Message-Id: <20180911184137.35897-3-dennisszhou@gmail.com> X-Mailer: git-send-email 2.13.5 In-Reply-To: <20180911184137.35897-1-dennisszhou@gmail.com> References: <20180911184137.35897-1-dennisszhou@gmail.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: "Dennis Zhou (Facebook)" To know when to create a blkg, the general pattern is to do a blkg_lookup and if that fails, lock and then do a lookup again and if that fails finally create. It doesn't make much sense for everyone who wants to do creation to write this themselves. This changes blkg_lookup_create to do locking and implement this pattern. The old blkg_lookup_create is renamed to __blkg_lookup_create. If a call site wants to do its own error handling or already owns the queue lock, they can use __blkg_lookup_create. This will be used in upcoming patches. Signed-off-by: Dennis Zhou Reviewed-by: Josef Bacik Acked-by: Tejun Heo Reviewed-by: Liu Bo --- block/blk-cgroup.c | 31 ++++++++++++++++++++++++++++--- block/blk-iolatency.c | 2 +- include/linux/blk-cgroup.h | 4 +++- 3 files changed, 32 insertions(+), 5 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index c19f9078da1e..cd0d97bed83d 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -259,7 +259,7 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, } /** - * blkg_lookup_create - lookup blkg, try to create one if not there + * __blkg_lookup_create - lookup blkg, try to create one if not there * @blkcg: blkcg of interest * @q: request_queue of interest * @@ -272,8 +272,8 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, * value on error. If @q is dead, returns ERR_PTR(-EINVAL). If @q is not * dead and bypassing, returns ERR_PTR(-EBUSY). */ -struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg, - struct request_queue *q) +struct blkcg_gq *__blkg_lookup_create(struct blkcg *blkcg, + struct request_queue *q) { struct blkcg_gq *blkg; @@ -310,6 +310,31 @@ struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg, } } +/** + * blkg_lookup_create - find or create a blkg + * @blkcg: target block cgroup + * @q: target request_queue + * + * This looks up or creates the blkg representing the unique pair + * of the blkcg and the request_queue. + */ +struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg, + struct request_queue *q) +{ + struct blkcg_gq *blkg = blkg_lookup(blkcg, q); + unsigned long flags; + + if (unlikely(!blkg)) { + spin_lock_irqsave(q->queue_lock, flags); + + blkg = __blkg_lookup_create(blkcg, q); + + spin_unlock_irqrestore(q->queue_lock, flags); + } + + return blkg; +} + static void blkg_destroy(struct blkcg_gq *blkg) { struct blkcg *blkcg = blkg->blkcg; diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index 62fdd9002c29..22b2ff0440cc 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -410,7 +410,7 @@ static void blkcg_iolatency_throttle(struct rq_qos *rqos, struct bio *bio, if (unlikely(!blkg)) { if (!lock) spin_lock_irq(q->queue_lock); - blkg = blkg_lookup_create(blkcg, q); + blkg = __blkg_lookup_create(blkcg, q); if (IS_ERR(blkg)) blkg = NULL; if (!lock) diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index 24067a1f8b36..cc0f238530f6 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -184,6 +184,8 @@ extern struct cgroup_subsys_state * const blkcg_root_css; struct blkcg_gq *blkg_lookup_slowpath(struct blkcg *blkcg, struct request_queue *q, bool update_hint); +struct blkcg_gq *__blkg_lookup_create(struct blkcg *blkcg, + struct request_queue *q); struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg, struct request_queue *q); int blkcg_init_queue(struct request_queue *q); @@ -897,7 +899,7 @@ static inline bool blkcg_bio_issue_check(struct request_queue *q, blkg = blkg_lookup(blkcg, q); if (unlikely(!blkg)) { spin_lock_irq(q->queue_lock); - blkg = blkg_lookup_create(blkcg, q); + blkg = __blkg_lookup_create(blkcg, q); if (IS_ERR(blkg)) blkg = NULL; spin_unlock_irq(q->queue_lock); From patchwork Tue Sep 11 18:41:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 10596169 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B878D14F9 for ; Tue, 11 Sep 2018 18:42:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A9B6F284E8 for ; Tue, 11 Sep 2018 18:42:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9D82F285F2; Tue, 11 Sep 2018 18:42:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 196D6284E8 for ; Tue, 11 Sep 2018 18:42:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727835AbeIKXmX (ORCPT ); Tue, 11 Sep 2018 19:42:23 -0400 Received: from mail-yb1-f195.google.com ([209.85.219.195]:46498 "EHLO mail-yb1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728127AbeIKXmW (ORCPT ); Tue, 11 Sep 2018 19:42:22 -0400 Received: by mail-yb1-f195.google.com with SMTP id y20-v6so9720807ybi.13; Tue, 11 Sep 2018 11:41:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0vFvSHzEg8C5aAk8MUn2HD/7B9Z1ZlOCDdmp1KeTvd0=; b=By7vpYN5SSFvh9S236XgvWLLPWN69zPWXl78tdPTVfQLdiOR8NXC1hauzGbN3WhaTG hM4ilN8GhwkkRSk10Lyf6XsJbs91J/BWivdQDesdPsEwS/xtvjHRDDenfGQJUlfefUWZ LGqDl2XlzQ/ZWZC/18n4QIZoAKPKCnWYGeO5fWps1ZYA2BnNFTAbZn0WiNGzxmPB7L0f 8ra+5pBWbqB/oexvBStmAdOv3PeMsiKCD4vuFEofEe4sMB3o4QLWorFZomYZlyLftEZa MUrJArYE8XEjRpi5j9LlfspbFuXU4NdGghZUJZYhyIdcOazTTg8qqgmPa9utze8PYu51 fQOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=0vFvSHzEg8C5aAk8MUn2HD/7B9Z1ZlOCDdmp1KeTvd0=; b=bF3cQcM2RvTtRB/yKTTq5wOVQ1KjjJv4YO1SWO7afnWL895/cZljwlWCRfQDXVAXaC UnKufudVtAsvJpSDlLLhWChJTtlenx1yLoru6Vwfvo1Rlf2XjifIRre9R/q062Oylb/p j6LzpocluESh1Ck615RuKvUngeVGhKQputjr2JwxpKmFMexmalGis73tS6/xMJXMTMsS tmDTjW2ya+diV+o3Ohp37FAPUE9Mnzsk9hzCPdYMMBwpfpE+w6eSlHu1k//eorr5a/Ql k9tNiKgRImk2M0YLwIqXkKWEuAxy6sHH8PTPpd9FBCinNk75vvLeZWwjkp6KOjAKzMPb Nang== X-Gm-Message-State: APzg51DzjJ3vr+8DOQ9KdAqmSieGflTLWshsyHFAgGhuReRSxRw3a6VO q6iXGWTRgGMFMOiR02OHBS0= X-Google-Smtp-Source: ANB0VdYjNjBI2TTPxYw4kaBfXxwBdTkhkYnhfOx7YmrBk+QULCyK4QOZ0z7G8/e+j+tJ0A1YEeNbSg== X-Received: by 2002:a25:9cc1:: with SMTP id z1-v6mr12995474ybo.2.1536691306139; Tue, 11 Sep 2018 11:41:46 -0700 (PDT) Received: from dennisz-mbp.thefacebook.com ([199.201.65.129]) by smtp.gmail.com with ESMTPSA id i123-v6sm7902875ywe.14.2018.09.11.11.41.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 11:41:45 -0700 (PDT) From: Dennis Zhou To: Jens Axboe , Tejun Heo , Johannes Weiner , Josef Bacik Cc: kernel-team@fb.com, linux-block@vger.kernel.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, "Dennis Zhou (Facebook)" Subject: [PATCH 03/12] blkcg: convert blkg_lookup_create to find closest blkg Date: Tue, 11 Sep 2018 14:41:28 -0400 Message-Id: <20180911184137.35897-4-dennisszhou@gmail.com> X-Mailer: git-send-email 2.13.5 In-Reply-To: <20180911184137.35897-1-dennisszhou@gmail.com> References: <20180911184137.35897-1-dennisszhou@gmail.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: "Dennis Zhou (Facebook)" There are several scenarios where blkg_lookup_create can fail. Examples include the blkcg dying, request_queue is dying, or simply being OOM. At the end of the day, most handle this by simply falling back to the q->root_blkg and calling it a day. This patch implements the notion of closest blkg. During blkg_lookup_create, if it fails to create, return the closest blkg found or the q->root_blkg. blkg_try_get_closest is introduced and used during association so a bio is always attached to a blkg. Signed-off-by: Dennis Zhou Acked-by: Tejun Heo --- block/bio.c | 17 ++++++++++------- block/blk-cgroup.c | 25 +++++++++++++++++-------- include/linux/blk-cgroup.h | 14 ++++++++++++++ 3 files changed, 41 insertions(+), 15 deletions(-) diff --git a/block/bio.c b/block/bio.c index 6ca4dda481ca..49a28abd9772 100644 --- a/block/bio.c +++ b/block/bio.c @@ -2008,21 +2008,24 @@ int bio_associate_blkcg(struct bio *bio, struct cgroup_subsys_state *blkcg_css) EXPORT_SYMBOL_GPL(bio_associate_blkcg); /** - * bio_associate_blkg - associate a bio with the specified blkg + * bio_associate_blkg - associate a bio with the a blkg * @bio: target bio * @blkg: the blkg to associate * - * Associate @bio with the blkg specified by @blkg. This is the queue specific - * blkcg information associated with the @bio, a reference will be taken on the - * @blkg and will be freed when the bio is freed. + * This tries to associate @bio with the specified blkg. Association failure + * is handled by walking up the blkg tree. Therefore, the blkg associated can + * be anything between @blkg and the root_blkg. This situation only happens + * when a cgroup is dying and then the remaining bios will spill to the closest + * alive blkg. + * + * A reference will be taken on the @blkg and will be released when @bio is + * freed. */ int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg) { if (unlikely(bio->bi_blkg)) return -EBUSY; - if (!blkg_try_get(blkg)) - return -ENODEV; - bio->bi_blkg = blkg; + bio->bi_blkg = blkg_try_get_closest(blkg); return 0; } diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index cd0d97bed83d..e9e3a955f61a 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -268,9 +268,8 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, * that all non-root blkg's have access to the parent blkg. This function * should be called under RCU read lock and @q->queue_lock. * - * Returns pointer to the looked up or created blkg on success, ERR_PTR() - * value on error. If @q is dead, returns ERR_PTR(-EINVAL). If @q is not - * dead and bypassing, returns ERR_PTR(-EBUSY). + * Returns the blkg or the closest blkg if blkg_create fails as it walks + * down from root. */ struct blkcg_gq *__blkg_lookup_create(struct blkcg *blkcg, struct request_queue *q) @@ -285,7 +284,7 @@ struct blkcg_gq *__blkg_lookup_create(struct blkcg *blkcg, * we shouldn't allow anything to go through for a bypassing queue. */ if (unlikely(blk_queue_bypass(q))) - return ERR_PTR(blk_queue_dying(q) ? -ENODEV : -EBUSY); + return q->root_blkg; blkg = __blkg_lookup(blkcg, q, true); if (blkg) @@ -293,19 +292,29 @@ struct blkcg_gq *__blkg_lookup_create(struct blkcg *blkcg, /* * Create blkgs walking down from blkcg_root to @blkcg, so that all - * non-root blkgs have access to their parents. + * non-root blkgs have access to their parents. Returns the closest + * blkg to the intended blkg should blkg_create() fail. */ while (true) { struct blkcg *pos = blkcg; struct blkcg *parent = blkcg_parent(blkcg); - - while (parent && !__blkg_lookup(parent, q, false)) { + struct blkcg_gq *ret_blkg = q->root_blkg; + + while (parent) { + blkg = __blkg_lookup(parent, q, false); + if (blkg) { + /* remember closest blkg */ + ret_blkg = blkg; + break; + } pos = parent; parent = blkcg_parent(parent); } blkg = blkg_create(pos, q, NULL); - if (pos == blkcg || IS_ERR(blkg)) + if (IS_ERR(blkg)) + return ret_blkg; + if (pos == blkcg) return blkg; } } diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index cc0f238530f6..1fbff1bbb651 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -549,6 +549,20 @@ static inline struct blkcg_gq *blkg_try_get(struct blkcg_gq *blkg) return NULL; } +/** + * blkg_try_get_closest - try and get a blkg ref on the closet blkg + * @blkg: blkg to get + * + * This walks up the blkg tree to find the closest non-dying blkg and returns + * the blkg that it did association with as it may not be the passed in blkg. + */ +static inline struct blkcg_gq *blkg_try_get_closest(struct blkcg_gq *blkg) +{ + while (!atomic_inc_not_zero(&blkg->refcnt)) + blkg = blkg->parent; + + return blkg; +} void __blkg_release_rcu(struct rcu_head *rcu); From patchwork Tue Sep 11 18:41:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 10596167 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B713414F9 for ; Tue, 11 Sep 2018 18:42:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A73E5271CB for ; Tue, 11 Sep 2018 18:42:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 993B4284E8; Tue, 11 Sep 2018 18:42:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0160529C8C for ; Tue, 11 Sep 2018 18:42:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728234AbeIKXmZ (ORCPT ); Tue, 11 Sep 2018 19:42:25 -0400 Received: from mail-yw1-f42.google.com ([209.85.161.42]:35705 "EHLO mail-yw1-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726689AbeIKXmY (ORCPT ); Tue, 11 Sep 2018 19:42:24 -0400 Received: by mail-yw1-f42.google.com with SMTP id 14-v6so9624477ywe.2; Tue, 11 Sep 2018 11:41:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=N0pKhA7/QEQ1I21aBpLNJq0PPd2R4B39KJXaiCvhNQI=; b=qIYBLZ/1tuspipkuIJwiZ2whRfmILbilgJgMRE/IeOEZe1gQMLHiI2NpNYG9c4uv8H FAehYBc9zluSa2+oVQMunhC0bsVfe1EjHfm7ZqZnFxIdMj3aZ7IENB6dLrncEOlMTLlV +wzg29W8vQ9c9m17wyUD2hKpbnwixCOcPf0JrGQsNxsHsZJCEcydW+tvsbrP413+fPFY RVuoZFIcf3bFA+5ohomdSUH1SIa4HbsQIwmMdqF6rSSSGELbI76TQmsoFS2ibTnCK6B6 dyZ/UjCUhTT/FDuqrmRWRAVc5LY20fFF+x5N9JI6E4JKi99garHHlAcDN+QqDSskW+gC Dvgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=N0pKhA7/QEQ1I21aBpLNJq0PPd2R4B39KJXaiCvhNQI=; b=GRk6ms8jnaPxRZUZGqXrIZq8PcLcYtslSXrfLxHm8mGrQf8zhEMW+wZOZPZ87l/nCk 7emr/J9qgikRSxFtRS8ETU6zufEhKPXMyrYtjQUYmPpDjIO/Mjx6gjulC6KgKfKvIyTY R5qed0nbGqXIHuXx2/CMePRF3EvAjQm4iLXSoNQ4iIy2Y539EgV2PXj6Lq46T8sOUXJr qcmDek1Wrl2UGKfPv8EDDKaBvCF5HWo3ur5HCrDSqLrbXGWBHovkb3wpbV/5r/iPB1wG VWlutkwwmYBuKgmM8GLDpFP1HQVUxbvZURg9ycDnPWf2W7mBM+rgPIsM/hasY1ltYwnW lAEA== X-Gm-Message-State: APzg51CHixJ08xZqd7zixw1qbsgHiuu7oTg/Hi5rsK/k/COzSUCtwB2I myuWRly6lxF82U5WnwfGTI8cB9gLPMI= X-Google-Smtp-Source: ANB0Vda8WzUYE15kA4EBL+XLgME903zeTOvtntg31pkePgblKXcwOBrU5qVLWOZZVP8gLdtxJqj3XQ== X-Received: by 2002:a81:c85:: with SMTP id 127-v6mr13278991ywm.364.1536691307359; Tue, 11 Sep 2018 11:41:47 -0700 (PDT) Received: from dennisz-mbp.thefacebook.com ([199.201.65.129]) by smtp.gmail.com with ESMTPSA id i123-v6sm7902875ywe.14.2018.09.11.11.41.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 11:41:46 -0700 (PDT) From: Dennis Zhou To: Jens Axboe , Tejun Heo , Johannes Weiner , Josef Bacik Cc: kernel-team@fb.com, linux-block@vger.kernel.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, "Dennis Zhou (Facebook)" Subject: [PATCH 04/12] blkcg: always associate a bio with a blkg Date: Tue, 11 Sep 2018 14:41:29 -0400 Message-Id: <20180911184137.35897-5-dennisszhou@gmail.com> X-Mailer: git-send-email 2.13.5 In-Reply-To: <20180911184137.35897-1-dennisszhou@gmail.com> References: <20180911184137.35897-1-dennisszhou@gmail.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: "Dennis Zhou (Facebook)" Previously, blkg's were only assigned as needed by blk-iolatency and blk-throttle. bio->css was also always being associated while blkg was being looked up and then thrown away in blkcg_bio_issue_check. This patch begins the cleanup of bio->css and bio->bi_blkg by always associating a blkg in blkcg_bio_issue_check. This tries to create the blkg, but if it is not possible, falls back to using the root_blkg of the request_queue. Therefore, a bio will always be associated with a blkg. The duplicate association logic is removed from blk-throttle and blk-iolatency. Signed-off-by: Dennis Zhou Acked-by: Tejun Heo --- block/bio.c | 38 ++++++++++++++++++++++++++++++++++++++ block/blk-iolatency.c | 24 ++---------------------- block/blk-throttle.c | 5 +---- include/linux/bio.h | 3 +++ include/linux/blk-cgroup.h | 16 ++-------------- 5 files changed, 46 insertions(+), 40 deletions(-) diff --git a/block/bio.c b/block/bio.c index 49a28abd9772..97c1c4bf8df6 100644 --- a/block/bio.c +++ b/block/bio.c @@ -2029,6 +2029,41 @@ int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg) return 0; } +/** + * bio_associate_create_blkg - associate a bio with a blkg from q + * @q: request_queue where bio is going + * @bio: target bio + * + * Associate @bio with the blkg found from the bio's css and the request_queue. + * If one is not found, bio_lookup_blkg creates the blkg. + */ +int bio_associate_create_blkg(struct request_queue *q, struct bio *bio) +{ + struct blkcg *blkcg; + struct blkcg_gq *blkg; + int ret = 0; + + /* someone has already associated this bio with a blkg */ + if (bio->bi_blkg) + return ret; + + rcu_read_lock(); + + bio_associate_blkcg(bio, NULL); + blkcg = bio_blkcg(bio); + + if (!blkcg->css.parent) { + ret = bio_associate_blkg(bio, q->root_blkg); + } else { + blkg = blkg_lookup_create(blkcg, q); + + ret = bio_associate_blkg(bio, blkg); + } + + rcu_read_unlock(); + return ret; +} + /** * bio_disassociate_task - undo bio_associate_current() * @bio: target bio @@ -2058,6 +2093,9 @@ void bio_clone_blkcg_association(struct bio *dst, struct bio *src) { if (src->bi_css) WARN_ON(bio_associate_blkcg(dst, src->bi_css)); + + if (src->bi_blkg) + bio_associate_blkg(dst, src->bi_blkg); } EXPORT_SYMBOL_GPL(bio_clone_blkcg_association); #endif /* CONFIG_BLK_CGROUP */ diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index 22b2ff0440cc..79a7549e2062 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -395,34 +395,14 @@ static void blkcg_iolatency_throttle(struct rq_qos *rqos, struct bio *bio, spinlock_t *lock) { struct blk_iolatency *blkiolat = BLKIOLATENCY(rqos); - struct blkcg *blkcg; - struct blkcg_gq *blkg; - struct request_queue *q = rqos->q; + struct blkcg_gq *blkg = bio->bi_blkg; bool issue_as_root = bio_issue_as_root_blkg(bio); if (!blk_iolatency_enabled(blkiolat)) return; - rcu_read_lock(); - bio_associate_blkcg(bio, NULL); - blkcg = bio_blkcg(bio); - blkg = blkg_lookup(blkcg, q); - if (unlikely(!blkg)) { - if (!lock) - spin_lock_irq(q->queue_lock); - blkg = __blkg_lookup_create(blkcg, q); - if (IS_ERR(blkg)) - blkg = NULL; - if (!lock) - spin_unlock_irq(q->queue_lock); - } - if (!blkg) - goto out; - bio_issue_init(&bio->bi_issue, bio_sectors(bio)); - bio_associate_blkg(bio, blkg); -out: - rcu_read_unlock(); + while (blkg && blkg->parent) { struct iolatency_grp *iolat = blkg_to_lat(blkg); if (!iolat) { diff --git a/block/blk-throttle.c b/block/blk-throttle.c index 01d0620a4e4a..b7b5cc4defc2 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -2129,9 +2129,6 @@ static inline void throtl_update_latency_buckets(struct throtl_data *td) static void blk_throtl_assoc_bio(struct throtl_grp *tg, struct bio *bio) { #ifdef CONFIG_BLK_DEV_THROTTLING_LOW - /* fallback to root_blkg if we fail to get a blkg ref */ - if (bio->bi_css && (bio_associate_blkg(bio, tg_to_blkg(tg)) == -ENODEV)) - bio_associate_blkg(bio, bio->bi_disk->queue->root_blkg); bio_issue_init(&bio->bi_issue, bio_sectors(bio)); #endif } @@ -2140,7 +2137,7 @@ bool blk_throtl_bio(struct request_queue *q, struct blkcg_gq *blkg, struct bio *bio) { struct throtl_qnode *qn = NULL; - struct throtl_grp *tg = blkg_to_tg(blkg ?: q->root_blkg); + struct throtl_grp *tg = blkg_to_tg(blkg); struct throtl_service_queue *sq; bool rw = bio_data_dir(bio); bool throttled = false; diff --git a/include/linux/bio.h b/include/linux/bio.h index 14b4fa266357..829cd0bb407d 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -542,11 +542,14 @@ static inline int bio_associate_blkcg_from_page(struct bio *bio, #ifdef CONFIG_BLK_CGROUP int bio_associate_blkcg(struct bio *bio, struct cgroup_subsys_state *blkcg_css); int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg); +int bio_associate_create_blkg(struct request_queue *q, struct bio *bio); void bio_disassociate_task(struct bio *bio); void bio_clone_blkcg_association(struct bio *dst, struct bio *src); #else /* CONFIG_BLK_CGROUP */ static inline int bio_associate_blkcg(struct bio *bio, struct cgroup_subsys_state *blkcg_css) { return 0; } +static inline int bio_associate_create_blkg(struct request_queue *q, + struct bio *bio) { return 0; } static inline void bio_disassociate_task(struct bio *bio) { } static inline void bio_clone_blkcg_association(struct bio *dst, struct bio *src) { } diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index 1fbff1bbb651..6e33ad1d92b4 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -900,29 +900,17 @@ static inline bool blk_throtl_bio(struct request_queue *q, struct blkcg_gq *blkg static inline bool blkcg_bio_issue_check(struct request_queue *q, struct bio *bio) { - struct blkcg *blkcg; struct blkcg_gq *blkg; bool throtl = false; rcu_read_lock(); - /* associate blkcg if bio hasn't attached one */ - bio_associate_blkcg(bio, NULL); - blkcg = bio_blkcg(bio); - - blkg = blkg_lookup(blkcg, q); - if (unlikely(!blkg)) { - spin_lock_irq(q->queue_lock); - blkg = __blkg_lookup_create(blkcg, q); - if (IS_ERR(blkg)) - blkg = NULL; - spin_unlock_irq(q->queue_lock); - } + bio_associate_create_blkg(q, bio); + blkg = bio->bi_blkg; throtl = blk_throtl_bio(q, blkg, bio); if (!throtl) { - blkg = blkg ?: q->root_blkg; /* * If the bio is flagged with BIO_QUEUE_ENTERED it means this * is a split bio and we would have already accounted for the From patchwork Tue Sep 11 18:41:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 10596151 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4A93E14F9 for ; Tue, 11 Sep 2018 18:41:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3B69029CD1 for ; Tue, 11 Sep 2018 18:41:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2EF9029CE9; Tue, 11 Sep 2018 18:41:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9FC2529CD1 for ; Tue, 11 Sep 2018 18:41:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727760AbeIKXmZ (ORCPT ); Tue, 11 Sep 2018 19:42:25 -0400 Received: from mail-yw1-f65.google.com ([209.85.161.65]:39310 "EHLO mail-yw1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726980AbeIKXmY (ORCPT ); Tue, 11 Sep 2018 19:42:24 -0400 Received: by mail-yw1-f65.google.com with SMTP id m62-v6so9618264ywd.6; Tue, 11 Sep 2018 11:41:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GI3MCHL3A7+L+J3rxJiRbY56/62rtOZb0piAM7a57EY=; b=Bj66zp2gN3+rTR3LC8usrQseah6hbuFRqF+Fl2FtJ+bgWtoazQ18dVYdThuOukpu1L WeVYsZFmKa7ib+wHOAD96GJekDCdpd+6jQ6Q8C4H7Jz2/0EyfWdOB5ZdRgNHmcw8Ywhi 6p/SVEoEjNaRyPjr9TGINo6r1oS0/Z9BVFgbj5mR0y2vayC0YY+RNJ4gIFQ5AEQJCyhL IZNCwu4Q8Kb/JzoYEj4UGXnQtFt+KlB7JoXR2e5X3CARfHggQmXhrVCpcwxcfMRWsP7q cRyovgt6rHGGIf3iv/wVz2p16sRZtPalNdQP7opzoqwKjWRpjVKLTalEZu4wlPhpVb1F I2yA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GI3MCHL3A7+L+J3rxJiRbY56/62rtOZb0piAM7a57EY=; b=LmUiW9u2z/lvoRlsEa7E30pjcuboHCHG6+KF2L0KinR6ThlZ/C4ScUu6v8yrcoL4TX l/wKzWh889KSWnyiAIMPP4LTXOy8sKr1rrjVP1KJtoybdDgVXicHnV8Z7BKdSqKcM8pK Jsv47reQ+xv4iwVIcEQ/QkEVJY+CqcYJZ9sVMeGWmNUpAC/0CGh7VRTdRSdXXPARw0Bv WCoP4755kTDXtx8ygIMVDQRkcRTk0MYebxiC71NT6a9HQyAJfGgwDaGD1zhYMZcdoWik oViNZvjvWWHKPlHoKG5ay/Tmz+R2vvQrLeYb+vtxWW8/rs0I+AZvOjvcvtlZl3E03Q2R CqNQ== X-Gm-Message-State: APzg51Bad1DFGpNAXpIErxC9if158hh/wRKND0TGkbkE1PMF9sEsHW+A ZFvLFQ1PhKtgPROZDgowh9Y= X-Google-Smtp-Source: ANB0VdZQcqmbwFvv5QEA+wzFUFpts2c87FGwv0D6lbZS2JXiBBbLdAv5id+9C9uwX7ZiDPmmGX1big== X-Received: by 2002:a81:f95:: with SMTP id 143-v6mr12540403ywp.91.1536691308587; Tue, 11 Sep 2018 11:41:48 -0700 (PDT) Received: from dennisz-mbp.thefacebook.com ([199.201.65.129]) by smtp.gmail.com with ESMTPSA id i123-v6sm7902875ywe.14.2018.09.11.11.41.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 11:41:47 -0700 (PDT) From: Dennis Zhou To: Jens Axboe , Tejun Heo , Johannes Weiner , Josef Bacik Cc: kernel-team@fb.com, linux-block@vger.kernel.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, "Dennis Zhou (Facebook)" Subject: [PATCH 05/12] blkcg: consolidate bio_issue_init to be a part of core Date: Tue, 11 Sep 2018 14:41:30 -0400 Message-Id: <20180911184137.35897-6-dennisszhou@gmail.com> X-Mailer: git-send-email 2.13.5 In-Reply-To: <20180911184137.35897-1-dennisszhou@gmail.com> References: <20180911184137.35897-1-dennisszhou@gmail.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: "Dennis Zhou (Facebook)" bio_issue_init among other things initializes the timestamp for an IO. Rather than have this logic handled by policies, this consolidates it to be on the init paths (normal, clone, bounce clone). Signed-off-by: Dennis Zhou Acked-by: Tejun Heo Reviewed-by: Liu Bo --- block/bio.c | 2 ++ block/blk-iolatency.c | 2 -- block/blk-throttle.c | 8 -------- block/bounce.c | 2 ++ include/linux/blk-cgroup.h | 9 +++++++++ 5 files changed, 13 insertions(+), 10 deletions(-) diff --git a/block/bio.c b/block/bio.c index 97c1c4bf8df6..1d28d7ad2342 100644 --- a/block/bio.c +++ b/block/bio.c @@ -610,6 +610,8 @@ void __bio_clone_fast(struct bio *bio, struct bio *bio_src) bio->bi_io_vec = bio_src->bi_io_vec; bio_clone_blkcg_association(bio, bio_src); + + blkcg_bio_issue_init(bio); } EXPORT_SYMBOL(__bio_clone_fast); diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index 79a7549e2062..9d7052bad6f7 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -401,8 +401,6 @@ static void blkcg_iolatency_throttle(struct rq_qos *rqos, struct bio *bio, if (!blk_iolatency_enabled(blkiolat)) return; - bio_issue_init(&bio->bi_issue, bio_sectors(bio)); - while (blkg && blkg->parent) { struct iolatency_grp *iolat = blkg_to_lat(blkg); if (!iolat) { diff --git a/block/blk-throttle.c b/block/blk-throttle.c index b7b5cc4defc2..f2b355338894 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -2126,13 +2126,6 @@ static inline void throtl_update_latency_buckets(struct throtl_data *td) } #endif -static void blk_throtl_assoc_bio(struct throtl_grp *tg, struct bio *bio) -{ -#ifdef CONFIG_BLK_DEV_THROTTLING_LOW - bio_issue_init(&bio->bi_issue, bio_sectors(bio)); -#endif -} - bool blk_throtl_bio(struct request_queue *q, struct blkcg_gq *blkg, struct bio *bio) { @@ -2156,7 +2149,6 @@ bool blk_throtl_bio(struct request_queue *q, struct blkcg_gq *blkg, if (unlikely(blk_queue_bypass(q))) goto out_unlock; - blk_throtl_assoc_bio(tg, bio); blk_throtl_update_idletime(tg); sq = &tg->service_queue; diff --git a/block/bounce.c b/block/bounce.c index bc63b3a2d18c..7a08703b1204 100644 --- a/block/bounce.c +++ b/block/bounce.c @@ -259,6 +259,8 @@ static struct bio *bounce_clone_bio(struct bio *bio_src, gfp_t gfp_mask, bio_clone_blkcg_association(bio, bio_src); + blkcg_bio_issue_init(bio); + return bio; } diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index 6e33ad1d92b4..a6b6e741a75e 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -897,6 +897,12 @@ static inline bool blk_throtl_bio(struct request_queue *q, struct blkcg_gq *blkg struct bio *bio) { return false; } #endif + +static inline void blkcg_bio_issue_init(struct bio *bio) +{ + bio_issue_init(&bio->bi_issue, bio_sectors(bio)); +} + static inline bool blkcg_bio_issue_check(struct request_queue *q, struct bio *bio) { @@ -922,6 +928,8 @@ static inline bool blkcg_bio_issue_check(struct request_queue *q, blkg_rwstat_add(&blkg->stat_ios, bio->bi_opf, 1); } + blkcg_bio_issue_init(bio); + rcu_read_unlock(); return !throtl; } @@ -1034,6 +1042,7 @@ static inline void blk_put_rl(struct request_list *rl) { } static inline void blk_rq_set_rl(struct request *rq, struct request_list *rl) { } static inline struct request_list *blk_rq_rl(struct request *rq) { return &rq->q->root_rl; } +static inline void blkcg_bio_issue_init(struct bio *bio) { } static inline bool blkcg_bio_issue_check(struct request_queue *q, struct bio *bio) { return true; } From patchwork Tue Sep 11 18:41:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 10596165 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ECDE513B8 for ; Tue, 11 Sep 2018 18:42:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DD70129CBA for ; Tue, 11 Sep 2018 18:42:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D20A929CC4; Tue, 11 Sep 2018 18:42:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4AC7229CBA for ; Tue, 11 Sep 2018 18:42:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728477AbeIKXnU (ORCPT ); Tue, 11 Sep 2018 19:43:20 -0400 Received: from mail-yb1-f169.google.com ([209.85.219.169]:37300 "EHLO mail-yb1-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727413AbeIKXmZ (ORCPT ); Tue, 11 Sep 2018 19:42:25 -0400 Received: by mail-yb1-f169.google.com with SMTP id f145-v6so9731542ybg.4; Tue, 11 Sep 2018 11:41:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=wxEqMCKwjjKDgqsi2Aa/uNaLSYyWsKKaIQb8WIwFgtw=; b=vUYAewH7ZEmPcMGBMl4qTvyNbL8xLHId0YuiBzMs1dGoIRoAqw5SqDK51+j3O7HfEH J7Ul9kqlzE9AJ3tgMNzaEgVALdpfYoTPQcLwh0jq2vq3FhkaIPmc6uyD6j81H92MnD33 xmLz9j9VZJKJeDD1l9XJxTbWrIbZ6ZrplKu+Rr0/+h8lkqNWG8tsE85o7cmUHqSpUnsQ 4Z7MzxBDRbWgU2hymPl6U7jxLvAQNMqjLomw1edZaXtjCNvRw0lrep/Xy+s3tVyA+bNG D/DAApaFbFncIEc3Bx/pcuhmRWlLTj2/w2aYjqthid4cUAYx+hqIWk1BIFrHJwp+BAyj Ff7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=wxEqMCKwjjKDgqsi2Aa/uNaLSYyWsKKaIQb8WIwFgtw=; b=EVmQPF5PO1sW6z/CtXSp68XvUwS5emacDhyZI8p8e5OvOzXfUQ7UYJUNIqr4/TZJRp NwOiy6/AjF1aiMP4aMSsmH7ll9bTdI+o35I1VTHbfw7+5/EVp7rqkNIYcwknuc+o5ilu G0F6SboPB2pGja2sc2d9m0tACP14Dit9P5cY+XimTNYDN/QOpWV+QdN8UZDLM70JqeUg 3Yw+yfPMrgRoCPmR9mXqBEQMMNtrGCDun6X8zKRD+/rpMVforDBW54rCVxPQhYgbAVcX A8wzaFGww8q5ehRMIVyKTqlIfamPoFn3giBC/w7Wa9xYPbY0k87maL/4N9DrCXzoHi7I oHUg== X-Gm-Message-State: APzg51D/vVIBeD1k+MLXZiq+kOsBocxTrGa6TswEgCepqVsD+27x8SXw 8QMUGsbvMAQjXXPWJAxOENU= X-Google-Smtp-Source: ANB0VdY0tqdgpSu3HPtCDngUPBSSlsETMQgYvgEQg2kPo4gI5Kww+xxveKlCywE72r7BQuyX2YlMpg== X-Received: by 2002:a25:2c03:: with SMTP id s3-v6mr13061628ybs.378.1536691309818; Tue, 11 Sep 2018 11:41:49 -0700 (PDT) Received: from dennisz-mbp.thefacebook.com ([199.201.65.129]) by smtp.gmail.com with ESMTPSA id i123-v6sm7902875ywe.14.2018.09.11.11.41.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 11:41:49 -0700 (PDT) From: Dennis Zhou To: Jens Axboe , Tejun Heo , Johannes Weiner , Josef Bacik Cc: kernel-team@fb.com, linux-block@vger.kernel.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, "Dennis Zhou (Facebook)" Subject: [PATCH 06/12] blkcg: associate a blkg for pages being evicted by swap Date: Tue, 11 Sep 2018 14:41:31 -0400 Message-Id: <20180911184137.35897-7-dennisszhou@gmail.com> X-Mailer: git-send-email 2.13.5 In-Reply-To: <20180911184137.35897-1-dennisszhou@gmail.com> References: <20180911184137.35897-1-dennisszhou@gmail.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: "Dennis Zhou (Facebook)" A prior patch in this series added blkg association to bios issued by cgroups. There are two other paths that we want to attribute work back to the appropriate cgroup: swap and writeback. Here we modify the way swap tags bios to include the blkg. Writeback will be tackle in the next patch. Signed-off-by: Dennis Zhou Reviewed-by: Josef Bacik Acked-by: Tejun Heo --- block/bio.c | 83 ++++++++++++++++++++++++++++++++------------- include/linux/bio.h | 11 ++++-- mm/page_io.c | 2 +- 3 files changed, 68 insertions(+), 28 deletions(-) diff --git a/block/bio.c b/block/bio.c index 1d28d7ad2342..897c8ed9e572 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1957,30 +1957,6 @@ EXPORT_SYMBOL(bioset_init_from_src); #ifdef CONFIG_BLK_CGROUP -#ifdef CONFIG_MEMCG -/** - * bio_associate_blkcg_from_page - associate a bio with the page's blkcg - * @bio: target bio - * @page: the page to lookup the blkcg from - * - * Associate @bio with the blkcg from @page's owning memcg. This works like - * every other associate function wrt references. - */ -int bio_associate_blkcg_from_page(struct bio *bio, struct page *page) -{ - struct cgroup_subsys_state *blkcg_css; - - if (unlikely(bio->bi_css)) - return -EBUSY; - if (!page->mem_cgroup) - return 0; - blkcg_css = cgroup_get_e_css(page->mem_cgroup->css.cgroup, - &io_cgrp_subsys); - bio->bi_css = blkcg_css; - return 0; -} -#endif /* CONFIG_MEMCG */ - /** * bio_associate_blkcg - associate a bio with the specified blkcg * @bio: target bio @@ -2031,6 +2007,65 @@ int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg) return 0; } +static int __bio_associate_blkg_from_css(struct bio *bio, + struct cgroup_subsys_state *css) +{ + struct blkcg_gq *blkg; + + rcu_read_lock(); + + blkg = blkg_lookup_create(css_to_blkcg(css), bio->bi_disk->queue); + + rcu_read_unlock(); + + return bio_associate_blkg(bio, blkg); +} + +/** + * bio_associate_blkg_from_css - associate a bio with a specified css + * @bio: target bio + * @css: target css + * + * Associate @bio with the blkg found by combining the css's blkg and the + * request_queue of the @bio. This takes a reference on the css that will + * be put upon freeing of @bio. + */ +int bio_associate_blkg_from_css(struct bio *bio, + struct cgroup_subsys_state *css) +{ + css_get(css); + bio->bi_css = css; + return __bio_associate_blkg_from_css(bio, css); +} +EXPORT_SYMBOL_GPL(bio_associate_blkg_from_css); + +#ifdef CONFIG_MEMCG +/** + * bio_associate_blkg_from_page - associate a bio with the page's blkg + * @bio: target bio + * @page: the page to lookup the blkcg from + * + * Associate @bio with the blkg from @page's owning memcg and the respective + * request_queue. This works like every other associate function wrt + * references. + * + * Note: this must be called after bio has an associated device. + */ +int bio_associate_blkg_from_page(struct bio *bio, struct page *page) +{ + struct cgroup_subsys_state *css; + + if (unlikely(bio->bi_css)) + return -EBUSY; + if (!page->mem_cgroup) + return 0; + css = cgroup_get_e_css(page->mem_cgroup->css.cgroup, &io_cgrp_subsys); + bio->bi_css = css; + + return __bio_associate_blkg_from_css(bio, css); +} +#endif /* CONFIG_MEMCG */ + /** * bio_associate_create_blkg - associate a bio with a blkg from q * @q: request_queue where bio is going diff --git a/include/linux/bio.h b/include/linux/bio.h index 829cd0bb407d..c73a870ebc0e 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -533,21 +533,26 @@ do { \ disk_devt((bio)->bi_disk) #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP) -int bio_associate_blkcg_from_page(struct bio *bio, struct page *page); +int bio_associate_blkg_from_page(struct bio *bio, struct page *page); #else -static inline int bio_associate_blkcg_from_page(struct bio *bio, - struct page *page) { return 0; } +static inline int bio_associate_blkg_from_page(struct bio *bio, + struct page *page) { return 0; } #endif #ifdef CONFIG_BLK_CGROUP int bio_associate_blkcg(struct bio *bio, struct cgroup_subsys_state *blkcg_css); int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg); +int bio_associate_blkg_from_css(struct bio *bio, + struct cgroup_subsys_state *css); int bio_associate_create_blkg(struct request_queue *q, struct bio *bio); void bio_disassociate_task(struct bio *bio); void bio_clone_blkcg_association(struct bio *dst, struct bio *src); #else /* CONFIG_BLK_CGROUP */ static inline int bio_associate_blkcg(struct bio *bio, struct cgroup_subsys_state *blkcg_css) { return 0; } +static inline int bio_associate_blkg_from_css(struct bio *bio, + struct cgroup_subsys_state *css) +{ return 0; } static inline int bio_associate_create_blkg(struct request_queue *q, struct bio *bio) { return 0; } static inline void bio_disassociate_task(struct bio *bio) { } diff --git a/mm/page_io.c b/mm/page_io.c index aafd19ec1db4..573d3663d846 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -339,7 +339,7 @@ int __swap_writepage(struct page *page, struct writeback_control *wbc, goto out; } bio->bi_opf = REQ_OP_WRITE | REQ_SWAP | wbc_to_write_flags(wbc); - bio_associate_blkcg_from_page(bio, page); + bio_associate_blkg_from_page(bio, page); count_swpout_vm_event(page); set_page_writeback(page); unlock_page(page); From patchwork Tue Sep 11 18:41:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 10596161 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A959C14F9 for ; Tue, 11 Sep 2018 18:42:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9A92329CA8 for ; Tue, 11 Sep 2018 18:42:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8F14C29CBA; Tue, 11 Sep 2018 18:42:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1160129CA9 for ; Tue, 11 Sep 2018 18:42:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727246AbeIKXm2 (ORCPT ); Tue, 11 Sep 2018 19:42:28 -0400 Received: from mail-yb1-f193.google.com ([209.85.219.193]:33176 "EHLO mail-yb1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726622AbeIKXm1 (ORCPT ); Tue, 11 Sep 2018 19:42:27 -0400 Received: by mail-yb1-f193.google.com with SMTP id m123-v6so9741880ybm.0; Tue, 11 Sep 2018 11:41:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FkDgmX6YNgFCNFUn3Et8v5b6B07m4GOk66oKL1F2CRg=; b=mGygZNE6R8vykkRJpASf0GghNDi/HMMohnHWuxa5UMirclwqDK6yWQFiht/cECSYxa Q5RGNy0B6E4DoS01BWEZoT7RVb3FVqaq+u4qS6MiWnOt2/utnbtTYotziTsNJAlx4f2/ 4yBS8HuTEAuTkdUzi7ZW1aTgR7Q/QYxq0eDbUdSqd6PMN/l3fliYR534AyjF1iSBbRPp l9A6TJxM0c4Vym3fWG9vFcOTce5dI9EKiGi0qgeX6L+UZIUc+nEUgwPNiFbgx0r2ZAJF LmlvIM92ZsElKIxZFBirxLyvzJ9LPu/z6u8YfEGm6XSPd++52sv1rxvAJKru7bCRx0jz jflg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=FkDgmX6YNgFCNFUn3Et8v5b6B07m4GOk66oKL1F2CRg=; b=MM1zXbPV+mPXDA/w64KJE3iHPT3k0n3JDmKkGQHn+FB2lyOWI28LXxY4CEcr6NyPmj +5QvnFj/9vDSAtfRBxZuEQ1nwb9YBszE6Z8T8YPwrMsA1s29NkX9BrpwqwYOWGT8IqO5 Sc1RtTC4yvehfzD1NZcsInwr9zM0s02Q5ij4cdfRSk01Ipgx9BU/SWDl476qeJy/1O6n BMGGzY08Xhzxi+EujniSTKDRaJVn5gwVIqc+xWm8BYWNaFTqKUadY7LK6HLg+DSX4CtD EPylIpKvMTQZhSdD1znmvHTHUMQKHRG8BVXA+ldzAAsMve7FyAAHkk4mQvg6v7NdQHx0 IWhw== X-Gm-Message-State: APzg51DN7zma1UwJYU5hw/TZw2d2D39scKb3YQMqRE7GSQZajGyw/Dgs z2gmaviM+kAqvP0y4ycWiPThE7igpTY= X-Google-Smtp-Source: ANB0VdZFMjBueQsW+LhHFGB094Me3z5a1gtZiu+BfO4jswBUfOn8HHV7/uXsQHFIUkaJXJJ1DHNtBw== X-Received: by 2002:a25:220a:: with SMTP id i10-v6mr12983355ybi.489.1536691311120; Tue, 11 Sep 2018 11:41:51 -0700 (PDT) Received: from dennisz-mbp.thefacebook.com ([199.201.65.129]) by smtp.gmail.com with ESMTPSA id i123-v6sm7902875ywe.14.2018.09.11.11.41.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 11:41:50 -0700 (PDT) From: Dennis Zhou To: Jens Axboe , Tejun Heo , Johannes Weiner , Josef Bacik Cc: kernel-team@fb.com, linux-block@vger.kernel.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, "Dennis Zhou (Facebook)" Subject: [PATCH 07/12] blkcg: associate writeback bios with a blkg Date: Tue, 11 Sep 2018 14:41:32 -0400 Message-Id: <20180911184137.35897-8-dennisszhou@gmail.com> X-Mailer: git-send-email 2.13.5 In-Reply-To: <20180911184137.35897-1-dennisszhou@gmail.com> References: <20180911184137.35897-1-dennisszhou@gmail.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: "Dennis Zhou (Facebook)" One of the goals of this series is to remove a separate reference to the css of the bio. This can and should be accessed via bio_blkcg. In this patch, the wbc_init_bio call is changed such that it must be called after a queue has been associated with the bio. Signed-off-by: Dennis Zhou Reviewed-by: Josef Bacik Acked-by: Tejun Heo --- Documentation/admin-guide/cgroup-v2.rst | 8 +++++--- fs/buffer.c | 10 +++++----- fs/ext4/page-io.c | 2 +- include/linux/writeback.h | 5 +++-- 4 files changed, 14 insertions(+), 11 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 184193bcb262..caf36105a1c7 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -1857,8 +1857,10 @@ following two functions. wbc_init_bio(@wbc, @bio) Should be called for each bio carrying writeback data and - associates the bio with the inode's owner cgroup. Can be - called anytime between bio allocation and submission. + associates the bio with the inode's owner cgroup and the + corresponding request queue. This must be called after + a queue (device) has been associated with the bio and + before submission. wbc_account_io(@wbc, @page, @bytes) Should be called for each data segment being written out. @@ -1877,7 +1879,7 @@ the configuration, the bio may be executed at a lower priority and if the writeback session is holding shared resources, e.g. a journal entry, may lead to priority inversion. There is no one easy solution for the problem. Filesystems can try to work around specific problem -cases by skipping wbc_init_bio() or using bio_associate_blkcg() +cases by skipping wbc_init_bio() or using bio_associate_create_blkg() directly. diff --git a/fs/buffer.c b/fs/buffer.c index 6f1ae3ac9789..109f55196866 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -3060,11 +3060,6 @@ static int submit_bh_wbc(int op, int op_flags, struct buffer_head *bh, */ bio = bio_alloc(GFP_NOIO, 1); - if (wbc) { - wbc_init_bio(wbc, bio); - wbc_account_io(wbc, bh->b_page, bh->b_size); - } - bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9); bio_set_dev(bio, bh->b_bdev); bio->bi_write_hint = write_hint; @@ -3084,6 +3079,11 @@ static int submit_bh_wbc(int op, int op_flags, struct buffer_head *bh, op_flags |= REQ_PRIO; bio_set_op_attrs(bio, op, op_flags); + if (wbc) { + wbc_init_bio(wbc, bio); + wbc_account_io(wbc, bh->b_page, bh->b_size); + } + submit_bio(bio); return 0; } diff --git a/fs/ext4/page-io.c b/fs/ext4/page-io.c index db7590178dfc..2aa62d58d8dd 100644 --- a/fs/ext4/page-io.c +++ b/fs/ext4/page-io.c @@ -374,13 +374,13 @@ static int io_submit_init_bio(struct ext4_io_submit *io, bio = bio_alloc(GFP_NOIO, BIO_MAX_PAGES); if (!bio) return -ENOMEM; - wbc_init_bio(io->io_wbc, bio); bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9); bio_set_dev(bio, bh->b_bdev); bio->bi_end_io = ext4_end_bio; bio->bi_private = ext4_get_io_end(io->io_end); io->io_bio = bio; io->io_next_block = bh->b_blocknr; + wbc_init_bio(io->io_wbc, bio); return 0; } diff --git a/include/linux/writeback.h b/include/linux/writeback.h index fdfd04e348f6..738a0c24874f 100644 --- a/include/linux/writeback.h +++ b/include/linux/writeback.h @@ -246,7 +246,8 @@ static inline void wbc_attach_fdatawrite_inode(struct writeback_control *wbc, * * @bio is a part of the writeback in progress controlled by @wbc. Perform * writeback specific initialization. This is used to apply the cgroup - * writeback context. + * writeback context. Must be called after the bio has been associated with + * a device. */ static inline void wbc_init_bio(struct writeback_control *wbc, struct bio *bio) { @@ -257,7 +258,7 @@ static inline void wbc_init_bio(struct writeback_control *wbc, struct bio *bio) * regular writeback instead of writing things out itself. */ if (wbc->wb) - bio_associate_blkcg(bio, wbc->wb->blkcg_css); + bio_associate_blkg_from_css(bio, wbc->wb->blkcg_css); } #else /* CONFIG_CGROUP_WRITEBACK */ From patchwork Tue Sep 11 18:41:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 10596163 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4D8CC14F9 for ; Tue, 11 Sep 2018 18:42:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3F06329CA8 for ; Tue, 11 Sep 2018 18:42:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 338B529CBA; Tue, 11 Sep 2018 18:42:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7198329CA8 for ; Tue, 11 Sep 2018 18:42:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728466AbeIKXnI (ORCPT ); Tue, 11 Sep 2018 19:43:08 -0400 Received: from mail-yb1-f193.google.com ([209.85.219.193]:39458 "EHLO mail-yb1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726689AbeIKXm2 (ORCPT ); Tue, 11 Sep 2018 19:42:28 -0400 Received: by mail-yb1-f193.google.com with SMTP id c4-v6so9730870ybl.6; Tue, 11 Sep 2018 11:41:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=YMAVRklk/4sYhDVPwGuayrITVLMrMHDpd9GIsM9ItTE=; b=KW6b1MZPVu3BJezWeGzrg04KX0BrEsLMLcJQzw118+EBaStzNH2pgzEl9xxdNRsROI ovPQKk0QyEqRc2IqufjFJvrBXFYWJHqYHKZXxkkjGPVMBd8XuX62VgkKgmAAuH1FpLlx G4GsuDik2yynehzTyMqS9Mukx+0Nf8Vrw9kIKPuFB9ZBeQiCXeFhks+WPq1D9dQ+VJ75 0KESbPjR5MtOZqbJvln3nXke9PXrIO/HjJU2my6pe2gnvzJH3LHkKa3D22w5KOHIZFgF aC7ubJK0+vaQFHFrWd13QVq3xEyf3Ln3wZR2HHYcXKp0PGF73miF51aMmv8lOvtf3cGo RoSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=YMAVRklk/4sYhDVPwGuayrITVLMrMHDpd9GIsM9ItTE=; b=U/kHAph4wDrPNODETx+7Ymg6gH9Fgg7p18hgqbzwyxO+QojS7+UpEk/OM3SpumNTkt q279hxUw2/qjIQX0lUAmsbrLSNQ/cbHfiJEEUd9DvQV1r2ch9H/twMrHioVc//ATkDvb SBREG5wQSFeRspO4b0j16zdCOAAsnnkBgsb3Rj/PmnL3tgWUS7LAYIeGjQ9I2+2Z4MKH o4cKxTF4L/aKyF8kTQgI1EXzybmzwQtg2VU81WlSxn5VdIFdSpsae5++Y/yZWqKjxFfL +EBoFGZek+JX1VYnzsV/4VGTsYfQNQ6cm2thwpkLaOxU9beAOW80PAxWiKIQhL2D4wa6 Zx0g== X-Gm-Message-State: APzg51Bj7PtTssmgO40vXq4/x1F80gSiQtgzhQJPAkkX13ShIbPGCrD3 yPkMcGonoq6eP5fc9Y3WiNMBPgPDoow= X-Google-Smtp-Source: ANB0VdYZVxbVYPIMO9J9pstUNhaIDMlNHRocUJE7UpVFaFMR1C97xTaaM/6zON2q9/AKjcLz3WGNLw== X-Received: by 2002:a25:5bc3:: with SMTP id p186-v6mr12388257ybb.439.1536691312419; Tue, 11 Sep 2018 11:41:52 -0700 (PDT) Received: from dennisz-mbp.thefacebook.com ([199.201.65.129]) by smtp.gmail.com with ESMTPSA id i123-v6sm7902875ywe.14.2018.09.11.11.41.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 11:41:51 -0700 (PDT) From: Dennis Zhou To: Jens Axboe , Tejun Heo , Johannes Weiner , Josef Bacik Cc: kernel-team@fb.com, linux-block@vger.kernel.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, "Dennis Zhou (Facebook)" Subject: [PATCH 08/12] blkcg: remove bio->bi_css and instead use bio->bi_blkg Date: Tue, 11 Sep 2018 14:41:33 -0400 Message-Id: <20180911184137.35897-9-dennisszhou@gmail.com> X-Mailer: git-send-email 2.13.5 In-Reply-To: <20180911184137.35897-1-dennisszhou@gmail.com> References: <20180911184137.35897-1-dennisszhou@gmail.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: "Dennis Zhou (Facebook)" Prior patches ensured that all bios are now associated with some blkg. This now makes bio->bi_css unnecessary as blkg maintains a reference to the blkcg already. This patch removes the field bi_css and transfers corresponding uses to access via bi_blkg. Signed-off-by: Dennis Zhou Reviewed-by: Josef Bacik Acked-by: Tejun Heo --- block/bio.c | 56 ++++++++------------------------------ block/bounce.c | 2 +- drivers/block/loop.c | 5 ++-- drivers/md/raid0.c | 2 +- include/linux/bio.h | 9 ++---- include/linux/blk-cgroup.h | 8 +++--- include/linux/blk_types.h | 1 - kernel/trace/blktrace.c | 4 +-- 8 files changed, 25 insertions(+), 62 deletions(-) diff --git a/block/bio.c b/block/bio.c index 897c8ed9e572..f32b4757a232 100644 --- a/block/bio.c +++ b/block/bio.c @@ -609,7 +609,7 @@ void __bio_clone_fast(struct bio *bio, struct bio *bio_src) bio->bi_iter = bio_src->bi_iter; bio->bi_io_vec = bio_src->bi_io_vec; - bio_clone_blkcg_association(bio, bio_src); + bio_clone_blkg_association(bio, bio_src); blkcg_bio_issue_init(bio); } @@ -1957,34 +1957,6 @@ EXPORT_SYMBOL(bioset_init_from_src); #ifdef CONFIG_BLK_CGROUP -/** - * bio_associate_blkcg - associate a bio with the specified blkcg - * @bio: target bio - * @blkcg_css: css of the blkcg to associate - * - * Associate @bio with the blkcg specified by @blkcg_css. Block layer will - * treat @bio as if it were issued by a task which belongs to the blkcg. - * - * This function takes an extra reference of @blkcg_css which will be put - * when @bio is released. The caller must own @bio and is responsible for - * synchronizing calls to this function. If @blkcg_css is NULL, a call to - * blkcg_get_css finds the current css from the kthread or task. - */ -int bio_associate_blkcg(struct bio *bio, struct cgroup_subsys_state *blkcg_css) -{ - if (unlikely(bio->bi_css)) - return -EBUSY; - - if (blkcg_css) - css_get(blkcg_css); - else - blkcg_css = blkcg_get_css(); - - bio->bi_css = blkcg_css; - return 0; -} -EXPORT_SYMBOL_GPL(bio_associate_blkcg); - /** * bio_associate_blkg - associate a bio with the a blkg * @bio: target bio @@ -2034,7 +2006,6 @@ int bio_associate_blkg_from_css(struct bio *bio, struct cgroup_subsys_state *css) { css_get(css); - bio->bi_css = css; return __bio_associate_blkg_from_css(bio, css); } EXPORT_SYMBOL_GPL(bio_associate_blkg_from_css); @@ -2055,12 +2026,11 @@ int bio_associate_blkg_from_page(struct bio *bio, struct page *page) { struct cgroup_subsys_state *css; - if (unlikely(bio->bi_css)) + if (unlikely(bio->bi_blkg)) return -EBUSY; if (!page->mem_cgroup) return 0; css = cgroup_get_e_css(page->mem_cgroup->css.cgroup, &io_cgrp_subsys); - bio->bi_css = css; return __bio_associate_blkg_from_css(bio, css); } @@ -2086,8 +2056,7 @@ int bio_associate_create_blkg(struct request_queue *q, struct bio *bio) rcu_read_lock(); - bio_associate_blkcg(bio, NULL); - blkcg = bio_blkcg(bio); + blkcg = css_to_blkcg(blkcg_get_css()); if (!blkcg->css.parent) { ret = bio_associate_blkg(bio, q->root_blkg); @@ -2111,30 +2080,27 @@ void bio_disassociate_task(struct bio *bio) put_io_context(bio->bi_ioc); bio->bi_ioc = NULL; } - if (bio->bi_css) { - css_put(bio->bi_css); - bio->bi_css = NULL; - } if (bio->bi_blkg) { + /* a ref is always taken on css */ + css_put(&bio_blkcg(bio)->css); blkg_put(bio->bi_blkg); bio->bi_blkg = NULL; } } /** - * bio_clone_blkcg_association - clone blkcg association from src to dst bio + * bio_clone_blkg_association - clone blkg association from src to dst bio * @dst: destination bio * @src: source bio */ -void bio_clone_blkcg_association(struct bio *dst, struct bio *src) +void bio_clone_blkg_association(struct bio *dst, struct bio *src) { - if (src->bi_css) - WARN_ON(bio_associate_blkcg(dst, src->bi_css)); - - if (src->bi_blkg) + if (src->bi_blkg) { + css_get(&bio_blkcg(src)->css); bio_associate_blkg(dst, src->bi_blkg); + } } -EXPORT_SYMBOL_GPL(bio_clone_blkcg_association); +EXPORT_SYMBOL_GPL(bio_clone_blkg_association); #endif /* CONFIG_BLK_CGROUP */ static void __init biovec_init_slabs(void) diff --git a/block/bounce.c b/block/bounce.c index 7a08703b1204..b30071ac4ec6 100644 --- a/block/bounce.c +++ b/block/bounce.c @@ -257,7 +257,7 @@ static struct bio *bounce_clone_bio(struct bio *bio_src, gfp_t gfp_mask, } } - bio_clone_blkcg_association(bio, bio_src); + bio_clone_blkg_association(bio, bio_src); blkcg_bio_issue_init(bio); diff --git a/drivers/block/loop.c b/drivers/block/loop.c index ea9debf59b22..abad6d15f956 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -77,6 +77,7 @@ #include #include #include +#include #include "loop.h" @@ -1760,8 +1761,8 @@ static blk_status_t loop_queue_rq(struct blk_mq_hw_ctx *hctx, /* always use the first bio's css */ #ifdef CONFIG_BLK_CGROUP - if (cmd->use_aio && rq->bio && rq->bio->bi_css) { - cmd->css = rq->bio->bi_css; + if (cmd->use_aio && rq->bio && rq->bio->bi_blkg) { + cmd->css = &bio_blkcg(rq->bio)->css; css_get(cmd->css); } else #endif diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c index ac1cffd2a09b..f3fb5bb8c82a 100644 --- a/drivers/md/raid0.c +++ b/drivers/md/raid0.c @@ -542,7 +542,7 @@ static void raid0_handle_discard(struct mddev *mddev, struct bio *bio) !discard_bio) continue; bio_chain(discard_bio, bio); - bio_clone_blkcg_association(discard_bio, bio); + bio_clone_blkg_association(discard_bio, bio); if (mddev->gendisk) trace_block_bio_remap(bdev_get_queue(rdev->bdev), discard_bio, disk_devt(mddev->gendisk), diff --git a/include/linux/bio.h b/include/linux/bio.h index c73a870ebc0e..e973876625a8 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -540,24 +540,21 @@ static inline int bio_associate_blkg_from_page(struct bio *bio, #endif #ifdef CONFIG_BLK_CGROUP -int bio_associate_blkcg(struct bio *bio, struct cgroup_subsys_state *blkcg_css); int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg); int bio_associate_blkg_from_css(struct bio *bio, struct cgroup_subsys_state *css); int bio_associate_create_blkg(struct request_queue *q, struct bio *bio); void bio_disassociate_task(struct bio *bio); -void bio_clone_blkcg_association(struct bio *dst, struct bio *src); +void bio_clone_blkg_association(struct bio *dst, struct bio *src); #else /* CONFIG_BLK_CGROUP */ -static inline int bio_associate_blkcg(struct bio *bio, - struct cgroup_subsys_state *blkcg_css) { return 0; } static inline int bio_associate_blkg_from_css(struct bio *bio, struct cgroup_subsys_state *css) { return 0; } static inline int bio_associate_create_blkg(struct request_queue *q, struct bio *bio) { return 0; } static inline void bio_disassociate_task(struct bio *bio) { } -static inline void bio_clone_blkcg_association(struct bio *dst, - struct bio *src) { } +static inline void bio_clone_blkg_association(struct bio *dst, + struct bio *src) { } #endif /* CONFIG_BLK_CGROUP */ #ifdef CONFIG_HIGHMEM diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index a6b6e741a75e..c41cfcc2b4d8 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -308,8 +308,8 @@ static inline struct blkcg *css_to_blkcg(struct cgroup_subsys_state *css) */ static inline struct blkcg *__bio_blkcg(struct bio *bio) { - if (bio && bio->bi_css) - return css_to_blkcg(bio->bi_css); + if (bio && bio->bi_blkg) + return bio->bi_blkg->blkcg; return css_to_blkcg(blkcg_css()); } @@ -323,8 +323,8 @@ static inline struct blkcg *__bio_blkcg(struct bio *bio) */ static inline struct blkcg *bio_blkcg(struct bio *bio) { - if (bio && bio->bi_css) - return css_to_blkcg(bio->bi_css); + if (bio && bio->bi_blkg) + return bio->bi_blkg->blkcg; return NULL; } diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index f6dfb30737d8..9578c7ab1eb6 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -178,7 +178,6 @@ struct bio { * release. Read comment on top of bio_associate_current(). */ struct io_context *bi_ioc; - struct cgroup_subsys_state *bi_css; struct blkcg_gq *bi_blkg; struct bio_issue bi_issue; #endif diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c index 2868d85f1fb1..fac0ddf8a8e2 100644 --- a/kernel/trace/blktrace.c +++ b/kernel/trace/blktrace.c @@ -764,9 +764,9 @@ blk_trace_bio_get_cgid(struct request_queue *q, struct bio *bio) if (!bt || !(blk_tracer_flags.val & TRACE_BLK_OPT_CGROUP)) return NULL; - if (!bio->bi_css) + if (!bio->bi_blkg) return NULL; - return cgroup_get_kernfs_id(bio->bi_css->cgroup); + return cgroup_get_kernfs_id(bio_blkcg(bio)->css.cgroup); } #else static union kernfs_node_id * From patchwork Tue Sep 11 18:41:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 10596159 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0E19614F9 for ; Tue, 11 Sep 2018 18:42:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F236229CA8 for ; Tue, 11 Sep 2018 18:42:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E60C329CC0; Tue, 11 Sep 2018 18:42:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 132BC29CA8 for ; Tue, 11 Sep 2018 18:42:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728343AbeIKXmb (ORCPT ); Tue, 11 Sep 2018 19:42:31 -0400 Received: from mail-yb1-f195.google.com ([209.85.219.195]:42394 "EHLO mail-yb1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728310AbeIKXma (ORCPT ); Tue, 11 Sep 2018 19:42:30 -0400 Received: by mail-yb1-f195.google.com with SMTP id j8-v6so9713399ybg.9; Tue, 11 Sep 2018 11:41:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=JpX7At2uP82AEiO2R0RYzwv6ZW2l9R3i4u2KsNZHRcA=; b=kxujb3Do87hbX/w8Y1S1FimQkbRIymufwe2VDd6yjnT3cR5knvrQvGY8804azM8nA1 /EWOVvQhKB/nJAYcuawVL1VFeOoI0ZKKc7zQ+ghcX4YPGJIDAZDVKLRBJeIaVmGWbNpv IL2iM3ftlrJ2EmMFUV4/ycGV5cgMMRBxBkukNkFIsG4ThZUuwySE7x1U42zmgxdiqI35 uoSlIw/l59yw5wVkvPNSD20cMT63Cjyi29RH3pTVJNqukmw8LCTICgC+kG6T611Dgqtt m8kGAXweJoa/ZNpuBK8y8QMLaR9K0jSq/CF9Qbv1PEQ8/JmW5Cu7RaLM7Bf5py0y/B47 Egfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=JpX7At2uP82AEiO2R0RYzwv6ZW2l9R3i4u2KsNZHRcA=; b=ZgnZfpXsBZKsl894mXyVAcSnkuQ/h8HgfXYFSmpfLk4nbRaSfP717fN/Tt23QrLC/f YFH/Xiz5rO9zWLRViIZWfkssAgT39w5RIA1fG677KnYzIOeo/MLGD4uighkXHS0zLXGM Ziwds0+AAs9QD/4DDQT3VYST0jPjunF9qM3lKeI33GIJYzU+f0zeQdkVgSndLbb5pCbq zO8eiYmM3iG1FfooRCVax4hGjdQzj7XdMhJmVDKFgSfdLkkkzny/7eA0IWNrn90jqV0n d8+rjPrvDYrgP03F5D8Ev1fnEPUjRUtP6s+E9CJmQvAB4vHx/hMlVr4SBsa8uEq1hnxw rKsA== X-Gm-Message-State: APzg51CZKWfVIUpR6UNpiVsJaXeTHLphL21bQ9RNwu8zROUUofQwmLyx f8IUQdLgYcYtX87MSzpBvmEukZ2aoio= X-Google-Smtp-Source: ANB0VdaUsAHSdnhhABHCgX+GqiHgJDIRUjTqxawPpCLeYkfPegAd9C30zQrpGDj1rCpe1nTZRXKoVQ== X-Received: by 2002:a25:adc6:: with SMTP id d6-v6mr9839627ybe.342.1536691313567; Tue, 11 Sep 2018 11:41:53 -0700 (PDT) Received: from dennisz-mbp.thefacebook.com ([199.201.65.129]) by smtp.gmail.com with ESMTPSA id i123-v6sm7902875ywe.14.2018.09.11.11.41.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 11:41:52 -0700 (PDT) From: Dennis Zhou To: Jens Axboe , Tejun Heo , Johannes Weiner , Josef Bacik Cc: kernel-team@fb.com, linux-block@vger.kernel.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, "Dennis Zhou (Facebook)" Subject: [PATCH 09/12] blkcg: remove additional reference to the css Date: Tue, 11 Sep 2018 14:41:34 -0400 Message-Id: <20180911184137.35897-10-dennisszhou@gmail.com> X-Mailer: git-send-email 2.13.5 In-Reply-To: <20180911184137.35897-1-dennisszhou@gmail.com> References: <20180911184137.35897-1-dennisszhou@gmail.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: "Dennis Zhou (Facebook)" The previous patch in this series removed carrying around a pointer to the css in blkg. However, the blkg association logic still relied on taking a reference on the css to ensure we wouldn't fail in getting a reference for the blkg. Here the implicit dependency on the css is removed. The association continues to rely on the tryget logic walking up the blkg tree. This streamlines the three ways that association can happen: normal, swap, and writeback. Signed-off-by: Dennis Zhou Acked-by: Tejun Heo --- block/bio.c | 62 ++++++++++++++++++++++---------------- include/linux/blk-cgroup.h | 52 +++----------------------------- include/linux/cgroup.h | 2 ++ kernel/cgroup/cgroup.c | 48 +++++++++++++++++++++++------ 4 files changed, 81 insertions(+), 83 deletions(-) diff --git a/block/bio.c b/block/bio.c index f32b4757a232..a65ab3bacb4f 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1979,18 +1979,30 @@ int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg) return 0; } +/** + * __bio_associate_blkg_from_css - internal blkg association function + * + * This in the core association function that all association paths rely on. + * A blkg reference is taken which is released upon freeing of the bio. + */ static int __bio_associate_blkg_from_css(struct bio *bio, struct cgroup_subsys_state *css) { + struct request_queue *q = bio->bi_disk->queue; struct blkcg_gq *blkg; + int ret; rcu_read_lock(); - blkg = blkg_lookup_create(css_to_blkcg(css), bio->bi_disk->queue); + if (!css || !css->parent) + blkg = q->root_blkg; + else + blkg = blkg_lookup_create(css_to_blkcg(css), q); - rcu_read_unlock(); + ret = bio_associate_blkg(bio, blkg); - return bio_associate_blkg(bio, blkg); + rcu_read_unlock(); + return ret; } /** @@ -1999,13 +2011,14 @@ static int __bio_associate_blkg_from_css(struct bio *bio, * @css: target css * * Associate @bio with the blkg found by combining the css's blkg and the - * request_queue of the @bio. This takes a reference on the css that will - * be put upon freeing of @bio. + * request_queue of the @bio. This falls back to the queue's root_blkg if + * the association fails with the css. */ int bio_associate_blkg_from_css(struct bio *bio, struct cgroup_subsys_state *css) { - css_get(css); + if (unlikely(bio->bi_blkg)) + return -EBUSY; return __bio_associate_blkg_from_css(bio, css); } EXPORT_SYMBOL_GPL(bio_associate_blkg_from_css); @@ -2017,22 +2030,29 @@ EXPORT_SYMBOL_GPL(bio_associate_blkg_from_css); * @page: the page to lookup the blkcg from * * Associate @bio with the blkg from @page's owning memcg and the respective - * request_queue. This works like every other associate function wrt - * references. + * request_queue. If cgroup_e_css returns NULL, fall back to the queue's + * root_blkg. * * Note: this must be called after bio has an associated device. */ int bio_associate_blkg_from_page(struct bio *bio, struct page *page) { struct cgroup_subsys_state *css; + int ret; if (unlikely(bio->bi_blkg)) return -EBUSY; if (!page->mem_cgroup) return 0; - css = cgroup_get_e_css(page->mem_cgroup->css.cgroup, &io_cgrp_subsys); - return __bio_associate_blkg_from_css(bio, css); + rcu_read_lock(); + + css = cgroup_e_css(page->mem_cgroup->css.cgroup, &io_cgrp_subsys); + + ret = __bio_associate_blkg_from_css(bio, css); + + rcu_read_unlock(); + return ret; } #endif /* CONFIG_MEMCG */ @@ -2042,12 +2062,12 @@ int bio_associate_blkg_from_page(struct bio *bio, struct page *page) * @bio: target bio * * Associate @bio with the blkg found from the bio's css and the request_queue. - * If one is not found, bio_lookup_blkg creates the blkg. + * If one is not found, bio_lookup_blkg creates the blkg. This falls back to + * the queue's root_blkg if association fails. */ int bio_associate_create_blkg(struct request_queue *q, struct bio *bio) { - struct blkcg *blkcg; - struct blkcg_gq *blkg; + struct cgroup_subsys_state *css; int ret = 0; /* someone has already associated this bio with a blkg */ @@ -2056,15 +2076,9 @@ int bio_associate_create_blkg(struct request_queue *q, struct bio *bio) rcu_read_lock(); - blkcg = css_to_blkcg(blkcg_get_css()); + css = blkcg_css(); - if (!blkcg->css.parent) { - ret = bio_associate_blkg(bio, q->root_blkg); - } else { - blkg = blkg_lookup_create(blkcg, q); - - ret = bio_associate_blkg(bio, blkg); - } + ret = __bio_associate_blkg_from_css(bio, css); rcu_read_unlock(); return ret; @@ -2081,8 +2095,6 @@ void bio_disassociate_task(struct bio *bio) bio->bi_ioc = NULL; } if (bio->bi_blkg) { - /* a ref is always taken on css */ - css_put(&bio_blkcg(bio)->css); blkg_put(bio->bi_blkg); bio->bi_blkg = NULL; } @@ -2095,10 +2107,8 @@ void bio_disassociate_task(struct bio *bio) */ void bio_clone_blkg_association(struct bio *dst, struct bio *src) { - if (src->bi_blkg) { - css_get(&bio_blkcg(src)->css); + if (src->bi_blkg) bio_associate_blkg(dst, src->bi_blkg); - } } EXPORT_SYMBOL_GPL(bio_clone_blkg_association); #endif /* CONFIG_BLK_CGROUP */ diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index c41cfcc2b4d8..2951ea3541b1 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -249,47 +249,6 @@ static inline struct cgroup_subsys_state *blkcg_css(void) return task_css(current, io_cgrp_id); } -/** - * blkcg_get_css - find and get a reference to the css - * - * Find the css associated with either the kthread or the current task. - * This takes a reference on the blkcg which will need to be managed by the - * caller. - */ -static inline struct cgroup_subsys_state *blkcg_get_css(void) -{ - struct cgroup_subsys_state *css; - - rcu_read_lock(); - - css = kthread_blkcg(); - if (css) { - css_get(css); - } else { - /* - * This is a bit complicated. It is possible task_css is seeing - * an old css pointer here. This is caused by the current - * thread migrating away from this cgroup and this cgroup dying. - * css_tryget() will fail when trying to take a ref on a cgroup - * that's ref count has hit 0. - * - * Therefore, if it does fail, this means current must have - * been swapped away already and this is waiting for it to - * propagate on the polling cpu. Hence the use of cpu_relax(). - */ - while (true) { - css = task_css(current, io_cgrp_id); - if (likely(css_tryget(css))) - break; - cpu_relax(); - } - } - - rcu_read_unlock(); - - return css; -} - static inline struct blkcg *css_to_blkcg(struct cgroup_subsys_state *css) { return css ? container_of(css, struct blkcg, css) : NULL; @@ -628,10 +587,8 @@ static inline struct request_list *blk_get_rl(struct request_queue *q, rcu_read_lock(); blkcg = bio_blkcg(bio); - if (blkcg) - css_get(&blkcg->css); - else - blkcg = css_to_blkcg(blkcg_get_css()); + if (!blkcg) + blkcg = css_to_blkcg(blkcg_css()); /* bypass blkg lookup and use @q->root_rl directly for root */ if (blkcg == &blkcg_root) @@ -646,7 +603,8 @@ static inline struct request_list *blk_get_rl(struct request_queue *q, if (unlikely(!blkg)) goto root_rl; - blkg_get(blkg); + if (!blkg_try_get(blkg)) + goto root_rl; rcu_read_unlock(); return &blkg->rl; root_rl: @@ -663,8 +621,6 @@ static inline struct request_list *blk_get_rl(struct request_queue *q, */ static inline void blk_put_rl(struct request_list *rl) { - /* an additional ref is always taken for rl */ - css_put(&rl->blkg->blkcg->css); if (rl->blkg->blkcg != &blkcg_root) blkg_put(rl->blkg); } diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index 32c553556bbd..b8bcbdeb2eac 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -93,6 +93,8 @@ extern struct css_set init_css_set; bool css_has_online_children(struct cgroup_subsys_state *css); struct cgroup_subsys_state *css_from_id(int id, struct cgroup_subsys *ss); +struct cgroup_subsys_state *cgroup_e_css(struct cgroup *cgroup, + struct cgroup_subsys *ss); struct cgroup_subsys_state *cgroup_get_e_css(struct cgroup *cgroup, struct cgroup_subsys *ss); struct cgroup_subsys_state *css_tryget_online_from_dir(struct dentry *dentry, diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index aae10baf1902..48fb22e49467 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -492,7 +492,7 @@ static struct cgroup_subsys_state *cgroup_tryget_css(struct cgroup *cgrp, } /** - * cgroup_e_css - obtain a cgroup's effective css for the specified subsystem + * cgroup_e_css_by_mask - obtain a cgroup's effective css for the specified ss * @cgrp: the cgroup of interest * @ss: the subsystem of interest (%NULL returns @cgrp->self) * @@ -501,8 +501,8 @@ static struct cgroup_subsys_state *cgroup_tryget_css(struct cgroup *cgrp, * enabled. If @ss is associated with the hierarchy @cgrp is on, this * function is guaranteed to return non-NULL css. */ -static struct cgroup_subsys_state *cgroup_e_css(struct cgroup *cgrp, - struct cgroup_subsys *ss) +static struct cgroup_subsys_state *cgroup_e_css_by_mask(struct cgroup *cgrp, + struct cgroup_subsys *ss) { lockdep_assert_held(&cgroup_mutex); @@ -522,6 +522,35 @@ static struct cgroup_subsys_state *cgroup_e_css(struct cgroup *cgrp, return cgroup_css(cgrp, ss); } +/** + * cgroup_e_css - obtain a cgroup's effective css for the specified subsystem + * @cgrp: the cgroup of interest + * @ss: the subsystem of interest + * + * Find and get the effective css of @cgrp for @ss. The effective css is + * defined as the matching css of the nearest ancestor including self which + * has @ss enabled. If @ss is not mounted on the hierarchy @cgrp is on, + * the root css is returned, so this function always returns a valid css. + * + * The returned css is not guaranteed to be online, and therefore it is the + * callers responsiblity to tryget a reference for it. + */ +struct cgroup_subsys_state *cgroup_e_css(struct cgroup *cgrp, + struct cgroup_subsys *ss) +{ + struct cgroup_subsys_state *css; + + do { + css = cgroup_css(cgrp, ss); + + if (css) + return css; + cgrp = cgroup_parent(cgrp); + } while (cgrp); + + return init_css_set.subsys[ss->id]; +} + /** * cgroup_get_e_css - get a cgroup's effective css for the specified subsystem * @cgrp: the cgroup of interest @@ -604,10 +633,11 @@ EXPORT_SYMBOL_GPL(of_css); * * Should be called under cgroup_[tree_]mutex. */ -#define for_each_e_css(css, ssid, cgrp) \ - for ((ssid) = 0; (ssid) < CGROUP_SUBSYS_COUNT; (ssid)++) \ - if (!((css) = cgroup_e_css(cgrp, cgroup_subsys[(ssid)]))) \ - ; \ +#define for_each_e_css(css, ssid, cgrp) \ + for ((ssid) = 0; (ssid) < CGROUP_SUBSYS_COUNT; (ssid)++) \ + if (!((css) = cgroup_e_css_by_mask(cgrp, \ + cgroup_subsys[(ssid)]))) \ + ; \ else /** @@ -1006,7 +1036,7 @@ static struct css_set *find_existing_css_set(struct css_set *old_cset, * @ss is in this hierarchy, so we want the * effective css from @cgrp. */ - template[i] = cgroup_e_css(cgrp, ss); + template[i] = cgroup_e_css_by_mask(cgrp, ss); } else { /* * @ss is not in this hierarchy, so we don't want @@ -3019,7 +3049,7 @@ static int cgroup_apply_control(struct cgroup *cgrp) return ret; /* - * At this point, cgroup_e_css() results reflect the new csses + * At this point, cgroup_e_css_by_mask() results reflect the new csses * making the following cgroup_update_dfl_csses() properly update * css associations of all tasks in the subtree. */ From patchwork Tue Sep 11 18:41:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 10596157 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 13F6E14F9 for ; Tue, 11 Sep 2018 18:42:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 05B4729CA8 for ; Tue, 11 Sep 2018 18:42:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ED90129CC0; Tue, 11 Sep 2018 18:42:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9723729CA8 for ; Tue, 11 Sep 2018 18:42:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728436AbeIKXmx (ORCPT ); Tue, 11 Sep 2018 19:42:53 -0400 Received: from mail-yb1-f193.google.com ([209.85.219.193]:45591 "EHLO mail-yb1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726622AbeIKXmb (ORCPT ); Tue, 11 Sep 2018 19:42:31 -0400 Received: by mail-yb1-f193.google.com with SMTP id h22-v6so9713332ybg.12; Tue, 11 Sep 2018 11:41:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=r2kxmE2i/F4zOX3cOoXMsnwx5/PIHFb0myqGCKQDXp8=; b=tZuNmPTwD++oiilT/SKFfVULg1rxGXW6srQaCTr04qW0cVciRTV5lzHvQurnUDbR9S llwwaZ1SwqS6vKH/uRmSlH+ITvlRmRVjgvIK57VjYCvS8b5HS4M/6ez7ixHpkvVuoQ5y CJjmws4EfPwseBRRXBGqXFU3FnbcBww8pTN+3YvfM2aeItvfjYt2aq6QyLGrL5IuoKa7 666KDlW9aPbob9Pg6IqOb++VDEbZAcpdbLU4j/bHtqmN67tU9kfKhaicyqjPmD5Dzmq3 w86C42pzxSPdb3S26HRTyORjWXRc46Wd87zC33NFnnJ7zNoRl0yJrvedYk9PloZQ0/0g ygxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=r2kxmE2i/F4zOX3cOoXMsnwx5/PIHFb0myqGCKQDXp8=; b=gtHQiS5ndzO1CoQom9M4PQ5wvzQMDRwn5VFK7EgKZvHcK5R6CMZuFrLDG0UjLWk6nO 3FX5QtAOqRyZ+YBF0iGaxgo3rOJUFP+OshlAnSNZH+KDrG81ULAZY14oAl46j679BYEO 5Qj3jJoqx/o89z1amF6BnM+VE+EGDDTk308GyIyJMPjpcnscGPLM61oMKEqZ9Cw4Hf44 GMvCnPH3srdUmtihA5dpjNsUY7Drneyz3cZtqh2fOmruRC3oLoEi9YVNCrd290YXFpAJ 2hXSfoagZBweOTs3jAZqkCzfd+7p4MLABNpcYKVLHz1tSWmOy/m1yGtaV6/m/BFz3AfJ jfZA== X-Gm-Message-State: APzg51DLLtlmOrYFXyaAVxo6d3jykN0uXXWJ66QtqKMRmvkmHZq0OmYE mpazT0mfZtHUaD/db51iYAM= X-Google-Smtp-Source: ANB0VdaHkU2p3IGJxzGy2ULStRpRbMdsR8aZ+E6JzrwiU55GQp3d4BSD6ysNPvAJlnHL5DNbUSQtHg== X-Received: by 2002:a25:ea51:: with SMTP id o17-v6mr1341956ybe.172.1536691314982; Tue, 11 Sep 2018 11:41:54 -0700 (PDT) Received: from dennisz-mbp.thefacebook.com ([199.201.65.129]) by smtp.gmail.com with ESMTPSA id i123-v6sm7902875ywe.14.2018.09.11.11.41.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 11:41:53 -0700 (PDT) From: Dennis Zhou To: Jens Axboe , Tejun Heo , Johannes Weiner , Josef Bacik Cc: kernel-team@fb.com, linux-block@vger.kernel.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, "Dennis Zhou (Facebook)" Subject: [PATCH 10/12] blkcg: cleanup and make blk_get_rl use blkg_lookup_create Date: Tue, 11 Sep 2018 14:41:35 -0400 Message-Id: <20180911184137.35897-11-dennisszhou@gmail.com> X-Mailer: git-send-email 2.13.5 In-Reply-To: <20180911184137.35897-1-dennisszhou@gmail.com> References: <20180911184137.35897-1-dennisszhou@gmail.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: "Dennis Zhou (Facebook)" blk_get_rl is responsible for identifying which request_list a request should be allocated to. Try get logic was added earlier, but semantically the logic was not changed. This patch makes better use of the bio already having a reference to the blkg in the hot path. The cold path uses a better fallback of blkg_lookup_create rather than just blkg_lookup and then falling back to the q->root_rl. If lookup_create fails with anything but -ENODEV, it falls back to q->root_rl. A clarifying comment is added to explain why q->root_rl is used rather than the root blkg's rl. Signed-off-by: Dennis Zhou Acked-by: Tejun Heo --- include/linux/blk-cgroup.h | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index 2951ea3541b1..d2f7f1b00fcf 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -586,28 +586,36 @@ static inline struct request_list *blk_get_rl(struct request_queue *q, rcu_read_lock(); - blkcg = bio_blkcg(bio); - if (!blkcg) - blkcg = css_to_blkcg(blkcg_css()); + if (bio && bio->bi_blkg) { + blkcg = bio->bi_blkg->blkcg; + if (blkcg == &blkcg_root) + goto rl_use_root; + + blkg_get(bio->bi_blkg); + rcu_read_unlock(); + return &bio->bi_blkg->rl; + } - /* bypass blkg lookup and use @q->root_rl directly for root */ + blkcg = css_to_blkcg(blkcg_css()); if (blkcg == &blkcg_root) - goto root_rl; + goto rl_use_root; - /* - * Try to use blkg->rl. blkg lookup may fail under memory pressure - * or if either the blkcg or queue is going away. Fall back to - * root_rl in such cases. - */ blkg = blkg_lookup(blkcg, q); if (unlikely(!blkg)) - goto root_rl; + blkg = __blkg_lookup_create(blkcg, q); if (!blkg_try_get(blkg)) - goto root_rl; + goto rl_use_root; + rcu_read_unlock(); return &blkg->rl; -root_rl: + + /* + * Each blkg has its own request_list, however, the root blkcg + * uses the request_queue's root_rl. This is to avoid most + * overhead for the root blkcg. + */ +rl_use_root: rcu_read_unlock(); return &q->root_rl; } From patchwork Tue Sep 11 18:41:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 10596155 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BF7C314F9 for ; Tue, 11 Sep 2018 18:42:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B0CCB29CA8 for ; Tue, 11 Sep 2018 18:42:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A4D3D29CC0; Tue, 11 Sep 2018 18:42:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 20E8429CA9 for ; Tue, 11 Sep 2018 18:42:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728374AbeIKXmd (ORCPT ); Tue, 11 Sep 2018 19:42:33 -0400 Received: from mail-yw1-f66.google.com ([209.85.161.66]:42509 "EHLO mail-yw1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728310AbeIKXmc (ORCPT ); Tue, 11 Sep 2018 19:42:32 -0400 Received: by mail-yw1-f66.google.com with SMTP id n207-v6so9606480ywn.9; Tue, 11 Sep 2018 11:41:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=t00huGWlCQMYRW7qCaQabcFxN7tdJkCV+CAUTXuBTX8=; b=a/nroMDzIaeWZrVAWHgkzg/D4eri6/NtzkrEjTak/uFrZpTKBXscexN5rewvhhabUs yLWEyQwZNQlTsoqlHPYqTzCdRftTp3MxZyPpu5T2nVcw8uVWvyBkV0vhMg06dGJGVkng oWazU7eAy37gNHFVFRQn2usO/kNxmK07tyurKxxvxBjDZHg90GhKhoon3Re+dTnjigk4 70dbUwXNdjV+/YfwbMZHs23ami0vb72NL7sAFiUyaE3pUEFdoVGDinB/faSFi97zMav6 NI44hUsrLdS+FACYwn76Nf0klus7P/8+Sohb9XaTkci0mmDSPkH2Wu2oDR456Qa6uwNh i66A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=t00huGWlCQMYRW7qCaQabcFxN7tdJkCV+CAUTXuBTX8=; b=ejXpcCyWK0SOiv20zK4Fl+rPQ4na/CkppKpuSa5B7cyRjooKEu6m0FLOzwWY6ol+lL 14o0uTVCPvldyEDqF6fliasE+DNK4QAY+PdMpiDUL7ANEikYhHG91Ar1+/DyRLgsXCdq bCLdjIxDj2uGQ08LEnjeBAUWyrRzWPuO05scvAfLrCCkEWBsGKIrpHCAK0jqhl516gKM KW5eSrTabNu5bFVNshKdFJf6ncaJd3PCampgtHugG0M7S15FBcuDUi9yRhA4goRByBCO DfIgqzxyJphoCJurtwtiWBLxPPa28okFHSm/MmAswkCHVCYqFaSjBDyhY64P8HkU9X+Z W9CQ== X-Gm-Message-State: APzg51CUDcShE8oG+phJBwHKwuW3jTBDFAkWYusia0m8pH717i2rY4Gg O7S374+g891QCL9RiCuZQEGW0yaEe+4= X-Google-Smtp-Source: ANB0VdYhRWOAgvBQco6mE+/yPdP+qlqR6yZgfHqMfPwg0pprC8RYc47hWIH79IFrdZHHY+fbCC5+Qw== X-Received: by 2002:a0d:d945:: with SMTP id b66-v6mr13123868ywe.493.1536691316507; Tue, 11 Sep 2018 11:41:56 -0700 (PDT) Received: from dennisz-mbp.thefacebook.com ([199.201.65.129]) by smtp.gmail.com with ESMTPSA id i123-v6sm7902875ywe.14.2018.09.11.11.41.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 11:41:55 -0700 (PDT) From: Dennis Zhou To: Jens Axboe , Tejun Heo , Johannes Weiner , Josef Bacik Cc: kernel-team@fb.com, linux-block@vger.kernel.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, "Dennis Zhou (Facebook)" Subject: [PATCH 11/12] blkcg: change blkg reference counting to use percpu_ref Date: Tue, 11 Sep 2018 14:41:36 -0400 Message-Id: <20180911184137.35897-12-dennisszhou@gmail.com> X-Mailer: git-send-email 2.13.5 In-Reply-To: <20180911184137.35897-1-dennisszhou@gmail.com> References: <20180911184137.35897-1-dennisszhou@gmail.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: "Dennis Zhou (Facebook)" Now that every bio is associated with a blkg, this puts the use of blkg_get, blkg_try_get, and blkg_put on the hot path. This switches over the refcnt in blkg to use percpu_ref. Signed-off-by: Dennis Zhou Acked-by: Tejun Heo --- block/blk-cgroup.c | 64 +++++++++++++++++++++++--------------- include/linux/blk-cgroup.h | 15 +++------ 2 files changed, 44 insertions(+), 35 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index e9e3a955f61a..ab3676e1e15e 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -84,6 +84,37 @@ static void blkg_free(struct blkcg_gq *blkg) kfree(blkg); } +static void __blkg_release(struct rcu_head *rcu) +{ + struct blkcg_gq *blkg = container_of(rcu, struct blkcg_gq, rcu_head); + + percpu_ref_exit(&blkg->refcnt); + + /* release the blkcg and parent blkg refs this blkg has been holding */ + css_put(&blkg->blkcg->css); + if (blkg->parent) + blkg_put(blkg->parent); + + wb_congested_put(blkg->wb_congested); + + blkg_free(blkg); +} + +/* + * A group is RCU protected, but having an rcu lock does not mean that one + * can access all the fields of blkg and assume these are valid. For + * example, don't try to follow throtl_data and request queue links. + * + * Having a reference to blkg under an rcu allows accesses to only values + * local to groups like group stats and group rate limits. + */ +static void blkg_release(struct percpu_ref *ref) +{ + struct blkcg_gq *blkg = container_of(ref, struct blkcg_gq, refcnt); + + call_rcu(&blkg->rcu_head, __blkg_release); +} + /** * blkg_alloc - allocate a blkg * @blkcg: block cgroup the new blkg is associated with @@ -110,7 +141,6 @@ static struct blkcg_gq *blkg_alloc(struct blkcg *blkcg, struct request_queue *q, blkg->q = q; INIT_LIST_HEAD(&blkg->q_node); blkg->blkcg = blkcg; - atomic_set(&blkg->refcnt, 1); /* root blkg uses @q->root_rl, init rl only for !root blkgs */ if (blkcg != &blkcg_root) { @@ -217,6 +247,11 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, blkg_get(blkg->parent); } + ret = percpu_ref_init(&blkg->refcnt, blkg_release, 0, + GFP_NOWAIT | __GFP_NOWARN); + if (ret) + goto err_cancel_ref; + /* invoke per-policy init */ for (i = 0; i < BLKCG_MAX_POLS; i++) { struct blkcg_policy *pol = blkcg_policy[i]; @@ -249,6 +284,8 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, blkg_put(blkg); return ERR_PTR(ret); +err_cancel_ref: + percpu_ref_exit(&blkg->refcnt); err_put_congested: wb_congested_put(wb_congested); err_put_css: @@ -387,7 +424,7 @@ static void blkg_destroy(struct blkcg_gq *blkg) * Put the reference taken at the time of creation so that when all * queues are gone, group can be destroyed. */ - blkg_put(blkg); + percpu_ref_kill(&blkg->refcnt); } /** @@ -414,29 +451,6 @@ static void blkg_destroy_all(struct request_queue *q) q->root_rl.blkg = NULL; } -/* - * A group is RCU protected, but having an rcu lock does not mean that one - * can access all the fields of blkg and assume these are valid. For - * example, don't try to follow throtl_data and request queue links. - * - * Having a reference to blkg under an rcu allows accesses to only values - * local to groups like group stats and group rate limits. - */ -void __blkg_release_rcu(struct rcu_head *rcu_head) -{ - struct blkcg_gq *blkg = container_of(rcu_head, struct blkcg_gq, rcu_head); - - /* release the blkcg and parent blkg refs this blkg has been holding */ - css_put(&blkg->blkcg->css); - if (blkg->parent) - blkg_put(blkg->parent); - - wb_congested_put(blkg->wb_congested); - - blkg_free(blkg); -} -EXPORT_SYMBOL_GPL(__blkg_release_rcu); - /* * The next function used by blk_queue_for_each_rl(). It's a bit tricky * because the root blkg uses @q->root_rl instead of its own rl. diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index d2f7f1b00fcf..7ff5d8ba8c7a 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -126,7 +126,7 @@ struct blkcg_gq { struct request_list rl; /* reference count */ - atomic_t refcnt; + struct percpu_ref refcnt; /* is this blkg online? protected by both blkcg and q locks */ bool online; @@ -490,8 +490,7 @@ static inline int blkg_path(struct blkcg_gq *blkg, char *buf, int buflen) */ static inline void blkg_get(struct blkcg_gq *blkg) { - WARN_ON_ONCE(atomic_read(&blkg->refcnt) <= 0); - atomic_inc(&blkg->refcnt); + percpu_ref_get(&blkg->refcnt); } /** @@ -503,7 +502,7 @@ static inline void blkg_get(struct blkcg_gq *blkg) */ static inline struct blkcg_gq *blkg_try_get(struct blkcg_gq *blkg) { - if (atomic_inc_not_zero(&blkg->refcnt)) + if (percpu_ref_tryget(&blkg->refcnt)) return blkg; return NULL; } @@ -517,23 +516,19 @@ static inline struct blkcg_gq *blkg_try_get(struct blkcg_gq *blkg) */ static inline struct blkcg_gq *blkg_try_get_closest(struct blkcg_gq *blkg) { - while (!atomic_inc_not_zero(&blkg->refcnt)) + while (!percpu_ref_tryget(&blkg->refcnt)) blkg = blkg->parent; return blkg; } -void __blkg_release_rcu(struct rcu_head *rcu); - /** * blkg_put - put a blkg reference * @blkg: blkg to put */ static inline void blkg_put(struct blkcg_gq *blkg) { - WARN_ON_ONCE(atomic_read(&blkg->refcnt) <= 0); - if (atomic_dec_and_test(&blkg->refcnt)) - call_rcu(&blkg->rcu_head, __blkg_release_rcu); + percpu_ref_put(&blkg->refcnt); } /** From patchwork Tue Sep 11 18:41:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 10596153 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1DD6214F9 for ; Tue, 11 Sep 2018 18:42:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0E6C829CA8 for ; Tue, 11 Sep 2018 18:42:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 02AFF29CC0; Tue, 11 Sep 2018 18:42:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8A5C429CA8 for ; Tue, 11 Sep 2018 18:42:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727865AbeIKXmo (ORCPT ); Tue, 11 Sep 2018 19:42:44 -0400 Received: from mail-yw1-f68.google.com ([209.85.161.68]:47093 "EHLO mail-yw1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728368AbeIKXmd (ORCPT ); Tue, 11 Sep 2018 19:42:33 -0400 Received: by mail-yw1-f68.google.com with SMTP id j131-v6so9596783ywc.13; Tue, 11 Sep 2018 11:41:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=zhl/2Dyq9QvCeiNHqKLR3IjW7/h51nLcSK+lBauCWR0=; b=gdwOfXpUVXotNnENEC5Plz4ZaeFlcFnbenVx5eLjQXQUnNbYlwV7N8Qpyz6GpF8h56 /s1+KnWao0oPn02D0hq7deBhx2teTphXM4eOl/c0JEmcrZCl2ZV/fycMaEdZjQSWTkqW Gh2iRQvB6ez/gkVNU+TtgbjPuT9bPXjwkz5+PNQiZ4Q+z5sjfEGjbLX2+McLy3whXxWH l0dauewPyU+OcbMxjdL3acVpVhXqVrij9B5gHzEHa1EmLzEYRRfBcDUGmX2fteDAJfb9 YkEXu+/WRZcLPCZ8B7cjwMiIQhem4MKSR4jGmNmDbEiWRLafPk6vysEsHCroD6QMgKhz oAmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=zhl/2Dyq9QvCeiNHqKLR3IjW7/h51nLcSK+lBauCWR0=; b=PMuq/SZGaHkkT9m9R+wUBriH2dMNx1TjfXG9R9+46WDG05M2dcj7fTsbqNrRDrgUru QbFhhXbTKuXF1Op6X1/YEBnZWwOA6cc0O9XS+qR/5Xn6Ux2az53GzWOrAWizkxIXpILv benQD8JZ1EZg+rvXszP4pSDT8T8A1JGfcAkIec/RY2ko5Fje0L8jWgqgUyls0mROchjw u0wUrlL1AowU+k802uf4cfvx44Qcqvwn9puznTdDLMe2GVbbW/4+sqPTEvCB82/ixkLE oEnOwfpJ5WiL63W2Jmmd4BXCOONydH0RIXxIQXisyUVnny0wxtu3Msq3gU/CdhOaj1ZI 1Ciw== X-Gm-Message-State: APzg51AwxUOsajgBpecDq2yr08gFehAYDuiDW9j40SnEEDQ9KosYU4VI k/lNKqm/F8tWQigEAmfyNKA= X-Google-Smtp-Source: ANB0VdaKF2yAsoy5PZiVo8JgmBuuqnGfM6nULWyDOk0EkLPrk80xA5aw2nYUiKL72yPHcfnBi6R0/Q== X-Received: by 2002:a81:812:: with SMTP id 18-v6mr13280788ywi.475.1536691317739; Tue, 11 Sep 2018 11:41:57 -0700 (PDT) Received: from dennisz-mbp.thefacebook.com ([199.201.65.129]) by smtp.gmail.com with ESMTPSA id i123-v6sm7902875ywe.14.2018.09.11.11.41.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Sep 2018 11:41:57 -0700 (PDT) From: Dennis Zhou To: Jens Axboe , Tejun Heo , Johannes Weiner , Josef Bacik Cc: kernel-team@fb.com, linux-block@vger.kernel.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, "Dennis Zhou (Facebook)" Subject: [PATCH 12/12] blkcg: rename blkg_try_get to blkg_tryget Date: Tue, 11 Sep 2018 14:41:37 -0400 Message-Id: <20180911184137.35897-13-dennisszhou@gmail.com> X-Mailer: git-send-email 2.13.5 In-Reply-To: <20180911184137.35897-1-dennisszhou@gmail.com> References: <20180911184137.35897-1-dennisszhou@gmail.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: "Dennis Zhou (Facebook)" blkg reference counting now uses percpu_ref rather than atomic_t. Let's make this consistent with css_tryget. This renames blkg_try_get to blkg_tryget and now returns a bool rather than the blkg or NULL. Signed-off-by: Dennis Zhou Reviewed-by: Josef Bacik Acked-by: Tejun Heo --- block/bio.c | 2 +- block/blk-cgroup.c | 3 +-- block/blk-iolatency.c | 2 +- include/linux/blk-cgroup.h | 14 ++++++-------- 4 files changed, 9 insertions(+), 12 deletions(-) diff --git a/block/bio.c b/block/bio.c index a65ab3bacb4f..14351fac07cd 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1975,7 +1975,7 @@ int bio_associate_blkg(struct bio *bio, struct blkcg_gq *blkg) { if (unlikely(bio->bi_blkg)) return -EBUSY; - bio->bi_blkg = blkg_try_get_closest(blkg); + bio->bi_blkg = blkg_tryget_closest(blkg); return 0; } diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index ab3676e1e15e..76136bea7a7f 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1794,8 +1794,7 @@ void blkcg_maybe_throttle_current(void) blkg = blkg_lookup(blkcg, q); if (!blkg) goto out; - blkg = blkg_try_get(blkg); - if (!blkg) + if (!blkg_tryget(blkg)) goto out; rcu_read_unlock(); diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index 9d7052bad6f7..5a4cec54c998 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -628,7 +628,7 @@ static void blkiolatency_timer_fn(struct timer_list *t) * We could be exiting, don't access the pd unless we have a * ref on the blkg. */ - if (!blkg_try_get(blkg)) + if (!blkg_tryget(blkg)) continue; iolat = blkg_to_lat(blkg); diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index 7ff5d8ba8c7a..b7fd08013de2 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -494,27 +494,25 @@ static inline void blkg_get(struct blkcg_gq *blkg) } /** - * blkg_try_get - try and get a blkg reference + * blkg_tryget - try and get a blkg reference * @blkg: blkg to get * * This is for use when doing an RCU lookup of the blkg. We may be in the midst * of freeing this blkg, so we can only use it if the refcnt is not zero. */ -static inline struct blkcg_gq *blkg_try_get(struct blkcg_gq *blkg) +static inline bool blkg_tryget(struct blkcg_gq *blkg) { - if (percpu_ref_tryget(&blkg->refcnt)) - return blkg; - return NULL; + return percpu_ref_tryget(&blkg->refcnt); } /** - * blkg_try_get_closest - try and get a blkg ref on the closet blkg + * blkg_tryget_closest - try and get a blkg ref on the closet blkg * @blkg: blkg to get * * This walks up the blkg tree to find the closest non-dying blkg and returns * the blkg that it did association with as it may not be the passed in blkg. */ -static inline struct blkcg_gq *blkg_try_get_closest(struct blkcg_gq *blkg) +static inline struct blkcg_gq *blkg_tryget_closest(struct blkcg_gq *blkg) { while (!percpu_ref_tryget(&blkg->refcnt)) blkg = blkg->parent; @@ -599,7 +597,7 @@ static inline struct request_list *blk_get_rl(struct request_queue *q, if (unlikely(!blkg)) blkg = __blkg_lookup_create(blkcg, q); - if (!blkg_try_get(blkg)) + if (!blkg_tryget(blkg)) goto rl_use_root; rcu_read_unlock();