From patchwork Tue May 1 06:48:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Vagin X-Patchwork-Id: 10373185 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 78A206032A for ; Tue, 1 May 2018 06:48:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6723828BF7 for ; Tue, 1 May 2018 06:48:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5AF0B28BFA; Tue, 1 May 2018 06:48:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1A62428BF7 for ; Tue, 1 May 2018 06:48:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751176AbeEAGsv (ORCPT ); Tue, 1 May 2018 02:48:51 -0400 Received: from mail-pg0-f68.google.com ([74.125.83.68]:41726 "EHLO mail-pg0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751083AbeEAGsu (ORCPT ); Tue, 1 May 2018 02:48:50 -0400 Received: by mail-pg0-f68.google.com with SMTP id m21-v6so7853497pgv.8 for ; Mon, 30 Apr 2018 23:48:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id; bh=vywgyETshoCnaERxMcqMiHYzSHhLRaB0S7Gi3gbGmV8=; b=mrVl7ugGS2HELpOdZ/9tLTHer4+bUl/+yWPgjqlRdp5doTgngEy2wykQTTLpOB252B YBxon1b0BkvT0HMnpwkft/nQlLWe5wra8pDUjtfUBM6AkY139IRv1FFjmZ0GDk+LNUba k+2OxOZysuj9y3eGLP4dKjFfWHz4KCLIjVp7ocVNumvp3OqBbarR/aH230vsiYRupeba 6opIo6Bj/3gFB5cb09wI0dIOeShqTZBWfN4z9nyabgQMqakt0SJHzm5cbYw0ZnBWj84x kM7Px8Wcj/1jsK5EqoGKT5APDzl7nzjB3m0zKB7zgwGF15mZvcNjt2KeFT7N/WN3/r75 XaWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id; bh=vywgyETshoCnaERxMcqMiHYzSHhLRaB0S7Gi3gbGmV8=; b=K4PkFaSwH9s7VZMwtPHFnp+k/q9X+ChTJ2ld+DVWt1xe87Wr8q5MYJt+fymuwsaXF0 Q/TZj0bmp9afLSY1Cd9v+zUmQJWk2f8yTXQibL3/xIdEYrY8zUlzUFdQnz1Bl9fd6RYQ dkj+PknZJBDnghv5XC1UUsqFef1JhGGw9zAh4AbQ4RhmnbODmRRb3u2QDqwO6lSdWhzD G+0KSF5aoU3v+S4+IXvN2tX6eaqx6n4NzC6lvGX287TIcaYgwaFAb2S+BWowhck/Kv1U xpvcngX4v/VtgxQsDBM1vkJXf/KLQK8E5XvpJPPzzAQK5bc95vPUNSw0V2KocjnHwWTa eTMw== X-Gm-Message-State: ALQs6tAVSdkuMblrvV7ous8Ww7KHsQJoKHgx3H1rTfo1F9jvp2gTEjnr sJXI6pO7HBxnzo33Openq7Bx6DMO X-Google-Smtp-Source: AB8JxZq7v74qGiehcuUazjnh0tEhUa3UW1c1bC1tfOOsIBuGiRCndrkghqjqTwhpbRTHr3D7E8ju4w== X-Received: by 10.98.222.2 with SMTP id h2mr12681291pfg.205.1525157329248; Mon, 30 Apr 2018 23:48:49 -0700 (PDT) Received: from localhost.localdomain (c-73-140-212-29.hsd1.wa.comcast.net. [73.140.212.29]) by smtp.gmail.com with ESMTPSA id d8sm17566353pfh.177.2018.04.30.23.48.48 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 30 Apr 2018 23:48:48 -0700 (PDT) From: Andrei Vagin To: "Nicholas A. Bellinger" Cc: target-devel@vger.kernel.org, Andrei Vagin Subject: [PATCH] target: allow to set a blkio cgroup for a backstore Date: Mon, 30 Apr 2018 23:48:42 -0700 Message-Id: <20180501064842.4253-1-avagin@openvz.org> X-Mailer: git-send-email 2.14.3 Sender: target-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The Block I/O (blkio) subsystem controls and monitors access to I/O on block devices by tasks in cgroups. With the introduced changes, a backstore will be like a task in a specified group. One of interesting feature is an ability to set limits on a number of I/O operations and bytes per seconds. A new attribute is added for backstores, it is called blkio_cgroup. If we write 1 to the attribute file, a blkio cgroup from the current process is attached to the backstore. If we write 0 to the attribute file, a current group will be detached from the backstore. When we know a blkio cgroup the only thing, what we need to do to make it work, is to set this group for bio-s. How to use: # Create a test backstore $ targetcli targetcli shell version 2.1.fb46 Copyright 2011-2013 by Datera, Inc and others. /backstores/block> create dev=/dev/loop0 loop0 Created block storage object loop0 using /dev/loop0. /backstores/block> cd /loopback /loopback> create Created target naa.50014056fd3f341c. /loopback> cd naa.50014056fd3f341c/luns /loopback/naa...fd3f341c/luns> create /backstores/block/loop0 Created LUN 0. /loopback/naa...fd3f341c/luns> exit # Create a test cgroup and set it to a test backstore $ CG_PATH=/sys/fs/cgroup/blkio/test $ BS_PATH=/sys/kernel/config/target/core/iblock_0/loop0/attrib/blkio_cgroup $ mkdir -p $CG_PATH $ bash -c "echo 0 > $CG_PATH/tasks && echo 1 > $BS_PATH" $ cat $BS_PATH /test # Set 6 MB/sec for the backstore $ echo "7:0 6291456" > $CG_PATH/blkio.throttle.read_bps_device # Check that everything work as expected $ dd if=/dev/sda of=/dev/null iflag=direct bs=1M count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 16.6958 s, 6.3 MB/s Signed-off-by: Andrei Vagin --- drivers/target/target_core_configfs.c | 55 +++++++++++++++++++++++++++++++++++ drivers/target/target_core_device.c | 6 ++++ drivers/target/target_core_iblock.c | 18 +++++++++++- include/target/target_core_base.h | 2 ++ 4 files changed, 80 insertions(+), 1 deletion(-) diff --git a/drivers/target/target_core_configfs.c b/drivers/target/target_core_configfs.c index 3f4bf126eed0..62155cbd07cc 100644 --- a/drivers/target/target_core_configfs.c +++ b/drivers/target/target_core_configfs.c @@ -1021,6 +1021,59 @@ static ssize_t queue_depth_store(struct config_item *item, return count; } +static ssize_t blkio_cgroup_show(struct config_item *item, char *page) +{ + struct se_dev_attrib *da = to_attrib(item); + struct se_device *dev = da->da_dev; + int rb; + + read_lock(&dev->dev_attrib_lock); + if (dev->dev_attrib.blk_css) { + rb = cgroup_path(dev->dev_attrib.blk_css->cgroup, + page, PAGE_SIZE - 1); + if (rb == 0) + rb = strlen(page); + page[rb] = '\n'; + page[rb + 1] = 0; + rb++; + } else + rb = 0; + read_unlock(&dev->dev_attrib_lock); + + return rb; +} + +static ssize_t blkio_cgroup_store(struct config_item *item, + const char *page, size_t count) +{ + struct se_dev_attrib *da = to_attrib(item); + struct se_device *dev = da->da_dev; + struct cgroup_subsys_state *css, *pcss; + int ret; + u32 val; + + ret = kstrtou32(page, 0, &val); + if (ret < 0) + return ret; + + if (val > 1) + return -EINVAL; + if (val == 1) + css = task_get_css(current, io_cgrp_id); + else + css = NULL; + + write_lock(&dev->dev_attrib_lock); + pcss = dev->dev_attrib.blk_css; + dev->dev_attrib.blk_css = css; + write_unlock(&dev->dev_attrib_lock); + + if (pcss) + css_put(pcss); + + return count; +} + static ssize_t optimal_sectors_store(struct config_item *item, const char *page, size_t count) { @@ -1128,6 +1181,7 @@ CONFIGFS_ATTR_RO(, hw_max_sectors); CONFIGFS_ATTR(, optimal_sectors); CONFIGFS_ATTR_RO(, hw_queue_depth); CONFIGFS_ATTR(, queue_depth); +CONFIGFS_ATTR(, blkio_cgroup); CONFIGFS_ATTR(, max_unmap_lba_count); CONFIGFS_ATTR(, max_unmap_block_desc_count); CONFIGFS_ATTR(, unmap_granularity); @@ -1168,6 +1222,7 @@ struct configfs_attribute *sbc_attrib_attrs[] = { &attr_optimal_sectors, &attr_hw_queue_depth, &attr_queue_depth, + &attr_blkio_cgroup, &attr_max_unmap_lba_count, &attr_max_unmap_block_desc_count, &attr_unmap_granularity, diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c index e27db4d45a9d..64274a5a1b32 100644 --- a/drivers/target/target_core_device.c +++ b/drivers/target/target_core_device.c @@ -773,6 +773,7 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name) INIT_LIST_HEAD(&dev->delayed_cmd_list); INIT_LIST_HEAD(&dev->state_list); INIT_LIST_HEAD(&dev->qf_cmd_list); + rwlock_init(&dev->dev_attrib_lock); spin_lock_init(&dev->execute_task_lock); spin_lock_init(&dev->delayed_cmd_lock); spin_lock_init(&dev->dev_reservation_lock); @@ -820,6 +821,7 @@ struct se_device *target_alloc_device(struct se_hba *hba, const char *name) dev->dev_attrib.unmap_zeroes_data = DA_UNMAP_ZEROES_DATA_DEFAULT; dev->dev_attrib.max_write_same_len = DA_MAX_WRITE_SAME_LEN; + dev->dev_attrib.blk_css = NULL; xcopy_lun = &dev->xcopy_lun; rcu_assign_pointer(xcopy_lun->lun_se_dev, dev); @@ -1080,6 +1082,10 @@ void target_free_device(struct se_device *dev) if (dev->transport->free_prot) dev->transport->free_prot(dev); + if (dev->dev_attrib.blk_css) + css_put(dev->dev_attrib.blk_css); + dev->dev_attrib.blk_css = NULL; + dev->transport->free_device(dev); } diff --git a/drivers/target/target_core_iblock.c b/drivers/target/target_core_iblock.c index 07c814c42648..b952eacb84ed 100644 --- a/drivers/target/target_core_iblock.c +++ b/drivers/target/target_core_iblock.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include @@ -694,6 +695,7 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, u32 sg_num = sgl_nents; unsigned bio_cnt; int i, op, op_flags = 0; + struct cgroup_subsys_state *blk_css = NULL; if (data_direction == DMA_TO_DEVICE) { struct iblock_dev *ib_dev = IBLOCK_DEV(dev); @@ -724,9 +726,17 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, return 0; } + read_lock(&dev->dev_attrib_lock); + blk_css = dev->dev_attrib.blk_css; + if (blk_css) + css_get(blk_css); + read_unlock(&dev->dev_attrib_lock); + bio = iblock_get_bio(cmd, block_lba, sgl_nents, op, op_flags); if (!bio) goto fail_free_ibr; + if (blk_css) + bio_associate_blkcg(bio, blk_css); bio_start = bio; bio_list_init(&list); @@ -752,6 +762,8 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, op_flags); if (!bio) goto fail_put_bios; + if (blk_css) + bio_associate_blkcg(bio, blk_css); refcount_inc(&ibr->pending); bio_list_add(&list, bio); @@ -774,9 +786,13 @@ iblock_execute_rw(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, return 0; fail_put_bios: - while ((bio = bio_list_pop(&list))) + while ((bio = bio_list_pop(&list))) { + bio_disassociate_task(bio); bio_put(bio); + } fail_free_ibr: + if (blk_css) + css_put(blk_css); kfree(ibr); fail: return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE; diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h index 9f9f5902af38..014bee5657b4 100644 --- a/include/target/target_core_base.h +++ b/include/target/target_core_base.h @@ -688,6 +688,7 @@ struct se_dev_attrib { u32 max_bytes_per_io; struct se_device *da_dev; struct config_group da_group; + struct cgroup_subsys_state *blk_css; }; struct se_port_stat_grps { @@ -777,6 +778,7 @@ struct se_device { atomic_t dev_ordered_sync; atomic_t dev_qf_count; u32 export_count; + rwlock_t dev_attrib_lock; spinlock_t delayed_cmd_lock; spinlock_t execute_task_lock; spinlock_t dev_reservation_lock;