From patchwork Thu Sep 22 11:59:19 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pradeep X-Patchwork-Id: 9345161 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 1B33B60757 for ; Thu, 22 Sep 2016 12:02:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0B6482AA43 for ; Thu, 22 Sep 2016 12:02:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F3EAB2AA48; Thu, 22 Sep 2016 12:02:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B17E52AA43 for ; Thu, 22 Sep 2016 12:02:10 +0000 (UTC) Received: from localhost ([::1]:42551 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bn2hd-0003QQ-PB for patchwork-qemu-devel@patchwork.kernel.org; Thu, 22 Sep 2016 08:02:09 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48793) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bn2gF-0002iW-IS for qemu-devel@nongnu.org; Thu, 22 Sep 2016 08:00:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bn2g9-0000qW-4E for qemu-devel@nongnu.org; Thu, 22 Sep 2016 08:00:42 -0400 Received: from mail-wm0-f66.google.com ([74.125.82.66]:34021) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bn2g8-0000qI-OA for qemu-devel@nongnu.org; Thu, 22 Sep 2016 08:00:36 -0400 Received: by mail-wm0-f66.google.com with SMTP id l132so13601777wmf.1 for ; Thu, 22 Sep 2016 05:00:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id; bh=laopXExAbvPYByYRR4QQIzcsGTPmsgXYmsuRf2v9n4c=; b=UeRPR4ogLPSSEba2IPkLAz/QOLgP1NAhfsAAFBCEaQ6iwUeoZn8uju0ubX2Pc4BuGw wPgpHkigN8C3VDaIDM+d32D/X/kpuuy4YXL9P/1Uv4e7kq/T+VZ1qF4Y0MgZlLdXOa9K nofnT1PluncgX9ALd9Am6QIeZdXua1Yfb4jrZH9l+03lk5IJDeQSJjBTFLMoBQ0PYSLx mpm0WotzQYmA0NCZONAdt4Gi7RdBURwaYt9jKAHJrKiJMNTMWpAkaaLfW0ijIZtfYPmR UBo9nKlA/Nt5lhCKgbmIci3sV2BWpdGokmOXRVEP+EzFDuprZ2UcitKPD82C0z2L9MKG edGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=laopXExAbvPYByYRR4QQIzcsGTPmsgXYmsuRf2v9n4c=; b=WXXaInq2CHuObWbF059F/vh8aVPoGBGauUtmTKNSKEZ/mCRM1XLT8YQlcu8e4W1plP U0vmA3bDYKze5CwkxrFq3U0ZqG7ojuSG0hc6CUGDN1o5S3UUAfThQXLQkEzezO8yLFdN pGsD9UXO832ohpzzf8SLv3/2nK6MEZgBSkgXNtsLAUHr76yAGXZvOf/SbCgWHpKCioke 4+7pJlxJeOFnR9kee0UYtO1mavdE43xEELbQkA9fVoHTZZBUheEMyY3f4h2kZfXOUUoQ lfjcy4TKnySE0LxBOdhiwdT9b5t702mDcCoOHW/ZaPtG27R+8bIS3ZDw9VYU9EVyrBkO 6Kig== X-Gm-Message-State: AA6/9RnEtEBFE8sKRrNOhAnpJhhYBimUAEId7AMIp2P2QGkOE9zk/QIT6rb2dwuuDzs6Hg== X-Received: by 10.194.101.226 with SMTP id fj2mr1958925wjb.0.1474545575562; Thu, 22 Sep 2016 04:59:35 -0700 (PDT) Received: from localhost.localdomain ([217.111.50.162]) by smtp.gmail.com with ESMTPSA id w138sm14851077wmd.1.2016.09.22.04.59.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 22 Sep 2016 04:59:34 -0700 (PDT) From: Pradeep Jagadeesh X-Google-Original-From: Pradeep Jagadeesh To: "Greg Kurz" , "Alberto Garcia" Date: Thu, 22 Sep 2016 07:59:19 -0400 Message-Id: <1474545559-23870-1-git-send-email-pradeep.jagadeesh@huawei.com> X-Mailer: git-send-email 1.8.3.1 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 74.125.82.66 Subject: [Qemu-devel] [PATCH v4] fsdev: add IO throttle support to fsdev devices X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-devel@nongnu.org, Claudio Fontana , Pradeep Jagadeesh , "Aneesh Kumar K.V" Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Uses throttling APIs to limit I/O bandwidth and number of operations on the devices which use 9p-local driver. Signed-off-by: Pradeep Jagadeesh --- fsdev/Makefile.objs | 1 + fsdev/file-op-9p.h | 3 + fsdev/qemu-fsdev-opts.c | 76 +++++++++++++++++++++++ fsdev/qemu-fsdev-throttle.c | 146 ++++++++++++++++++++++++++++++++++++++++++++ fsdev/qemu-fsdev-throttle.h | 36 +++++++++++ hw/9pfs/9p-local.c | 9 ++- hw/9pfs/9p.c | 6 ++ hw/9pfs/cofile.c | 3 + 8 files changed, 278 insertions(+), 2 deletions(-) create mode 100644 fsdev/qemu-fsdev-throttle.c create mode 100644 fsdev/qemu-fsdev-throttle.h This adds the support for the 9p-local driver. For now this functionality can be enabled only through qemu cli options. QMP interface and support to other drivers need further extensions. To make it simple for other drivers, the throttle code has been put in separate files. v1 -> v2: -Fixed FsContext redeclaration issue -Removed couple of function declarations from 9p-throttle.h -Fixed some of the .help messages v2 -> v3: -Addressed follwing comments by Claudio Fontana -Removed redundant memset calls in fsdev_throttle_configure_iolimits function -Checking throttle structure validity before initializing other structures in fsdev_throttle_configure_iolimits -Addressed following comments by Greg Kurz -Moved the code from 9pfs directory to fsdev directory, because the throttling is for the fsdev devices.Renamed the files and functions to fsdev_ from 9pfs_ -Renamed throttling cli options to throttling.*, as in QMP cli options -Removed some of the unwanted .h files from qemu-fsdev-throttle.[ch] -Using throttle_enabled() function to set the thottle enabled flag for fsdev. v3 -> v4: -Addressed following comments by Alberto Garcia -Removed the unwanted locking and other data structures in qemu-fsdev-throttle.[ch] -Addressed following comments by Greg Kurz -Removed fsdev_iolimitsenable/disable functions, instead using throttle_enabled function diff --git a/fsdev/Makefile.objs b/fsdev/Makefile.objs index 1b120a4..2c6da2d 100644 --- a/fsdev/Makefile.objs +++ b/fsdev/Makefile.objs @@ -7,6 +7,7 @@ common-obj-y = qemu-fsdev-dummy.o endif common-obj-y += qemu-fsdev-opts.o +common-obj-y += qemu-fsdev-throttle.o # Toplevel always builds this; targets without virtio will put it in # common-obj-y common-obj-$(CONFIG_ALL) += qemu-fsdev-dummy.o diff --git a/fsdev/file-op-9p.h b/fsdev/file-op-9p.h index 6db9fea..33fe822 100644 --- a/fsdev/file-op-9p.h +++ b/fsdev/file-op-9p.h @@ -17,6 +17,7 @@ #include #include #include +#include "qemu-fsdev-throttle.h" #define SM_LOCAL_MODE_BITS 0600 #define SM_LOCAL_DIR_MODE_BITS 0700 @@ -74,6 +75,7 @@ typedef struct FsDriverEntry { char *path; int export_flags; FileOperations *ops; + FsThrottle fst; } FsDriverEntry; typedef struct FsContext @@ -83,6 +85,7 @@ typedef struct FsContext int export_flags; struct xattr_operations **xops; struct extended_ops exops; + FsThrottle *fst; /* fs driver specific data */ void *private; } FsContext; diff --git a/fsdev/qemu-fsdev-opts.c b/fsdev/qemu-fsdev-opts.c index 1dd8c7a..395d497 100644 --- a/fsdev/qemu-fsdev-opts.c +++ b/fsdev/qemu-fsdev-opts.c @@ -37,6 +37,82 @@ static QemuOptsList qemu_fsdev_opts = { }, { .name = "sock_fd", .type = QEMU_OPT_NUMBER, + }, { + .name = "throttling.iops-total", + .type = QEMU_OPT_NUMBER, + .help = "limit total I/O operations per second", + },{ + .name = "throttling.iops-read", + .type = QEMU_OPT_NUMBER, + .help = "limit read operations per second", + },{ + .name = "throttling.iops-write", + .type = QEMU_OPT_NUMBER, + .help = "limit write operations per second", + },{ + .name = "throttling.bps-total", + .type = QEMU_OPT_NUMBER, + .help = "limit total bytes per second", + },{ + .name = "throttling.bps-read", + .type = QEMU_OPT_NUMBER, + .help = "limit read bytes per second", + },{ + .name = "throttling.bps-write", + .type = QEMU_OPT_NUMBER, + .help = "limit write bytes per second", + },{ + .name = "throttling.iops-total-max", + .type = QEMU_OPT_NUMBER, + .help = "I/O operations burst", + },{ + .name = "throttling.iops-read-max", + .type = QEMU_OPT_NUMBER, + .help = "I/O operations read burst", + },{ + .name = "throttling.iops-write-max", + .type = QEMU_OPT_NUMBER, + .help = "I/O operations write burst", + },{ + .name = "throttling.bps-total-max", + .type = QEMU_OPT_NUMBER, + .help = "total bytes burst", + },{ + .name = "throttling.bps-read-max", + .type = QEMU_OPT_NUMBER, + .help = "total bytes read burst", + },{ + .name = "throttling.bps-write-max", + .type = QEMU_OPT_NUMBER, + .help = "total bytes write burst", + },{ + .name = "throttling.iops-total-max-length", + .type = QEMU_OPT_NUMBER, + .help = "length of the iops-total-max burst period, in seconds", + },{ + .name = "throttling.iops-read-max-length", + .type = QEMU_OPT_NUMBER, + .help = "length of the iops-read-max burst period, in seconds", + },{ + .name = "throttling.iops-write-max-length", + .type = QEMU_OPT_NUMBER, + .help = "length of the iops-write-max burst period, in seconds", + },{ + .name = "throttling.bps-total-max-length", + .type = QEMU_OPT_NUMBER, + .help = "length of the bps-total-max burst period, in seconds", + },{ + .name = "throttling.bps-read-max-length", + .type = QEMU_OPT_NUMBER, + .help = "length of the bps-read-max burst period, in seconds", + },{ + .name = "throttling.bps-write-max-length", + .type = QEMU_OPT_NUMBER, + .help = "length of the bps-write-max burst period, in seconds", + },{ + .name = "throttling.iops-size", + .type = QEMU_OPT_NUMBER, + .help = "when limiting by iops max size of an I/O in bytes", }, { /*End of list */ } diff --git a/fsdev/qemu-fsdev-throttle.c b/fsdev/qemu-fsdev-throttle.c new file mode 100644 index 0000000..704c687 --- /dev/null +++ b/fsdev/qemu-fsdev-throttle.c @@ -0,0 +1,146 @@ +/* + * Fsdev Throttle + * + * Copyright (C) 2016 Huawei Technologies Duesseldorf GmbH + * + * Author: Pradeep Jagadeesh + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * (at your option) any later version. + * + * See the COPYING file in the top-level directory for details. + * + */ + +#include "qemu/osdep.h" +#include "qemu/error-report.h" +#include "qapi/error.h" +#include "qemu-fsdev-throttle.h" + +static bool fsdev_throttle_check_for_wait(FsThrottle *fst, bool is_write) +{ + return throttle_schedule_timer(&fst->ts, &fst->tt, is_write); +} + +static void fsdev_throttle_schedule_next_request(FsThrottle *fst, bool is_write) +{ + bool must_wait = fsdev_throttle_check_for_wait(fst, is_write); + if (!must_wait) { + if (qemu_in_coroutine() && + qemu_co_queue_next(&fst->throttled_reqs[is_write])) { + ; + } else { + int64_t now = qemu_clock_get_ns(fst->tt.clock_type); + timer_mod(fst->tt.timers[is_write], now + 1); + } + } +} + +static void fsdev_throttle_timer_cb(FsThrottle *fst, bool is_write) +{ + bool empty_queue; + + empty_queue = !qemu_co_enter_next(&fst->throttled_reqs[is_write]); + if (empty_queue) { + fsdev_throttle_schedule_next_request(fst, is_write); + } +} + +static void fsdev_throttle_read_timer_cb(void *opaque) +{ + fsdev_throttle_timer_cb(opaque, false); +} + +static void fsdev_throttle_write_timer_cb(void *opaque) +{ + fsdev_throttle_timer_cb(opaque, true); +} + +static void fsdev_parse_io_limit_opts(QemuOpts *opts, FsThrottle *fst) +{ + fst->cfg.buckets[THROTTLE_BPS_TOTAL].avg = + qemu_opt_get_number(opts, "throttling.bps-total", 0); + fst->cfg.buckets[THROTTLE_BPS_READ].avg = + qemu_opt_get_number(opts, "throttling.bps-read", 0); + fst->cfg.buckets[THROTTLE_BPS_WRITE].avg = + qemu_opt_get_number(opts, "throttling.bps-write", 0); + fst->cfg.buckets[THROTTLE_OPS_TOTAL].avg = + qemu_opt_get_number(opts, "throttling.iops-total", 0); + fst->cfg.buckets[THROTTLE_OPS_READ].avg = + qemu_opt_get_number(opts, "throttling.iops-read", 0); + fst->cfg.buckets[THROTTLE_OPS_WRITE].avg = + qemu_opt_get_number(opts, "throttling.iops-write", 0); + + fst->cfg.buckets[THROTTLE_BPS_TOTAL].max = + qemu_opt_get_number(opts, "throttling.bps-total-max", 0); + fst->cfg.buckets[THROTTLE_BPS_READ].max = + qemu_opt_get_number(opts, "throttling.bps-read-max", 0); + fst->cfg.buckets[THROTTLE_BPS_WRITE].max = + qemu_opt_get_number(opts, "throttling.bps-write-max", 0); + fst->cfg.buckets[THROTTLE_OPS_TOTAL].max = + qemu_opt_get_number(opts, "throttling.iops-total-max", 0); + fst->cfg.buckets[THROTTLE_OPS_READ].max = + qemu_opt_get_number(opts, "throttling.iops-read-max", 0); + fst->cfg.buckets[THROTTLE_OPS_WRITE].max = + qemu_opt_get_number(opts, "throttling.iops-write-max", 0); + + fst->cfg.buckets[THROTTLE_BPS_TOTAL].burst_length = + qemu_opt_get_number(opts, "throttling.bps-total-max-length", 1); + fst->cfg.buckets[THROTTLE_BPS_READ].burst_length = + qemu_opt_get_number(opts, "throttling.bps-read-max-length", 1); + fst->cfg.buckets[THROTTLE_BPS_WRITE].burst_length = + qemu_opt_get_number(opts, "throttling.bps-write-max-length", 1); + fst->cfg.buckets[THROTTLE_OPS_TOTAL].burst_length = + qemu_opt_get_number(opts, "throttling.iops-total-max-length", 1); + fst->cfg.buckets[THROTTLE_OPS_READ].burst_length = + qemu_opt_get_number(opts, "throttling.iops-read-max-length", 1); + fst->cfg.buckets[THROTTLE_OPS_WRITE].burst_length = + qemu_opt_get_number(opts, "throttling.iops-write-max-length", 1); + fst->cfg.op_size = + qemu_opt_get_number(opts, "throttling.iops-size", 0); +} + +int fsdev_throttle_configure_iolimits(QemuOpts *opts, FsThrottle *fst) +{ + Error *err = NULL; + + /* Parse and set populate the cfg structure */ + fsdev_parse_io_limit_opts(opts, fst); + + if (!throttle_is_valid(&fst->cfg, &err)) { + error_reportf_err(err, "Throttle configuration is not valid: "); + return -1; + } + if (throttle_enabled(&fst->cfg)) { + g_assert((fst->aioctx = qemu_get_aio_context())); + throttle_init(&fst->ts); + throttle_timers_init(&fst->tt, + fst->aioctx, + QEMU_CLOCK_REALTIME, + fsdev_throttle_read_timer_cb, + fsdev_throttle_write_timer_cb, + fst); + throttle_config(&fst->ts, &fst->tt, &fst->cfg); + qemu_co_queue_init(&fst->throttled_reqs[0]); + qemu_co_queue_init(&fst->throttled_reqs[1]); + } + return 0; +} + +void fsdev_throttle_request(FsThrottle *fst, bool is_write, ssize_t bytes) +{ + bool must_wait; + + if (throttle_enabled(&fst->cfg)) { + must_wait = fsdev_throttle_check_for_wait(fst, is_write); + if (must_wait) { + qemu_co_queue_wait(&fst->throttled_reqs[is_write]); + } + throttle_account(&fst->ts, is_write, bytes); + fsdev_throttle_schedule_next_request(fst, is_write); + } +} +void fsdev_throttle_cleanup(FsThrottle *fst) +{ + throttle_timers_destroy(&fst->tt); +} diff --git a/fsdev/qemu-fsdev-throttle.h b/fsdev/qemu-fsdev-throttle.h new file mode 100644 index 0000000..5ae452c --- /dev/null +++ b/fsdev/qemu-fsdev-throttle.h @@ -0,0 +1,36 @@ +/* + * Fsdev Throttle + * + * Copyright (C) 2016 Huawei Technologies Duesseldorf GmbH + * + * Author: Pradeep Jagadeesh + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * (at your option) any later version. + * + * See the COPYING file in the top-level directory for details. + * + */ + +#ifndef _FSDEV_THROTTLE_H +#define _FSDEV_THROTTLE_H + +#include "block/aio.h" +#include "qemu/main-loop.h" +#include "qemu/coroutine.h" +#include "qemu/throttle.h" + +typedef struct FsThrottle { + ThrottleState ts; + ThrottleTimers tt; + AioContext *aioctx; + ThrottleConfig cfg; + CoQueue throttled_reqs[2]; +} FsThrottle; + +int fsdev_throttle_configure_iolimits(QemuOpts *, FsThrottle *); + +void fsdev_throttle_request(FsThrottle *, bool , ssize_t); + +void fsdev_throttle_cleanup(FsThrottle *); +#endif /* _FSDEV_THROTTLE_H */ diff --git a/hw/9pfs/9p-local.c b/hw/9pfs/9p-local.c index 3f271fc..54459f2 100644 --- a/hw/9pfs/9p-local.c +++ b/hw/9pfs/9p-local.c @@ -436,8 +436,8 @@ static ssize_t local_pwritev(FsContext *ctx, V9fsFidOpenState *fs, const struct iovec *iov, int iovcnt, off_t offset) { - ssize_t ret -; + ssize_t ret; + #ifdef CONFIG_PREADV ret = pwritev(fs->fd, iov, iovcnt, offset); #else @@ -1240,6 +1240,11 @@ static int local_parse_opts(QemuOpts *opts, struct FsDriverEntry *fse) error_report("fsdev: No path specified"); return -1; } + + if (fsdev_throttle_configure_iolimits(opts, &fse->fst) < 0) { + return -1; + } + fse->path = g_strdup(path); return 0; diff --git a/hw/9pfs/9p.c b/hw/9pfs/9p.c index dfe293d..ccf6d0c 100644 --- a/hw/9pfs/9p.c +++ b/hw/9pfs/9p.c @@ -3490,6 +3490,9 @@ int v9fs_device_realize_common(V9fsState *s, Error **errp) error_setg(errp, "share path %s is not a directory", fse->path); goto out; } + + s->ctx.fst = &fse->fst; + v9fs_path_free(&path); rc = 0; @@ -3504,6 +3507,9 @@ out: void v9fs_device_unrealize_common(V9fsState *s, Error **errp) { + if (throttle_enabled(&s->ctx.fst->cfg)) { + fsdev_throttle_cleanup(s->ctx.fst); + } g_free(s->ctx.fs_root); g_free(s->tag); } diff --git a/hw/9pfs/cofile.c b/hw/9pfs/cofile.c index 10343c0..5b0ebe0 100644 --- a/hw/9pfs/cofile.c +++ b/hw/9pfs/cofile.c @@ -16,6 +16,7 @@ #include "qemu/thread.h" #include "qemu/coroutine.h" #include "coth.h" +#include "fsdev/qemu-fsdev-throttle.h" int v9fs_co_st_gen(V9fsPDU *pdu, V9fsPath *path, mode_t st_mode, V9fsStatDotl *v9stat) @@ -245,6 +246,7 @@ int v9fs_co_pwritev(V9fsPDU *pdu, V9fsFidState *fidp, if (v9fs_request_cancelled(pdu)) { return -EINTR; } + fsdev_throttle_request(s->ctx.fst, true, iov->iov_len); v9fs_co_run_in_worker( { err = s->ops->pwritev(&s->ctx, &fidp->fs, iov, iovcnt, offset); @@ -264,6 +266,7 @@ int v9fs_co_preadv(V9fsPDU *pdu, V9fsFidState *fidp, if (v9fs_request_cancelled(pdu)) { return -EINTR; } + fsdev_throttle_request(s->ctx.fst, false, iov->iov_len); v9fs_co_run_in_worker( { err = s->ops->preadv(&s->ctx, &fidp->fs, iov, iovcnt, offset);