From patchwork Tue Jul 19 12:27:43 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Pen X-Patchwork-Id: 9236921 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 26B0960574 for ; Tue, 19 Jul 2016 12:32:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 16E7B1FF21 for ; Tue, 19 Jul 2016 12:32:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0AC61205A4; Tue, 19 Jul 2016 12:32:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id C1D001FF21 for ; Tue, 19 Jul 2016 12:32:00 +0000 (UTC) Received: from localhost ([::1]:55553 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bPUBr-00039T-O4 for patchwork-qemu-devel@patchwork.kernel.org; Tue, 19 Jul 2016 08:31:59 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36205) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bPU81-0006Sl-OP for qemu-devel@nongnu.org; Tue, 19 Jul 2016 08:28:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bPU80-0005fO-CI for qemu-devel@nongnu.org; Tue, 19 Jul 2016 08:28:01 -0400 Received: from mail-wm0-x22d.google.com ([2a00:1450:400c:c09::22d]:36301) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bPU80-0005es-3Z for qemu-devel@nongnu.org; Tue, 19 Jul 2016 08:28:00 -0400 Received: by mail-wm0-x22d.google.com with SMTP id q128so17583612wma.1 for ; Tue, 19 Jul 2016 05:27:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=profitbricks-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=iYVOOXBofADMBI0bJ+uHwHsaR/oqLRmfkREfba3jKR0=; b=ghvz9l8Pfal4QVJqO+yse3USEeYM/aCVNr5CuDrSMReDDvpN6ke5h1LCGUeut9+pJ+ JbqX3YnpYFlMLFvb8wx0u89zaCk/N4phFCbRj+0vu0P3EqVg/KraErm0soM0OHqSvzzH 3DT31bEPpPUhSjiUe4U0petbcnV1owZlBGysJYpOOVYMf2ctRHA9rFfjZtKKrAd0kgQW GP2jf3wUM1lAjeJfTaNnK/mD3bfgN60+okt56zRQD9fYkHuDh/lAdfyeu2cdM93oqirh V2WNcyucsl+eq7aCjhbEvMRM9rbhVqAeM7n4IHEtCIUKcPhxDAWB6hy+FIZv5cKhtxA5 e8zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=iYVOOXBofADMBI0bJ+uHwHsaR/oqLRmfkREfba3jKR0=; b=K/AMNjkRzHorkDXSGfNKqeyFCfZcNcbopeBcvjpRgP+Ugss+IS2yhLfPvXptZ4etFY hV88/5HAKgbGDkCysXy5tvrEkIrqwQ/ITfpusPnpWvE6amBiv7VDD/ct6RElTYK0v6AX 404OGBp8q9CjlprhgbIIgWtdTyfXvU2oG4FMqiV8+ghuXqGWVyMqVlQ4EjZ5aFzj+D9C JEaia0UkBjpAJT+aqbtqrfAiIhZeq19EKyCtm8RjnQ+nVcf4bgJxBBSNhGdTrufkAjwL y71WF0WJsAVuLExfWxUAd+rZoVSBrPSMvvTauLZQMYoIqK785YlPh3QgNtkr836gyvEi ISiw== X-Gm-Message-State: ALyK8tInDg/6ryg1Nq/iP85fOrwLbf9Wc3onkcK4RIyrctaYSciZ1H7RTGFg8JHEPKPOHIYO X-Received: by 10.28.141.4 with SMTP id p4mr4198004wmd.46.1468931279423; Tue, 19 Jul 2016 05:27:59 -0700 (PDT) Received: from pb.pb.local ([62.217.45.26]) by smtp.gmail.com with ESMTPSA id d62sm22581855wmd.7.2016.07.19.05.27.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 19 Jul 2016 05:27:58 -0700 (PDT) From: Roman Pen To: Date: Tue, 19 Jul 2016 14:27:43 +0200 Message-Id: <1468931263-32667-4-git-send-email-roman.penyaev@profitbricks.com> X-Mailer: git-send-email 2.8.2 In-Reply-To: <1468931263-32667-1-git-send-email-roman.penyaev@profitbricks.com> References: <1468931263-32667-1-git-send-email-roman.penyaev@profitbricks.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a00:1450:400c:c09::22d Subject: [Qemu-devel] [PATCH v2 3/3] linux-aio: process completions from ioq_submit() X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , Stefan Hajnoczi , Roman Pen , qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP In order to reduce completion latency it makes sense to harvest completed requests ASAP. Very fast backend device can complete requests just after submission, so it is worth trying to check ring buffer in order to peek completed requests directly after io_submit() has been called. Indeed, this patch reduces the completions latencies and increases the overall throughput, e.g. the following is the percentiles of number of completed requests at once: 1th 10th 20th 30th 40th 50th 60th 70th 80th 90th 99.99th Before 2 4 42 112 128 128 128 128 128 128 128 After 1 1 4 14 33 45 47 48 50 51 108 That means, that before the current patch is applied the ring buffer is observed as full (128 requests were consumed at once) in 60% of calls. After patch is applied the distribution of number of completed requests is "smoother" and the queue (requests in-flight) is almost never full. The fio read results are the following (write results are almost the same and are not showed here): Before ------ job: (groupid=0, jobs=8): err= 0: pid=2227: Tue Jul 19 11:29:50 2016 Description : [Emulation of Storage Server Access Pattern] read : io=54681MB, bw=1822.7MB/s, iops=179779, runt= 30001msec slat (usec): min=172, max=16883, avg=338.35, stdev=109.66 clat (usec): min=1, max=21977, avg=1051.45, stdev=299.29 lat (usec): min=317, max=22521, avg=1389.83, stdev=300.73 clat percentiles (usec): | 1.00th=[ 346], 5.00th=[ 596], 10.00th=[ 708], 20.00th=[ 852], | 30.00th=[ 932], 40.00th=[ 996], 50.00th=[ 1048], 60.00th=[ 1112], | 70.00th=[ 1176], 80.00th=[ 1256], 90.00th=[ 1384], 95.00th=[ 1496], | 99.00th=[ 1800], 99.50th=[ 1928], 99.90th=[ 2320], 99.95th=[ 2672], | 99.99th=[ 4704] bw (KB /s): min=205229, max=553181, per=12.50%, avg=233278.26, stdev=18383.51 After ------ job: (groupid=0, jobs=8): err= 0: pid=2220: Tue Jul 19 11:31:51 2016 Description : [Emulation of Storage Server Access Pattern] read : io=57637MB, bw=1921.2MB/s, iops=189529, runt= 30002msec slat (usec): min=169, max=20636, avg=329.61, stdev=124.18 clat (usec): min=2, max=19592, avg=988.78, stdev=251.04 lat (usec): min=381, max=21067, avg=1318.42, stdev=243.58 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 580], 10.00th=[ 748], 20.00th=[ 876], | 30.00th=[ 908], 40.00th=[ 948], 50.00th=[ 1012], 60.00th=[ 1064], | 70.00th=[ 1080], 80.00th=[ 1128], 90.00th=[ 1224], 95.00th=[ 1288], | 99.00th=[ 1496], 99.50th=[ 1608], 99.90th=[ 1960], 99.95th=[ 2256], | 99.99th=[ 5408] bw (KB /s): min=212149, max=390160, per=12.49%, avg=245746.04, stdev=11606.75 Throughput increased from 1822MB/s to 1921MB/s, average completion latencies decreased from 1051us to 988us. Signed-off-by: Roman Pen Cc: Stefan Hajnoczi Cc: Paolo Bonzini Cc: qemu-devel@nongnu.org --- block/linux-aio.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/block/linux-aio.c b/block/linux-aio.c index fae5c82..53fd10f 100644 --- a/block/linux-aio.c +++ b/block/linux-aio.c @@ -92,7 +92,11 @@ static void qemu_laio_process_completion(struct qemu_laiocb *laiocb) laiocb->ret = ret; if (laiocb->co) { - qemu_coroutine_enter(laiocb->co, NULL); + /* Jump and continue completion for foreign requests, don't do + * anything for current request, it will be completed shortly. */ + if (laiocb->co != qemu_coroutine_self()) { + qemu_coroutine_enter(laiocb->co, NULL); + } } else { laiocb->common.cb(laiocb->common.opaque, ret); qemu_aio_unref(laiocb); @@ -312,6 +316,19 @@ static void ioq_submit(LinuxAioState *s) QSIMPLEQ_SPLIT_AFTER(&s->io_q.pending, aiocb, next, &completed); } while (ret == len && !QSIMPLEQ_EMPTY(&s->io_q.pending)); s->io_q.blocked = (s->io_q.in_queue > 0); + + if (s->io_q.in_flight) { + /* We can try to complete something just right away if there are + * still requests in-flight. */ + qemu_laio_process_completions(s); + /* + * Even we have completed everything (in_flight == 0), the queue can + * have still pended requests (in_queue > 0). We do not attempt to + * repeat submission to avoid IO hang. The reason is simple: s->e is + * still set and completion callback will be called shortly and all + * pended requests will be submitted from there. + */ + } } void laio_io_plug(BlockDriverState *bs, LinuxAioState *s) @@ -370,6 +387,7 @@ int coroutine_fn laio_co_submit(BlockDriverState *bs, LinuxAioState *s, int fd, .co = qemu_coroutine_self(), .nbytes = qiov->size, .ctx = s, + .ret = -EINPROGRESS, .is_read = (type == QEMU_AIO_READ), .qiov = qiov, }; @@ -379,7 +397,9 @@ int coroutine_fn laio_co_submit(BlockDriverState *bs, LinuxAioState *s, int fd, return ret; } - qemu_coroutine_yield(); + if (laiocb.ret == -EINPROGRESS) { + qemu_coroutine_yield(); + } return laiocb.ret; }