From patchwork Fri Apr 15 11:32:05 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Bonzini X-Patchwork-Id: 8850461 Return-Path: X-Original-To: patchwork-qemu-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 9E208C0553 for ; Fri, 15 Apr 2016 11:40:20 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id B3D6720295 for ; Fri, 15 Apr 2016 11:40:19 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C656E20256 for ; Fri, 15 Apr 2016 11:40:18 +0000 (UTC) Received: from localhost ([::1]:60616 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ar26k-00037E-6G for patchwork-qemu-devel@patchwork.kernel.org; Fri, 15 Apr 2016 07:40:18 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55612) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ar1zC-00064w-Cb for qemu-devel@nongnu.org; Fri, 15 Apr 2016 07:32:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ar1z9-00006a-6Z for qemu-devel@nongnu.org; Fri, 15 Apr 2016 07:32:30 -0400 Received: from mail-wm0-x243.google.com ([2a00:1450:400c:c09::243]:36069) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ar1z8-00005H-S1 for qemu-devel@nongnu.org; Fri, 15 Apr 2016 07:32:27 -0400 Received: by mail-wm0-x243.google.com with SMTP id l6so5402985wml.3 for ; Fri, 15 Apr 2016 04:32:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=XdHH2dURiZ0lLwC08hPMGha6RBK0cYHpVme2q4ua+Es=; b=LCJTxaORAEUW2DZBlaoKTrdgHR/l/y8aXmlMg9xeTnfA53gdeG4GSIYDNejwhbDto9 6nA7DXvQtBa1liiFkcRGlPSQb+/4OXDcEKxRQpplQW7hIteqCtoLiyeA6XLIMw3ANlFC pXwpV89zZLvEkXs1RF2vnrwm5FOKp6y9zeRHo82bLETFvJPwM6Gd83p57Uc8BdXvJ8xh SSsAMHimOyaqmlPAdshdiBQz2akJ/lPcq95b0j8KHv/GljGAac1Tg1EKD36SAwKVgpXh 1uwwTXcnzwRq7qXq8jpV2Zvw8MBbFfKvEMCee1uRnUN+27ecQldJ5Uyb0BQLvq2CMCSX Iggw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=XdHH2dURiZ0lLwC08hPMGha6RBK0cYHpVme2q4ua+Es=; b=hhWxJrzM35A4pPFbYwCzQBBc7QwhwfAX6ls/kRPqbaKw7aRvNizDHSZqGehBg9e+T1 7ORVatdJhXpiB7ZP60sEgGBJ+aFnVi/WG3JZRU6q4hfMoEwSTp+LWs9Od36NJDRSgdRz uyJ5qfZaKx2UBPdL+i+xX6rZOLGNJq95GXqkrEDTcXZ8oKLJhGEUsILe/LEzziO9Phjx kz3gWUQQpvtrdFyo2IZIMgyQyWBTUBVds/abGjCJH1qORQDGH5kQiw754O5hDZ/JJcKC dtNU7YLajU4ZeEz4z14jA1tGtCpAXImTQBT4z7v8Gro/tUf4mybm/fzSH9pFQ8FbcMxp BHzQ== X-Gm-Message-State: AOPr4FW8wFAorWLYqoJ3SobdSZRZJg7GgiaoTD+OGSBBKSSCkHonfMwZnl3G3pqPKoIhwQ== X-Received: by 10.194.68.138 with SMTP id w10mr21276985wjt.28.1460719946288; Fri, 15 Apr 2016 04:32:26 -0700 (PDT) Received: from donizetti.redhat.com (94-39-141-76.adsl-ull.clienti.tiscali.it. [94.39.141.76]) by smtp.gmail.com with ESMTPSA id qt3sm15217493wjc.32.2016.04.15.04.32.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 15 Apr 2016 04:32:25 -0700 (PDT) From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Fri, 15 Apr 2016 13:32:05 +0200 Message-Id: <1460719926-12950-11-git-send-email-pbonzini@redhat.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1460719926-12950-1-git-send-email-pbonzini@redhat.com> References: <1460719926-12950-1-git-send-email-pbonzini@redhat.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a00:1450:400c:c09::243 Subject: [Qemu-devel] [PATCH 10/11] coroutine-lock: add mutex argument to CoQueue APIs X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, berto@igalia.com, famz@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, T_DKIM_INVALID, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP All that CoQueue needs in order to become thread-safe is help from an external mutex. Add this to the API. Signed-off-by: Paolo Bonzini --- block/backup.c | 2 +- block/io.c | 2 +- block/qcow2-cluster.c | 4 +--- block/sheepdog.c | 2 +- block/throttle-groups.c | 2 +- hw/9pfs/9p.c | 2 +- include/qemu/coroutine.h | 8 +++++--- util/qemu-coroutine-lock.c | 24 +++++++++++++++++++++--- 8 files changed, 32 insertions(+), 14 deletions(-) diff --git a/block/backup.c b/block/backup.c index 11949f1..db0231f 100644 --- a/block/backup.c +++ b/block/backup.c @@ -68,7 +68,7 @@ static void coroutine_fn wait_for_overlapping_requests(BackupBlockJob *job, retry = false; QLIST_FOREACH(req, &job->inflight_reqs, list) { if (end > req->start && start < req->end) { - qemu_co_queue_wait(&req->wait_queue); + qemu_co_queue_wait(&req->wait_queue, NULL); retry = true; break; } diff --git a/block/io.c b/block/io.c index c6ea980..279d9dc 100644 --- a/block/io.c +++ b/block/io.c @@ -541,7 +541,7 @@ static bool coroutine_fn wait_serialising_requests(BdrvTrackedRequest *self) * (instead of producing a deadlock in the former case). */ if (!req->waiting_for) { self->waiting_for = req; - qemu_co_queue_wait(&req->wait_queue); + qemu_co_queue_wait(&req->wait_queue, NULL); self->waiting_for = NULL; retry = true; waited = true; diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 31ecc10..42bab15 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -927,9 +927,7 @@ static int handle_dependencies(BlockDriverState *bs, uint64_t guest_offset, if (bytes == 0) { /* Wait for the dependency to complete. We need to recheck * the free/allocated clusters when we continue. */ - qemu_co_mutex_unlock(&s->lock); - qemu_co_queue_wait(&old_alloc->dependent_requests); - qemu_co_mutex_lock(&s->lock); + qemu_co_queue_wait(&old_alloc->dependent_requests, &s->lock); return -EAGAIN; } } diff --git a/block/sheepdog.c b/block/sheepdog.c index 790541f..753ae59 100644 --- a/block/sheepdog.c +++ b/block/sheepdog.c @@ -2165,7 +2165,7 @@ static void wait_for_overlapping_aiocb(BDRVSheepdogState *s, SheepdogAIOCB *aioc retry: QLIST_FOREACH(cb, &s->inflight_aiocb_head, aiocb_siblings) { if (AIOCBOverlapping(aiocb, cb)) { - qemu_co_queue_wait(&s->overlapping_queue); + qemu_co_queue_wait(&s->overlapping_queue, NULL); goto retry; } } diff --git a/block/throttle-groups.c b/block/throttle-groups.c index 53e910e..5630606 100644 --- a/block/throttle-groups.c +++ b/block/throttle-groups.c @@ -302,7 +302,7 @@ void coroutine_fn throttle_group_co_io_limits_intercept(BlockDriverState *bs, if (must_wait || bs->pending_reqs[is_write]) { bs->pending_reqs[is_write]++; qemu_mutex_unlock(&tg->lock); - qemu_co_queue_wait(&bs->throttled_reqs[is_write]); + qemu_co_queue_wait(&bs->throttled_reqs[is_write], NULL); qemu_mutex_lock(&tg->lock); bs->pending_reqs[is_write]--; } diff --git a/hw/9pfs/9p.c b/hw/9pfs/9p.c index f5e3012..5f077e8 100644 --- a/hw/9pfs/9p.c +++ b/hw/9pfs/9p.c @@ -2282,7 +2282,7 @@ static void v9fs_flush(void *opaque) /* * Wait for pdu to complete. */ - qemu_co_queue_wait(&cancel_pdu->complete); + qemu_co_queue_wait(&cancel_pdu->complete, NULL); cancel_pdu->cancelled = 0; pdu_free(cancel_pdu); } diff --git a/include/qemu/coroutine.h b/include/qemu/coroutine.h index e8e3431..25e31a1 100644 --- a/include/qemu/coroutine.h +++ b/include/qemu/coroutine.h @@ -159,7 +159,8 @@ void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex); /** * CoQueues are a mechanism to queue coroutines in order to continue executing - * them later. + * them later. They are similar to condition variables, but they need help + * from an external mutex in order to maintain thread-safety. */ typedef struct CoQueue { QSIMPLEQ_HEAD(, Coroutine) entries; @@ -173,9 +174,10 @@ void qemu_co_queue_init(CoQueue *queue); /** * Adds the current coroutine to the CoQueue and transfers control to the - * caller of the coroutine. + * caller of the coroutine. The mutex is unlocked during the wait and + * locked again afterwards. */ -void coroutine_fn qemu_co_queue_wait(CoQueue *queue); +void coroutine_fn qemu_co_queue_wait(CoQueue *queue, CoMutex *mutex); /** * Restarts the next coroutine in the CoQueue and removes it from the queue. diff --git a/util/qemu-coroutine-lock.c b/util/qemu-coroutine-lock.c index aa59e82..828d79a 100644 --- a/util/qemu-coroutine-lock.c +++ b/util/qemu-coroutine-lock.c @@ -46,13 +46,31 @@ void qemu_co_queue_init(CoQueue *queue) QSIMPLEQ_INIT(&queue->entries); } -void coroutine_fn qemu_co_queue_wait(CoQueue *queue) +void coroutine_fn qemu_co_queue_wait(CoQueue *queue, CoMutex *mutex) { Coroutine *self = qemu_coroutine_self(); self->ctx = qemu_get_current_aio_context(); QSIMPLEQ_INSERT_TAIL(&queue->entries, self, co_queue_next); + + if (mutex) { + qemu_co_mutex_unlock(mutex); + } + + /* There is no race condition here. Other threads will call + * aio_co_schedule on our AioContext, which can reenter this + * coroutine but only after this yield and after the main loop + * has gone through the next iteration. + */ qemu_coroutine_yield(); assert(qemu_in_coroutine()); + + /* TODO: OSv implements wait morphing here, where the wakeup + * primitive automatically places the woken coroutine on the + * mutex's queue. This avoids the thundering herd effect. + */ + if (mutex) { + qemu_co_mutex_lock(mutex); + } } /** @@ -309,7 +327,7 @@ void qemu_co_rwlock_init(CoRwlock *lock) void qemu_co_rwlock_rdlock(CoRwlock *lock) { while (lock->writer) { - qemu_co_queue_wait(&lock->queue); + qemu_co_queue_wait(&lock->queue, NULL); } lock->reader++; } @@ -333,7 +351,7 @@ void qemu_co_rwlock_unlock(CoRwlock *lock) void qemu_co_rwlock_wrlock(CoRwlock *lock) { while (lock->writer || lock->reader) { - qemu_co_queue_wait(&lock->queue); + qemu_co_queue_wait(&lock->queue, NULL); } lock->writer = true; }