From patchwork Sun Feb 14 17:17:04 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Bonzini X-Patchwork-Id: 8302381 Return-Path: X-Original-To: patchwork-qemu-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 2F92DC02AA for ; Sun, 14 Feb 2016 17:17:39 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 31FE7203A9 for ; Sun, 14 Feb 2016 17:17:38 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2B5DB20270 for ; Sun, 14 Feb 2016 17:17:37 +0000 (UTC) Received: from localhost ([::1]:52191 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aV0Ii-0006pj-6x for patchwork-qemu-devel@patchwork.kernel.org; Sun, 14 Feb 2016 12:17:36 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34635) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aV0IX-0006pJ-Ki for qemu-devel@nongnu.org; Sun, 14 Feb 2016 12:17:26 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aV0IT-000341-Tx for qemu-devel@nongnu.org; Sun, 14 Feb 2016 12:17:25 -0500 Received: from mail-wm0-x244.google.com ([2a00:1450:400c:c09::244]:35802) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aV0IT-00033x-JG for qemu-devel@nongnu.org; Sun, 14 Feb 2016 12:17:21 -0500 Received: by mail-wm0-x244.google.com with SMTP id g62so11957793wme.2 for ; Sun, 14 Feb 2016 09:17:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=sqM2Dd0iq9ko5CA6NZ728TI2XKgUux6Op/g+29P6fwI=; b=JuCKRmchZTq6RXxeFynbvKI1nCr5EFMSQmYiU5yVErZCajYXT1HLKlwheuDjzeb/US MTvaAgBgADADWfRhy68aExl4wqgfBaXTPhVqOKp8u3q8Slati96Noidywcw3NOAU79F8 gW1XlDH3fhSvFb3TB8mbMp/WN5VPSdlZQO9wQWX85WRTiDAkTv9XPd2Yed0GaMkjDFAB k8R+ilD8TM/yHuLeiPwHs2HR+Q5hMfothCeuKINKnLcMHgYiDQz0wPSCdVRyk0WEWpfb OF8FbfC1wpNPDeP8OFjV/Aq0PsBFKU6sFqmqeBsqxFCm6QLBIbNPu88RlQMIYea6YiV0 Kz1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=sqM2Dd0iq9ko5CA6NZ728TI2XKgUux6Op/g+29P6fwI=; b=HusgGbc+avOEal3l5s65LkBHKilnj0v6pMc4FU7U4jDhGdcsv0sk7EqoDoQUoK4U/P zJQN8Fp81Kfz16lE1OCbwVrSf0L4UFbEymaNh/FvMyEobTPMipcziHdL6TUIb42+t/GT uQRpKpdtA6IRrDwoED7CNw1HE68m9PnsWtTKfDrlLvGx2u2z2vDMvyCscuUGGL0YXE0x brFHHSBl0ZzBJFjf3SujJcdzE4TAUbk72Nff6bBsnpO9GtHHEEphwA7HK1M0w9PH8eZP S608Bh1a60cHgFpbBMC9EE+PTLo+1QtRfLuiH5THCC119loA6IK/W6L4L0Jn60tHnl/j oP0Q== X-Gm-Message-State: AG10YOR5WDzHtggIkN/rwgXbdaG5ICuZ2f5Zu2F1zI0+p95fDEZf1gPE4Q7FiNsbsO127w== X-Received: by 10.194.90.77 with SMTP id bu13mr7810196wjb.60.1455470241024; Sun, 14 Feb 2016 09:17:21 -0800 (PST) Received: from donizetti.lan (94-39-152-251.adsl-ull.clienti.tiscali.it. [94.39.152.251]) by smtp.gmail.com with ESMTPSA id ko2sm21426149wjc.9.2016.02.14.09.17.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 14 Feb 2016 09:17:20 -0800 (PST) From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Sun, 14 Feb 2016 18:17:04 +0100 Message-Id: <1455470231-5223-2-git-send-email-pbonzini@redhat.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1455470231-5223-1-git-send-email-pbonzini@redhat.com> References: <1455470231-5223-1-git-send-email-pbonzini@redhat.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a00:1450:400c:c09::244 Cc: stefanha@redhat.com, mst@redhat.com Subject: [Qemu-devel] [PATCH 1/8] block-migration: acquire AioContext as necessary X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, T_DKIM_INVALID, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This is needed because dataplane will run during block migration as well. The block device migration code is quite liberal in taking the iothread mutex. For simplicity, keep it the same way, even though one could actually choose between the BQL (for regular BlockDriverStates) and the AioContext (for dataplane BlockDriverStates). When the block layer is made fully thread safe, aio_context_acquire shall go away altogether. Signed-off-by: Paolo Bonzini --- migration/block.c | 61 ++++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 49 insertions(+), 12 deletions(-) diff --git a/migration/block.c b/migration/block.c index a444058..6dd2327 100644 --- a/migration/block.c +++ b/migration/block.c @@ -60,9 +60,15 @@ typedef struct BlkMigDevState { int64_t cur_sector; int64_t cur_dirty; - /* Protected by block migration lock. */ + /* Data in the aio_bitmap is protected by block migration lock. + * Allocation and free happen during setup and cleanup respectively. + */ unsigned long *aio_bitmap; + + /* Protected by block migration lock. */ int64_t completed_sectors; + + /* Protected by iothread lock / AioContext. */ BdrvDirtyBitmap *dirty_bitmap; Error *blocker; } BlkMigDevState; @@ -100,7 +106,7 @@ typedef struct BlkMigState { int prev_progress; int bulk_completed; - /* Lock must be taken _inside_ the iothread lock. */ + /* Lock must be taken _inside_ the iothread lock and any AioContexts. */ QemuMutex lock; } BlkMigState; @@ -264,11 +270,13 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds) if (bmds->shared_base) { qemu_mutex_lock_iothread(); + aio_context_acquire(bdrv_get_aio_context(bs)); while (cur_sector < total_sectors && !bdrv_is_allocated(bs, cur_sector, MAX_IS_ALLOCATED_SEARCH, &nr_sectors)) { cur_sector += nr_sectors; } + aio_context_release(bdrv_get_aio_context(bs)); qemu_mutex_unlock_iothread(); } @@ -302,11 +310,21 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds) block_mig_state.submitted++; blk_mig_unlock(); + /* We do not know if bs is under the main thread (and thus does + * not acquire the AioContext when doing AIO) or rather under + * dataplane. Thus acquire both the iothread mutex and the + * AioContext. + * + * This is ugly and will disappear when we make bdrv_* thread-safe, + * without the need to acquire the AioContext. + */ qemu_mutex_lock_iothread(); + aio_context_acquire(bdrv_get_aio_context(bmds->bs)); blk->aiocb = bdrv_aio_readv(bs, cur_sector, &blk->qiov, nr_sectors, blk_mig_read_cb, blk); bdrv_reset_dirty_bitmap(bmds->dirty_bitmap, cur_sector, nr_sectors); + aio_context_release(bdrv_get_aio_context(bmds->bs)); qemu_mutex_unlock_iothread(); bmds->cur_sector = cur_sector + nr_sectors; @@ -321,8 +339,9 @@ static int set_dirty_tracking(void) int ret; QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) { + /* Creating/dropping dirty bitmaps only requires the big QEMU lock. */ bmds->dirty_bitmap = bdrv_create_dirty_bitmap(bmds->bs, BLOCK_SIZE, NULL, NULL); if (!bmds->dirty_bitmap) { ret = -errno; goto fail; @@ -332,11 +352,14 @@ static int set_dirty_tracking(void) return ret; } +/* Called with iothread lock taken. */ + static void unset_dirty_tracking(void) { BlkMigDevState *bmds; QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) { + /* Creating/dropping dirty bitmaps only requires the big QEMU lock. */ bdrv_release_dirty_bitmap(bmds->bs, bmds->dirty_bitmap); } } @@ -444,7 +470,7 @@ static void blk_mig_reset_dirty_cursor(void) } } -/* Called with iothread lock taken. */ +/* Called with iothread lock and AioContext taken. */ static int mig_save_device_dirty(QEMUFile *f, BlkMigDevState *bmds, int is_async) @@ -527,7 +553,9 @@ static int blk_mig_save_dirty_block(QEMUFile *f, int is_async) int ret = 1; QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) { + aio_context_acquire(bdrv_get_aio_context(bmds->bs)); ret = mig_save_device_dirty(f, bmds, is_async); + aio_context_release(bdrv_get_aio_context(bmds->bs)); if (ret <= 0) { break; } @@ -585,7 +613,9 @@ static int64_t get_remaining_dirty(void) int64_t dirty = 0; QSIMPLEQ_FOREACH(bmds, &block_mig_state.bmds_list, entry) { + aio_context_acquire(bdrv_get_aio_context(bmds->bs)); dirty += bdrv_get_dirty_count(bmds->dirty_bitmap); + aio_context_release(bdrv_get_aio_context(bmds->bs)); } return dirty << BDRV_SECTOR_BITS; @@ -597,21 +627,28 @@ static void block_migration_cleanup(void *opaque) { BlkMigDevState *bmds; BlkMigBlock *blk; + AioContext *ctx; bdrv_drain_all(); unset_dirty_tracking(); - blk_mig_lock(); while ((bmds = QSIMPLEQ_FIRST(&block_mig_state.bmds_list)) != NULL) { QSIMPLEQ_REMOVE_HEAD(&block_mig_state.bmds_list, entry); bdrv_op_unblock_all(bmds->bs, bmds->blocker); error_free(bmds->blocker); + + /* Save ctx, because bmds->bs can disappear during bdrv_unref. */ + ctx = bdrv_get_aio_context(bmds->bs); + aio_context_acquire(ctx); bdrv_unref(bmds->bs); + aio_context_release(ctx); + g_free(bmds->aio_bitmap); g_free(bmds); } + blk_mig_lock(); while ((blk = QSIMPLEQ_FIRST(&block_mig_state.blk_list)) != NULL) { QSIMPLEQ_REMOVE_HEAD(&block_mig_state.blk_list, entry); g_free(blk->buf); @@ -633,13 +670,12 @@ static int block_save_setup(QEMUFile *f, void *opaque) /* start track dirty blocks */ ret = set_dirty_tracking(); + qemu_mutex_unlock_iothread(); + if (ret) { - qemu_mutex_unlock_iothread(); return ret; } - qemu_mutex_unlock_iothread(); - ret = flush_blks(f); blk_mig_reset_dirty_cursor(); qemu_put_be64(f, BLK_MIG_FLAG_EOS); @@ -761,17 +797,18 @@ static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size, uint64_t pending; qemu_mutex_lock_iothread(); + pending = get_remaining_dirty(); + qemu_mutex_unlock_iothread(); + blk_mig_lock(); - pending = get_remaining_dirty() + - block_mig_state.submitted * BLOCK_SIZE + - block_mig_state.read_done * BLOCK_SIZE; + pending += block_mig_state.submitted * BLOCK_SIZE + + block_mig_state.read_done * BLOCK_SIZE; + blk_mig_unlock(); /* Report at least one block pending during bulk phase */ if (pending <= max_size && !block_mig_state.bulk_completed) { pending = max_size + BLOCK_SIZE; } - blk_mig_unlock(); - qemu_mutex_unlock_iothread(); DPRINTF("Enter save live pending %" PRIu64 "\n", pending); /* We don't do postcopy */