From patchwork Thu Mar 31 15:08:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12797230 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 21A3EC433F5 for ; Thu, 31 Mar 2022 15:17:54 +0000 (UTC) Received: from localhost ([::1]:41794 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwYb-00006B-2j for qemu-devel@archiver.kernel.org; Thu, 31 Mar 2022 11:17:53 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52564) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQJ-0002Df-SA for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:19 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:47031) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQG-0001NI-Cg for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739355; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ffvhKWyfzJn2iowy5Yr0REXiZWOmBeW+MY7e6yM3urw=; b=DnRYmjfTcSbluyFE/EwQHrltckB2kg4NPVcEMkWl6jp7HWfDykazeGWOC5p8mw5163nL9L IaquoEcm1p69K+gJTUrIJ8mAJenujxh53i8ycyc4NU+hMA5+13+tC9qbI2grlGwMz1HJwv nAg5vr8Nfd6ZwDYL9Sz4Zy33mfF66ng= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-205-c4ZWL_8ePfG8aoXPb3926A-1; Thu, 31 Mar 2022 11:09:14 -0400 X-MC-Unique: c4ZWL_8ePfG8aoXPb3926A-1 Received: by mail-qk1-f197.google.com with SMTP id bk23-20020a05620a1a1700b0067b32f93b90so959269qkb.16 for ; Thu, 31 Mar 2022 08:09:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ffvhKWyfzJn2iowy5Yr0REXiZWOmBeW+MY7e6yM3urw=; b=kT1fqMeZLjdCE7e2GrnYVS2QYkPfL43gffUx0jWuCNrve/QCcwDIeDQbHlSGNsBmOJ wN6n03hRp2r47JPVAPWiI0DDL2vrHUuBLw8pccFCOdJrAxH0+A3C0s023Et2NLutyPV9 RwqhBcRI7NDtejFRLjSruvbFOvfWmp3IN7BgOW8tBum4gcQjDdBYGmsPqpTVff5JER5U 9uPHb3wggmsY1zuVEzGFZbShZNhbM6Dho4lHmTUzgVsGCNO3S4G3SMr9K1VoFqcX/zwY cQtcG+zlKL589SViAX9i74TKOJzx7WXEzqRbBVvNspC9YMbcUYcvm/QJxifZEehciTmm +Z3g== X-Gm-Message-State: AOAM532mz0X7KKbVYvNpgmJDwwBnVvpExcjhFHCI0jZYcoitbEEYfDa6 kzKLvMXbqirl807kudG8xZJ6/sQIM1TTchuJCW02ImtvP5qdFvBmjgBuRvLgELdskKAa/W7KYVQ LdHJztWpJ8pf0P+BJXLQypXbEoc4flKB7sFSTsx+uQpvkH7cQRT8vGtyzpSrY210I X-Received: by 2002:ad4:5dc9:0:b0:441:56ad:8d93 with SMTP id m9-20020ad45dc9000000b0044156ad8d93mr4392119qvh.76.1648739352454; Thu, 31 Mar 2022 08:09:12 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy4ovHAdBIFnfqHPXTJXqRjgZisFw4gfPbdMN/rUL+RJIGcnqZyg8O/fECZxaAEEsiECZBpHA== X-Received: by 2002:ad4:5dc9:0:b0:441:56ad:8d93 with SMTP id m9-20020ad45dc9000000b0044156ad8d93mr4392071qvh.76.1648739351966; Thu, 31 Mar 2022 08:09:11 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:11 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 12/19] migration: Create the postcopy preempt channel asynchronously Date: Thu, 31 Mar 2022 11:08:50 -0400 Message-Id: <20220331150857.74406-13-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This patch allows the postcopy preempt channel to be created asynchronously. The benefit is that when the connection is slow, we won't take the BQL (and potentially block all things like QMP) for a long time without releasing. A function postcopy_preempt_wait_channel() is introduced, allowing the migration thread to be able to wait on the channel creation. The channel is always created by the main thread, in which we'll kick a new semaphore to tell the migration thread that the channel has created. We'll need to wait for the new channel in two places: (1) when there's a new postcopy migration that is starting, or (2) when there's a postcopy migration to resume. For the start of migration, we don't need to wait for this channel until when we want to start postcopy, aka, postcopy_start(). We'll fail the migration if we found that the channel creation failed (which should probably not happen at all in 99% of the cases, because the main channel is using the same network topology). For a postcopy recovery, we'll need to wait in postcopy_pause(). In that case if the channel creation failed, we can't fail the migration or we'll crash the VM, instead we keep in PAUSED state, waiting for yet another recovery. Signed-off-by: Peter Xu --- migration/migration.c | 16 ++++++++++++ migration/migration.h | 7 +++++ migration/postcopy-ram.c | 56 +++++++++++++++++++++++++++++++--------- migration/postcopy-ram.h | 1 + 4 files changed, 68 insertions(+), 12 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 157a34c844..33faa0ff6e 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -3021,6 +3021,12 @@ static int postcopy_start(MigrationState *ms) int64_t bandwidth = migrate_max_postcopy_bandwidth(); bool restart_block = false; int cur_state = MIGRATION_STATUS_ACTIVE; + + if (postcopy_preempt_wait_channel(ms)) { + migrate_set_state(&ms->state, ms->state, MIGRATION_STATUS_FAILED); + return -1; + } + if (!migrate_pause_before_switchover()) { migrate_set_state(&ms->state, MIGRATION_STATUS_ACTIVE, MIGRATION_STATUS_POSTCOPY_ACTIVE); @@ -3502,6 +3508,14 @@ static MigThrError postcopy_pause(MigrationState *s) if (s->state == MIGRATION_STATUS_POSTCOPY_RECOVER) { /* Woken up by a recover procedure. Give it a shot */ + if (postcopy_preempt_wait_channel(s)) { + /* + * Preempt enabled, and new channel create failed; loop + * back to wait for another recovery. + */ + continue; + } + /* * Firstly, let's wake up the return path now, with a new * return path channel. @@ -4361,6 +4375,7 @@ static void migration_instance_finalize(Object *obj) qemu_sem_destroy(&ms->postcopy_pause_sem); qemu_sem_destroy(&ms->postcopy_pause_rp_sem); qemu_sem_destroy(&ms->rp_state.rp_sem); + qemu_sem_destroy(&ms->postcopy_qemufile_src_sem); error_free(ms->error); } @@ -4407,6 +4422,7 @@ static void migration_instance_init(Object *obj) qemu_sem_init(&ms->rp_state.rp_sem, 0); qemu_sem_init(&ms->rate_limit_sem, 0); qemu_sem_init(&ms->wait_unplug_sem, 0); + qemu_sem_init(&ms->postcopy_qemufile_src_sem, 0); qemu_mutex_init(&ms->qemu_file_lock); } diff --git a/migration/migration.h b/migration/migration.h index 91f845e9e4..f898b8547a 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -219,6 +219,13 @@ struct MigrationState { QEMUFile *to_dst_file; /* Postcopy specific transfer channel */ QEMUFile *postcopy_qemufile_src; + /* + * It is posted when the preempt channel is established. Note: this is + * used for both the start or recover of a postcopy migration. We'll + * post to this sem every time a new preempt channel is created in the + * main thread, and we keep post() and wait() in pair. + */ + QemuSemaphore postcopy_qemufile_src_sem; QIOChannelBuffer *bioc; /* * Protects to_dst_file/from_dst_file pointers. We need to make sure we diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index e20305a9e2..ab2a50cf45 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -1552,10 +1552,50 @@ bool postcopy_preempt_new_channel(MigrationIncomingState *mis, QEMUFile *file) return true; } -int postcopy_preempt_setup(MigrationState *s, Error **errp) +static void +postcopy_preempt_send_channel_new(QIOTask *task, gpointer opaque) { - QIOChannel *ioc; + MigrationState *s = opaque; + QIOChannel *ioc = QIO_CHANNEL(qio_task_get_source(task)); + Error *local_err = NULL; + + if (qio_task_propagate_error(task, &local_err)) { + /* Something wrong happened.. */ + migrate_set_error(s, local_err); + error_free(local_err); + } else { + migration_ioc_register_yank(ioc); + s->postcopy_qemufile_src = qemu_fopen_channel_output(ioc); + trace_postcopy_preempt_new_channel(); + } + + /* + * Kick the waiter in all cases. The waiter should check upon + * postcopy_qemufile_src to know whether it failed or not. + */ + qemu_sem_post(&s->postcopy_qemufile_src_sem); + object_unref(OBJECT(ioc)); +} +/* Returns 0 if channel established, -1 for error. */ +int postcopy_preempt_wait_channel(MigrationState *s) +{ + /* If preempt not enabled, no need to wait */ + if (!migrate_postcopy_preempt()) { + return 0; + } + + /* + * We need the postcopy preempt channel to be established before + * starting doing anything. + */ + qemu_sem_wait(&s->postcopy_qemufile_src_sem); + + return s->postcopy_qemufile_src ? 0 : -1; +} + +int postcopy_preempt_setup(MigrationState *s, Error **errp) +{ if (!migrate_postcopy_preempt()) { return 0; } @@ -1566,16 +1606,8 @@ int postcopy_preempt_setup(MigrationState *s, Error **errp) return -1; } - ioc = socket_send_channel_create_sync(errp); - - if (ioc == NULL) { - return -1; - } - - migration_ioc_register_yank(ioc); - s->postcopy_qemufile_src = qemu_fopen_channel_output(ioc); - - trace_postcopy_preempt_new_channel(); + /* Kick an async task to connect */ + socket_send_channel_create(postcopy_preempt_send_channel_new, s); return 0; } diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index 34b1080cde..6147bf7d1d 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -192,5 +192,6 @@ enum PostcopyChannels { bool postcopy_preempt_new_channel(MigrationIncomingState *mis, QEMUFile *file); int postcopy_preempt_setup(MigrationState *s, Error **errp); +int postcopy_preempt_wait_channel(MigrationState *s); #endif