From patchwork Thu Jul 22 17:58:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12394487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F778C63797 for ; Thu, 22 Jul 2021 18:31:22 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B4E5060E52 for ; Thu, 22 Jul 2021 18:31:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B4E5060E52 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:55304 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6dD9-0002Wj-Hs for qemu-devel@archiver.kernel.org; Thu, 22 Jul 2021 14:14:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:32996) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6cyD-00055G-Pg for qemu-devel@nongnu.org; Thu, 22 Jul 2021 13:58:53 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:54138) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6cy9-00064n-AF for qemu-devel@nongnu.org; Thu, 22 Jul 2021 13:58:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626976728; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=psIdSZ42TG6zIOWJSN4HC5NZuEOzx+F89j4lZhOXZEk=; b=TmBFRX0BSWPYvCyY95dpHOuMdfDCFibysBfh3Udku4ARNVUenRb6Qmxz6nWKpmAGBpKOJo wo5HNG87NjRBSryaq7Sfzx1tUkdvyfZCh06s5+mau/mtijyPNI4CjjiBa3IVZ7IJ5Q1wRs T88j7WFVecIrkpmmzp1EzYtRdaAC40o= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-558-WZ26pYvpNfak9Sm10mDh4w-1; Thu, 22 Jul 2021 13:58:47 -0400 X-MC-Unique: WZ26pYvpNfak9Sm10mDh4w-1 Received: by mail-qt1-f198.google.com with SMTP id j10-20020ac8440a0000b02902647b518455so3821481qtn.3 for ; Thu, 22 Jul 2021 10:58:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=psIdSZ42TG6zIOWJSN4HC5NZuEOzx+F89j4lZhOXZEk=; b=AuRllzS4ETVksFsJ7sRw5KVg1wOHddSAR/AYCEWRzgFkXfP+kVpp3p4bc4k6encGJJ I0vlPBwFjVKTNB0/uhqNyPcgodMgD7t4syOm1RYmG9U5QOQn0PRRJZumMEm0Lq0f5ndi a2dCri7Z3Cdr/9tp0NJBD9KmS0qijoCC9xihctiACAGEinvkcACrau1cmBoTyu+fWzN7 Qttpzq4dXgFuQrlsJhkdLqxRdmBy3681NItlaS4LWPjW5UWbDIkxTnPpYBLZfb3cAeus KQ0TTwBArGwd44UFl36Piw60qF9JYlvXP9yRNmOp2JBfM4OJ410SZmcckUBwBisT+F/c 9+4g== X-Gm-Message-State: AOAM533r4/jGNzzN8xuvwPbov9+ADQD+1WB4huHUu+f6nW1Rg4rOcal1 Y5o0RU4i9snnbidFLaJuCwNeuQstHvcVkUQgCGRjaBs+Mxh4sYeZ1rh5pocFLODHYgXv5DLypyg qPG0ZLoy7MS+XS8p4Cl5Yj9s7G3gXoBz/VryJDhaWb4HfH0B7TSRf8bidFGaf9cbM X-Received: by 2002:a05:622a:491:: with SMTP id p17mr768458qtx.107.1626976726826; Thu, 22 Jul 2021 10:58:46 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwAyECFT5+u102yYRUrAeI+TrUes/gX4O1xUA9LSbKnyYmTKMIZGJ2Guwk2AUOVdxq3aMA4+g== X-Received: by 2002:a05:622a:491:: with SMTP id p17mr768425qtx.107.1626976726521; Thu, 22 Jul 2021 10:58:46 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id j20sm10418018qtq.14.2021.07.22.10.58.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jul 2021 10:58:45 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v3 1/5] migration: Fix missing join() of rp_thread Date: Thu, 22 Jul 2021 13:58:37 -0400 Message-Id: <20210722175841.938739-2-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210722175841.938739-1-peterx@redhat.com> References: <20210722175841.938739-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.472, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Lukas Straub , "Daniel P . Berrange" , Juan Quintela , "Dr . David Alan Gilbert" , peterx@redhat.com, Leonardo Bras Soares Passos , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" It's possible that the migration thread skip the join() of the rp_thread in below race and crash on src right at finishing migration: migration_thread rp_thread ---------------- --------- migration_completion() (before rp_thread quits) from_dst_file=NULL [thread got scheduled out] s->rp_state.from_dst_file==NULL (skip join() of rp_thread) migrate_fd_cleanup() qemu_fclose(s->to_dst_file) yank_unregister_instance() assert(yank_find_entry()) <------- crash It could mostly happen with postcopy, but that shouldn't be required, e.g., I think it could also trigger with MIGRATION_CAPABILITY_RETURN_PATH set. It's suspected that above race could be the root cause of a recent (but rare) migration-test break reported by either Dave or PMM: https://lore.kernel.org/qemu-devel/YPamXAHwan%2FPPXLf@work-vm/ The issue is: from_dst_file is reset in the rp_thread, so if the thread reset it to NULL fast enough then the migration thread will assume there's no rp_thread at all. This could potentially cause more severe issue (e.g. crash) after the yank code. Fix it by using a boolean to keep "whether we've created rp_thread". Cc: Dr. David Alan Gilbert Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/migration.c | 4 +++- migration/migration.h | 7 +++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/migration/migration.c b/migration/migration.c index 2d306582eb..21b94f75a3 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2867,6 +2867,7 @@ static int open_return_path_on_source(MigrationState *ms, qemu_thread_create(&ms->rp_state.rp_thread, "return path", source_return_path_thread, ms, QEMU_THREAD_JOINABLE); + ms->rp_state.rp_thread_created = true; trace_open_return_path_on_source_continue(); @@ -2891,6 +2892,7 @@ static int await_return_path_close_on_source(MigrationState *ms) } trace_await_return_path_close_on_source_joining(); qemu_thread_join(&ms->rp_state.rp_thread); + ms->rp_state.rp_thread_created = false; trace_await_return_path_close_on_source_close(); return ms->rp_state.error; } @@ -3170,7 +3172,7 @@ static void migration_completion(MigrationState *s) * it will wait for the destination to send it's status in * a SHUT command). */ - if (s->rp_state.from_dst_file) { + if (s->rp_state.rp_thread_created) { int rp_error; trace_migration_return_path_end_before(); rp_error = await_return_path_close_on_source(s); diff --git a/migration/migration.h b/migration/migration.h index 2ebb740dfa..c302879fad 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -195,6 +195,13 @@ struct MigrationState { QEMUFile *from_dst_file; QemuThread rp_thread; bool error; + /* + * We can also check non-zero of rp_thread, but there's no "official" + * way to do this, so this bool makes it slightly more elegant. + * Checking from_dst_file for this is racy because from_dst_file will + * be cleared in the rp_thread! + */ + bool rp_thread_created; QemuSemaphore rp_sem; } rp_state; From patchwork Thu Jul 22 17:58:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12394481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DF9AC6377D for ; Thu, 22 Jul 2021 18:29:38 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9E71260D07 for ; Thu, 22 Jul 2021 18:29:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9E71260D07 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:33486 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6dGC-0006vl-QC for qemu-devel@archiver.kernel.org; Thu, 22 Jul 2021 14:17:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33010) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6cyF-0005CL-Gg for qemu-devel@nongnu.org; Thu, 22 Jul 2021 13:58:55 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:28969) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6cyB-00066F-0s for qemu-devel@nongnu.org; Thu, 22 Jul 2021 13:58:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626976730; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d/YJ+I59XhsY2Uttqc/MajgUDnI5kxf+zGYbKg5ksfU=; b=cq5s2e27/8+ORjY1ZAbg/peoi+OWI/fNemqriEvG1brZaKUsmIWWDTG3D1nc30Xpm+A3Ut xaZMJYiQcg3Bx7tnIw9Zyqt7HO7pt6HncZ5gppbASImR+2xiaEHUase+cnGjLVA572JcUH XBTuMm6eUno9TyG/SYJT1iGrmIYVtwE= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-518-_PL1Src3P2i4olC3Sf_J7A-1; Thu, 22 Jul 2021 13:58:49 -0400 X-MC-Unique: _PL1Src3P2i4olC3Sf_J7A-1 Received: by mail-qk1-f198.google.com with SMTP id bk12-20020a05620a1a0cb02903b899a4309cso4680387qkb.14 for ; Thu, 22 Jul 2021 10:58:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=d/YJ+I59XhsY2Uttqc/MajgUDnI5kxf+zGYbKg5ksfU=; b=lMyij/b7F0y1ImV7FwThwIE5WBO7PHI64lnEnXkXIPEHrBZ9xnv8QXtiMMNIaaA3NY WVnLkUOHWI9qNoWx1mfjMcWCr6t3kMJ9jM1wXEh2zzxHQ/zp3Qfp7jOlmojOVijR8ME2 ANJPBfd7SEHK4f9Cbdnc0SsT32BGPKuqYhin673nxS949DkRDd49vxgh/Tk0RqVnyZft /l0RWqnLukEvy1HBzUHTe1igjXksy0cLrJdSwxdgq/iu0dbx9dgwpdXSnFKWbjQOd+XC TKIh8KxyDUElK1CJk3SpylRjfYmbEXLxDsaOGVrLUC3YRIaJwmbVkb8CkCjHffx2lbhY 07xQ== X-Gm-Message-State: AOAM531aDVR99RmllgRCjzFl9ZQ5lBcqvqD89oFnJnIHGAjfESgp1DHJ 6d3PTYTpudcbt21eynxEZhinE1YVQ9dsYjmYqGZqosRe63C4Y2xJPmt2nFNneLJ++v0/hz3LuII vNOcUVxn78s0kmufnhgREkrJa3OlInpsOryBevuqkmSp45pASXysoawYAE+IxjcOO X-Received: by 2002:a05:620a:c9b:: with SMTP id q27mr840496qki.131.1626976728630; Thu, 22 Jul 2021 10:58:48 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw0HreYA7CoBJQHo2dYJA2hzujLds+gTNSygqj7keixrYreZYrGpJRDvTDf7G+ZkuKwcLIeXg== X-Received: by 2002:a05:620a:c9b:: with SMTP id q27mr840463qki.131.1626976728330; Thu, 22 Jul 2021 10:58:48 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id j20sm10418018qtq.14.2021.07.22.10.58.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jul 2021 10:58:47 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v3 2/5] migration: Make from_dst_file accesses thread-safe Date: Thu, 22 Jul 2021 13:58:38 -0400 Message-Id: <20210722175841.938739-3-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210722175841.938739-1-peterx@redhat.com> References: <20210722175841.938739-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.472, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Lukas Straub , "Daniel P . Berrange" , Juan Quintela , "Dr . David Alan Gilbert" , peterx@redhat.com, Leonardo Bras Soares Passos , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Accessing from_dst_file is potentially racy in current code base like below: if (s->from_dst_file) do_something(s->from_dst_file); Because from_dst_file can be reset right after the check in another thread (rp_thread). One example is migrate_fd_cancel(). Use the same qemu_file_lock to protect it too, just like to_dst_file. When it's safe to access without lock, comment it. There's one special reference in migration_thread() that can be replaced by the newly introduced rp_thread_created flag. Reported-by: Dr. David Alan Gilbert Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/migration.c | 39 +++++++++++++++++++++++++++++---------- migration/migration.h | 8 +++++--- migration/ram.c | 1 + 3 files changed, 35 insertions(+), 13 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 21b94f75a3..b2c48b7e17 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1879,9 +1879,11 @@ static void migrate_fd_cancel(MigrationState *s) QEMUFile *f = migrate_get_current()->to_dst_file; trace_migrate_fd_cancel(); - if (s->rp_state.from_dst_file) { - /* shutdown the rp socket, so causing the rp thread to shutdown */ - qemu_file_shutdown(s->rp_state.from_dst_file); + WITH_QEMU_LOCK_GUARD(&s->qemu_file_lock) { + if (s->rp_state.from_dst_file) { + /* shutdown the rp socket, so causing the rp thread to shutdown */ + qemu_file_shutdown(s->rp_state.from_dst_file); + } } do { @@ -2686,6 +2688,23 @@ static int migrate_handle_rp_resume_ack(MigrationState *s, uint32_t value) return 0; } +/* Release ms->rp_state.from_dst_file in a safe way */ +static void migration_release_from_dst_file(MigrationState *ms) +{ + QEMUFile *file; + + WITH_QEMU_LOCK_GUARD(&s->qemu_file_lock) { + /* + * Reset the from_dst_file pointer first before releasing it, as we + * can't block within lock section + */ + file = ms->rp_state.from_dst_file; + ms->rp_state.from_dst_file = NULL; + } + + qemu_fclose(file); +} + /* * Handles messages sent on the return path towards the source VM * @@ -2827,11 +2846,13 @@ out: * Maybe there is something we can do: it looks like a * network down issue, and we pause for a recovery. */ - qemu_fclose(rp); - ms->rp_state.from_dst_file = NULL; + migration_release_from_dst_file(ms); rp = NULL; if (postcopy_pause_return_path_thread(ms)) { - /* Reload rp, reset the rest */ + /* + * Reload rp, reset the rest. Referencing it is safe since + * it's reset only by us above, or when migration completes + */ rp = ms->rp_state.from_dst_file; ms->rp_state.error = false; goto retry; @@ -2843,8 +2864,7 @@ out: } trace_source_return_path_thread_end(); - ms->rp_state.from_dst_file = NULL; - qemu_fclose(rp); + migration_release_from_dst_file(ms); rcu_unregister_thread(); return NULL; } @@ -2852,7 +2872,6 @@ out: static int open_return_path_on_source(MigrationState *ms, bool create_thread) { - ms->rp_state.from_dst_file = qemu_file_get_return_path(ms->to_dst_file); if (!ms->rp_state.from_dst_file) { return -1; @@ -3746,7 +3765,7 @@ static void *migration_thread(void *opaque) * If we opened the return path, we need to make sure dst has it * opened as well. */ - if (s->rp_state.from_dst_file) { + if (s->rp_state.rp_thread_created) { /* Now tell the dest that it should open its end so it can reply */ qemu_savevm_send_open_return_path(s->to_dst_file); diff --git a/migration/migration.h b/migration/migration.h index c302879fad..7a5aa8c2fd 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -154,12 +154,13 @@ struct MigrationState { QemuThread thread; QEMUBH *vm_start_bh; QEMUBH *cleanup_bh; + /* Protected by qemu_file_lock */ QEMUFile *to_dst_file; QIOChannelBuffer *bioc; /* - * Protects to_dst_file pointer. We need to make sure we won't - * yield or hang during the critical section, since this lock will - * be used in OOB command handler. + * Protects to_dst_file/from_dst_file pointers. We need to make sure we + * won't yield or hang during the critical section, since this lock will be + * used in OOB command handler. */ QemuMutex qemu_file_lock; @@ -192,6 +193,7 @@ struct MigrationState { /* State related to return path */ struct { + /* Protected by qemu_file_lock */ QEMUFile *from_dst_file; QemuThread rp_thread; bool error; diff --git a/migration/ram.c b/migration/ram.c index b5fc454b2f..f728f5072f 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -4012,6 +4012,7 @@ static void ram_dirty_bitmap_reload_notify(MigrationState *s) int ram_dirty_bitmap_reload(MigrationState *s, RAMBlock *block) { int ret = -EINVAL; + /* from_dst_file is always valid because we're within rp_thread */ QEMUFile *file = s->rp_state.from_dst_file; unsigned long *le_bitmap, nbits = block->used_length >> TARGET_PAGE_BITS; uint64_t local_size = DIV_ROUND_UP(nbits, 8); From patchwork Thu Jul 22 17:58:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12394455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5177C6377D for ; Thu, 22 Jul 2021 17:59:54 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6EC85610F7 for ; Thu, 22 Jul 2021 17:59:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6EC85610F7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:54736 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6czB-0007xc-Js for qemu-devel@archiver.kernel.org; Thu, 22 Jul 2021 13:59:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33008) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6cyF-0005As-Av for qemu-devel@nongnu.org; Thu, 22 Jul 2021 13:58:55 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:55345) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6cyC-00068K-Qg for qemu-devel@nongnu.org; Thu, 22 Jul 2021 13:58:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626976732; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wKMAloDbSrO635HHxiPXxMG2SXVNhWvMxDfE+ax/uns=; b=OnTJ+xlksZggKC57HVsWsedaZ66WhRx/SQY5I0kCfAiORfv71lKGmqbYHSiJvuv606nzW0 jWNEFiFhbb3vYnLQJN16QfGlYxDtSdrjHGEHJxsM/qYyb+fsb4Ca+nNUNo7pUco4Wxsrlp 4H9B/f+ln+5R00wqeJxJ4EWwDSgL/Qs= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-122-rbzGdDOlMHyBn2dNyZoVtg-1; Thu, 22 Jul 2021 13:58:51 -0400 X-MC-Unique: rbzGdDOlMHyBn2dNyZoVtg-1 Received: by mail-qv1-f70.google.com with SMTP id ff14-20020a0562140bceb02902fc3fd31414so65098qvb.16 for ; Thu, 22 Jul 2021 10:58:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wKMAloDbSrO635HHxiPXxMG2SXVNhWvMxDfE+ax/uns=; b=rjsByx/4+fzO+FKNsihsnGHCuoVjNOdPxwkt8BQhcwgOW0xkm+STC47/RUTEnkZ3GY 1BrNwXBHfg/qNogvj5GJmxKZRZur7WZId/z6XEclFNlzcEaW1b5vzIvlHlqmGNiUzE1f Z9UEjBY03+xKJDsbPSg+qfqIOgYYzAjn+oiCbvVKH2dk/NLeVX0/mbVLKGtM/U6YKo0q fiZlL7UKnJvKazQo/PtUJ7cpSAuhzDQSkgousKqD8rTT1x75kmOoVanV/HYhxuYSYEwa mPTn8u7FdRLnEM5vLvk+pXLhd8ZFCztDV1zsdHds/S4KqndNSMA/Xcqs9Ttc1NIIrUjC vc2A== X-Gm-Message-State: AOAM5312zpllqsXQC5xEQQvY721LRBK30SFV4aoTzuaot/IVDgIdOdTW /fXcR3+Kvg1cbuv4aj3yhrKT3SbjcYsJKngv/J8SYe7cFyUkaUn5yGJtHu2Rvz49HQtB05KH1Sv tp6idfmCAVh27776aheG2s3ICxrhPM33RfWyWygjDAH/0waWNuSJ8nPNYdkTzledF X-Received: by 2002:a37:54f:: with SMTP id 76mr800295qkf.479.1626976730338; Thu, 22 Jul 2021 10:58:50 -0700 (PDT) X-Google-Smtp-Source: ABdhPJztKipqOOvI0jY2llxpDVRbnGxi96GI55t11oHVeYT/485XOcAuIw83q8EQ9L84Hd9DePGTMw== X-Received: by 2002:a37:54f:: with SMTP id 76mr800267qkf.479.1626976730073; Thu, 22 Jul 2021 10:58:50 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id j20sm10418018qtq.14.2021.07.22.10.58.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jul 2021 10:58:49 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v3 3/5] migration: Introduce migration_ioc_[un]register_yank() Date: Thu, 22 Jul 2021 13:58:39 -0400 Message-Id: <20210722175841.938739-4-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210722175841.938739-1-peterx@redhat.com> References: <20210722175841.938739-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.472, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Lukas Straub , "Daniel P . Berrange" , Juan Quintela , "Dr . David Alan Gilbert" , peterx@redhat.com, Leonardo Bras Soares Passos , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" There're plenty of places in migration/* that checks against either socket or tls typed ioc for yank operations. Provide two helpers to hide all these information. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/channel.c | 15 ++------------- migration/multifd.c | 8 ++------ migration/qemu-file-channel.c | 8 ++------ migration/yank_functions.c | 28 ++++++++++++++++++++++++++++ migration/yank_functions.h | 2 ++ 5 files changed, 36 insertions(+), 25 deletions(-) diff --git a/migration/channel.c b/migration/channel.c index 01275a9162..c4fc000a1a 100644 --- a/migration/channel.c +++ b/migration/channel.c @@ -44,13 +44,7 @@ void migration_channel_process_incoming(QIOChannel *ioc) TYPE_QIO_CHANNEL_TLS)) { migration_tls_channel_process_incoming(s, ioc, &local_err); } else { - if (object_dynamic_cast(OBJECT(ioc), TYPE_QIO_CHANNEL_SOCKET) || - object_dynamic_cast(OBJECT(ioc), TYPE_QIO_CHANNEL_TLS)) { - yank_register_function(MIGRATION_YANK_INSTANCE, - migration_yank_iochannel, - QIO_CHANNEL(ioc)); - } - + migration_ioc_register_yank(ioc); migration_ioc_process_incoming(ioc, &local_err); } @@ -94,12 +88,7 @@ void migration_channel_connect(MigrationState *s, } else { QEMUFile *f = qemu_fopen_channel_output(ioc); - if (object_dynamic_cast(OBJECT(ioc), TYPE_QIO_CHANNEL_SOCKET) || - object_dynamic_cast(OBJECT(ioc), TYPE_QIO_CHANNEL_TLS)) { - yank_register_function(MIGRATION_YANK_INSTANCE, - migration_yank_iochannel, - QIO_CHANNEL(ioc)); - } + migration_ioc_register_yank(ioc); qemu_mutex_lock(&s->qemu_file_lock); s->to_dst_file = f; diff --git a/migration/multifd.c b/migration/multifd.c index ab41590e71..377da78f5b 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -987,12 +987,8 @@ int multifd_load_cleanup(Error **errp) for (i = 0; i < migrate_multifd_channels(); i++) { MultiFDRecvParams *p = &multifd_recv_state->params[i]; - if ((object_dynamic_cast(OBJECT(p->c), TYPE_QIO_CHANNEL_SOCKET) || - object_dynamic_cast(OBJECT(p->c), TYPE_QIO_CHANNEL_TLS)) - && OBJECT(p->c)->ref == 1) { - yank_unregister_function(MIGRATION_YANK_INSTANCE, - migration_yank_iochannel, - QIO_CHANNEL(p->c)); + if (OBJECT(p->c)->ref == 1) { + migration_ioc_unregister_yank(p->c); } object_unref(OBJECT(p->c)); diff --git a/migration/qemu-file-channel.c b/migration/qemu-file-channel.c index fad340ea7a..867a5ed0c3 100644 --- a/migration/qemu-file-channel.c +++ b/migration/qemu-file-channel.c @@ -107,12 +107,8 @@ static int channel_close(void *opaque, Error **errp) int ret; QIOChannel *ioc = QIO_CHANNEL(opaque); ret = qio_channel_close(ioc, errp); - if ((object_dynamic_cast(OBJECT(ioc), TYPE_QIO_CHANNEL_SOCKET) || - object_dynamic_cast(OBJECT(ioc), TYPE_QIO_CHANNEL_TLS)) - && OBJECT(ioc)->ref == 1) { - yank_unregister_function(MIGRATION_YANK_INSTANCE, - migration_yank_iochannel, - QIO_CHANNEL(ioc)); + if (OBJECT(ioc)->ref == 1) { + migration_ioc_unregister_yank(ioc); } object_unref(OBJECT(ioc)); return ret; diff --git a/migration/yank_functions.c b/migration/yank_functions.c index 96c90e17dc..23697173ae 100644 --- a/migration/yank_functions.c +++ b/migration/yank_functions.c @@ -11,6 +11,9 @@ #include "qapi/error.h" #include "io/channel.h" #include "yank_functions.h" +#include "qemu/yank.h" +#include "io/channel-socket.h" +#include "io/channel-tls.h" void migration_yank_iochannel(void *opaque) { @@ -18,3 +21,28 @@ void migration_yank_iochannel(void *opaque) qio_channel_shutdown(ioc, QIO_CHANNEL_SHUTDOWN_BOTH, NULL); } + +/* Return whether yank is supported on this ioc */ +static bool migration_ioc_yank_supported(QIOChannel *ioc) +{ + return object_dynamic_cast(OBJECT(ioc), TYPE_QIO_CHANNEL_SOCKET) || + object_dynamic_cast(OBJECT(ioc), TYPE_QIO_CHANNEL_TLS); +} + +void migration_ioc_register_yank(QIOChannel *ioc) +{ + if (migration_ioc_yank_supported(ioc)) { + yank_register_function(MIGRATION_YANK_INSTANCE, + migration_yank_iochannel, + QIO_CHANNEL(ioc)); + } +} + +void migration_ioc_unregister_yank(QIOChannel *ioc) +{ + if (migration_ioc_yank_supported(ioc)) { + yank_unregister_function(MIGRATION_YANK_INSTANCE, + migration_yank_iochannel, + QIO_CHANNEL(ioc)); + } +} diff --git a/migration/yank_functions.h b/migration/yank_functions.h index 055ea22523..74c7f18c91 100644 --- a/migration/yank_functions.h +++ b/migration/yank_functions.h @@ -15,3 +15,5 @@ * @opaque: QIOChannel to shutdown */ void migration_yank_iochannel(void *opaque); +void migration_ioc_register_yank(QIOChannel *ioc); +void migration_ioc_unregister_yank(QIOChannel *ioc); From patchwork Thu Jul 22 17:58:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12394457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B19DAC63797 for ; Thu, 22 Jul 2021 18:02:58 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 38E4A6135F for ; Thu, 22 Jul 2021 18:02:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 38E4A6135F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:34638 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6d29-00057F-A7 for qemu-devel@archiver.kernel.org; Thu, 22 Jul 2021 14:02:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33034) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6cyH-0005HV-Hl for qemu-devel@nongnu.org; Thu, 22 Jul 2021 13:58:57 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:51674) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6cyF-0006AU-Jb for qemu-devel@nongnu.org; Thu, 22 Jul 2021 13:58:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626976735; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eoyn2Rh6iR3ndRW7tw5apxj+fstIZvcYwTTYe10yeLI=; b=eYQUOaUsNHBG+FATG9sJJqVH09IFABbS9/8otjl/vsk/pkh/WGSzknDo2yC+tJ/IE81ztI +qjzEc4LNdir68fDoQfrw4bvU6dYk6xw1TzYq3MszyQk542AiuQW7pQUqkoeych+L/2Cn2 qunlN6gpq9Bv4Uns71+l0sRNPpFguDA= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-600-zry9sVUdO5iSZNHiD5-i9w-1; Thu, 22 Jul 2021 13:58:54 -0400 X-MC-Unique: zry9sVUdO5iSZNHiD5-i9w-1 Received: by mail-qk1-f198.google.com with SMTP id a6-20020a37b1060000b02903b488f9d348so4657033qkf.20 for ; Thu, 22 Jul 2021 10:58:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eoyn2Rh6iR3ndRW7tw5apxj+fstIZvcYwTTYe10yeLI=; b=sYKng7aw4hWSCMI22Zuq6sZ00qBvdUk4oQqmspDqThFRL8HCxEYKwCFQyrq2Zihk6S A0jRvOE1CvkWXBnCm2C9vC3d3X3PMQp6b0XVcBsLM+nj8Rylu+yZws4gGvaZRJqi5S7O uADjkDEllY68nnOdXHZeX9Kt1JDTB8kc2LGvAznpT59kx1qPYkTlJffmG2JwYhyODs46 CNB2WLEa1vafKLpv2DRGEEwjKMirCtsgylmRKuhBl1fdEo0+60EnLtqzwo3f36HHy0Tq Hb9T/USjyIdcN1a+3XD8SkbczHdplAbDSTINEH/RQyM9wbvzzordq41RIyTK4Zcl2+Gc vc3Q== X-Gm-Message-State: AOAM532kKNYitQe1ebK+rBJ5X9ioqkyyNVnfmgZu3UwufemdAhv9rous n+PI7FqfjDDxLii3H64ficfjZ7el056v9RaRIzORq2bt+hWL7Eh7Or2PqBcwFu3KTGwbnJPymoN yOTx/Aco89R4M9Izhn4pvKI/MLqS8KXMbxEihx4yfASqmA4KbTBDguiK2XPoNzkJa X-Received: by 2002:a37:e44:: with SMTP id 65mr841680qko.193.1626976733061; Thu, 22 Jul 2021 10:58:53 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzfjCPPkfmtQW6PfptPwWfCE6AfW385EgAWR1F30yHGh/D8PVz/W0NJe26ZXOQnHnJzD2OExw== X-Received: by 2002:a37:e44:: with SMTP id 65mr841657qko.193.1626976732766; Thu, 22 Jul 2021 10:58:52 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id j20sm10418018qtq.14.2021.07.22.10.58.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jul 2021 10:58:51 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v3 4/5] migration: Teach QEMUFile to be QIOChannel-aware Date: Thu, 22 Jul 2021 13:58:40 -0400 Message-Id: <20210722175841.938739-5-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210722175841.938739-1-peterx@redhat.com> References: <20210722175841.938739-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.472, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Lukas Straub , "Daniel P . Berrange" , Juan Quintela , "Dr . David Alan Gilbert" , peterx@redhat.com, Leonardo Bras Soares Passos , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" migration uses QIOChannel typed qemufiles. In follow up patches, we'll need the capability to identify this fact, so that we can get the backing QIOChannel from a QEMUFile. We can also define types for QEMUFile but so far since we only need to be able to identify QIOChannel, introduce a boolean which is simpler. Introduce another helper qemu_file_get_ioc() to return the ioc backend of a qemufile if has_ioc is set. No functional change. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/qemu-file-channel.c | 4 ++-- migration/qemu-file.c | 17 ++++++++++++++++- migration/qemu-file.h | 4 +++- migration/ram.c | 2 +- migration/savevm.c | 4 ++-- 5 files changed, 24 insertions(+), 7 deletions(-) diff --git a/migration/qemu-file-channel.c b/migration/qemu-file-channel.c index 867a5ed0c3..2f8b1fcd46 100644 --- a/migration/qemu-file-channel.c +++ b/migration/qemu-file-channel.c @@ -187,11 +187,11 @@ static const QEMUFileOps channel_output_ops = { QEMUFile *qemu_fopen_channel_input(QIOChannel *ioc) { object_ref(OBJECT(ioc)); - return qemu_fopen_ops(ioc, &channel_input_ops); + return qemu_fopen_ops(ioc, &channel_input_ops, true); } QEMUFile *qemu_fopen_channel_output(QIOChannel *ioc) { object_ref(OBJECT(ioc)); - return qemu_fopen_ops(ioc, &channel_output_ops); + return qemu_fopen_ops(ioc, &channel_output_ops, true); } diff --git a/migration/qemu-file.c b/migration/qemu-file.c index 1eacf9e831..6338d8e2ff 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -55,6 +55,8 @@ struct QEMUFile { Error *last_error_obj; /* has the file has been shutdown */ bool shutdown; + /* Whether opaque points to a QIOChannel */ + bool has_ioc; }; /* @@ -101,7 +103,7 @@ bool qemu_file_mode_is_not_valid(const char *mode) return false; } -QEMUFile *qemu_fopen_ops(void *opaque, const QEMUFileOps *ops) +QEMUFile *qemu_fopen_ops(void *opaque, const QEMUFileOps *ops, bool has_ioc) { QEMUFile *f; @@ -109,6 +111,7 @@ QEMUFile *qemu_fopen_ops(void *opaque, const QEMUFileOps *ops) f->opaque = opaque; f->ops = ops; + f->has_ioc = has_ioc; return f; } @@ -851,3 +854,15 @@ void qemu_file_set_blocking(QEMUFile *f, bool block) f->ops->set_blocking(f->opaque, block, NULL); } } + +/* + * Return the ioc object if it's a migration channel. Note: it can return NULL + * for callers passing in a non-migration qemufile. E.g. see qemu_fopen_bdrv() + * and its usage in e.g. load_snapshot(). So we need to check against NULL + * before using it. If without the check, migration_incoming_state_destroy() + * could fail for load_snapshot(). + */ +QIOChannel *qemu_file_get_ioc(QEMUFile *file) +{ + return file->has_ioc ? QIO_CHANNEL(file->opaque) : NULL; +} diff --git a/migration/qemu-file.h b/migration/qemu-file.h index a9b6d6ccb7..3f36d4dc8c 100644 --- a/migration/qemu-file.h +++ b/migration/qemu-file.h @@ -27,6 +27,7 @@ #include #include "exec/cpu-common.h" +#include "io/channel.h" /* Read a chunk of data from a file at the given position. The pos argument * can be ignored if the file is only be used for streaming. The number of @@ -119,7 +120,7 @@ typedef struct QEMUFileHooks { QEMURamSaveFunc *save_page; } QEMUFileHooks; -QEMUFile *qemu_fopen_ops(void *opaque, const QEMUFileOps *ops); +QEMUFile *qemu_fopen_ops(void *opaque, const QEMUFileOps *ops, bool has_ioc); void qemu_file_set_hooks(QEMUFile *f, const QEMUFileHooks *hooks); int qemu_get_fd(QEMUFile *f); int qemu_fclose(QEMUFile *f); @@ -179,5 +180,6 @@ void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data); size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset, ram_addr_t offset, size_t size, uint64_t *bytes_sent); +QIOChannel *qemu_file_get_ioc(QEMUFile *file); #endif diff --git a/migration/ram.c b/migration/ram.c index f728f5072f..08b3cb7a4a 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -550,7 +550,7 @@ static int compress_threads_save_setup(void) /* comp_param[i].file is just used as a dummy buffer to save data, * set its ops to empty. */ - comp_param[i].file = qemu_fopen_ops(NULL, &empty_ops); + comp_param[i].file = qemu_fopen_ops(NULL, &empty_ops, false); comp_param[i].done = true; comp_param[i].quit = false; qemu_mutex_init(&comp_param[i].mutex); diff --git a/migration/savevm.c b/migration/savevm.c index 72848b946c..96b5e5d639 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -168,9 +168,9 @@ static const QEMUFileOps bdrv_write_ops = { static QEMUFile *qemu_fopen_bdrv(BlockDriverState *bs, int is_writable) { if (is_writable) { - return qemu_fopen_ops(bs, &bdrv_write_ops); + return qemu_fopen_ops(bs, &bdrv_write_ops, false); } - return qemu_fopen_ops(bs, &bdrv_read_ops); + return qemu_fopen_ops(bs, &bdrv_read_ops, false); } From patchwork Thu Jul 22 17:58:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12394475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75A5DC63797 for ; Thu, 22 Jul 2021 18:19:39 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C1149600D0 for ; Thu, 22 Jul 2021 18:19:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org C1149600D0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:37610 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6dIH-0001KT-St for qemu-devel@archiver.kernel.org; Thu, 22 Jul 2021 14:19:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33046) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6cyJ-0005Nf-HB for qemu-devel@nongnu.org; Thu, 22 Jul 2021 13:58:59 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:22582) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6cyH-0006BX-Ez for qemu-devel@nongnu.org; Thu, 22 Jul 2021 13:58:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626976736; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+hF+7xtGuVKQZvHq4jyUAcy84OuZgE4buGXnLhdZjuc=; b=HvzFVxs6dkrKj4mdRxHVum+ekkSAsDlobdAVFRJ62p+Tj5pBpbbPm1AGgw7KPTgMenKKuL dIk96R5dgMHUs/AH3w0gYuJLAQxyZYxgAn/tJMyvieUEGwE+/I7SyhgNxFHQho39HynYLM wASD4dDR0uCX/OlWVv7GnhMeP2MsIrY= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-65-1ReLYlKnMc6uTWDjBCGYdQ-1; Thu, 22 Jul 2021 13:58:55 -0400 X-MC-Unique: 1ReLYlKnMc6uTWDjBCGYdQ-1 Received: by mail-qv1-f72.google.com with SMTP id h12-20020a0562140dacb029030449450d37so80359qvh.14 for ; Thu, 22 Jul 2021 10:58:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+hF+7xtGuVKQZvHq4jyUAcy84OuZgE4buGXnLhdZjuc=; b=r6vtYJS0+bGK9d99z/5yiiulVB9yWg2GmjE3QUCaZ9i1OXOAdrJhce+rPp7DVVCLgI +mkqLnlZ1YmSZZQNWoOa0OWeteg6IH0ZUk+MplOc00MtJUrELysvwihDUIjKzI2HTxjf K37bk5RWJjYrwUTeaxhpD6CmscBTAaazLln81iL5pf0O8pM1hmFJHb6FA0jgsi6VAXgN wZju1Dm2rbQiqv/K5X8zszcjolJEwqgbartEbgKOxtW4pUtPNrdhLcHytfBcAght0LCJ pc0VMlAQnrM+7Dy9k+6ktQhFf1YemiX8JYXM2TU+Jz5YRC8IbmznBOBoZMMcxWjljyTz ubDA== X-Gm-Message-State: AOAM530hlFOl/gAN6ojvAepENa04z5kq0dUXOF/WH5G8A+cBktOwO+6W rjXQo8/XjxcUKM40+86JLuyJ904u57+pChykfVRLHp/UjTjo9cUrBpwT8XxEt9VwdZI8WYSdBvr c5PgV/5bN4EeiG3ts3TCeR8BLvmJQzlM8G8CByj4+47PXbIE2SCMzbMD/CFTmcabC X-Received: by 2002:a37:9d41:: with SMTP id g62mr818075qke.55.1626976734725; Thu, 22 Jul 2021 10:58:54 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzPDkGswJTWHrh+SgGCfBUU+nwisKOl4U0VYZQMEGU2toXXt4WforeUhhFFQhYfv/RerP84MQ== X-Received: by 2002:a37:9d41:: with SMTP id g62mr818045qke.55.1626976734453; Thu, 22 Jul 2021 10:58:54 -0700 (PDT) Received: from localhost.localdomain (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id j20sm10418018qtq.14.2021.07.22.10.58.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jul 2021 10:58:53 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v3 5/5] migration: Move the yank unregister of channel_close out Date: Thu, 22 Jul 2021 13:58:41 -0400 Message-Id: <20210722175841.938739-6-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210722175841.938739-1-peterx@redhat.com> References: <20210722175841.938739-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.472, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Lukas Straub , "Daniel P . Berrange" , Juan Quintela , "Dr . David Alan Gilbert" , peterx@redhat.com, Leonardo Bras Soares Passos , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" It's efficient, but hackish to call yank unregister calls in channel_close(), especially it'll be hard to debug when qemu crashed with some yank function leaked. Remove that hack, but instead explicitly unregister yank functions at the places where needed, they are: (on src) - migrate_fd_cleanup - postcopy_pause (on dst) - migration_incoming_state_destroy - postcopy_pause_incoming Signed-off-by: Peter Xu Reviewed-by: Dr. David Alan Gilbert --- migration/migration.c | 14 +++++++++++++- migration/qemu-file-channel.c | 3 --- migration/savevm.c | 7 +++++++ migration/yank_functions.c | 14 ++++++++++++++ migration/yank_functions.h | 1 + 5 files changed, 35 insertions(+), 4 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index b2c48b7e17..a50330016c 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -59,6 +59,7 @@ #include "multifd.h" #include "qemu/yank.h" #include "sysemu/cpus.h" +#include "yank_functions.h" #define MAX_THROTTLE (128 << 20) /* Migration transfer speed throttling */ @@ -273,6 +274,7 @@ void migration_incoming_state_destroy(void) } if (mis->from_src_file) { + migration_ioc_unregister_yank_from_file(mis->from_src_file); qemu_fclose(mis->from_src_file); mis->from_src_file = NULL; } @@ -1811,6 +1813,7 @@ static void migrate_fd_cleanup(MigrationState *s) * Close the file handle without the lock to make sure the * critical section won't block for long. */ + migration_ioc_unregister_yank_from_file(tmp); qemu_fclose(tmp); } @@ -3351,8 +3354,17 @@ static MigThrError postcopy_pause(MigrationState *s) while (true) { QEMUFile *file; - /* Current channel is possibly broken. Release it. */ + /* + * Current channel is possibly broken. Release it. Note that this is + * guaranteed even without lock because to_dst_file should only be + * modified by the migration thread. That also guarantees that the + * unregister of yank is safe too without the lock. It should be safe + * even to be within the qemu_file_lock, but we didn't do that to avoid + * taking more mutex (yank_lock) within qemu_file_lock. TL;DR: we make + * the qemu_file_lock critical section as small as possible. + */ assert(s->to_dst_file); + migration_ioc_unregister_yank_from_file(s->to_dst_file); qemu_mutex_lock(&s->qemu_file_lock); file = s->to_dst_file; s->to_dst_file = NULL; diff --git a/migration/qemu-file-channel.c b/migration/qemu-file-channel.c index 2f8b1fcd46..bb5a5752df 100644 --- a/migration/qemu-file-channel.c +++ b/migration/qemu-file-channel.c @@ -107,9 +107,6 @@ static int channel_close(void *opaque, Error **errp) int ret; QIOChannel *ioc = QIO_CHANNEL(opaque); ret = qio_channel_close(ioc, errp); - if (OBJECT(ioc)->ref == 1) { - migration_ioc_unregister_yank(ioc); - } object_unref(OBJECT(ioc)); return ret; } diff --git a/migration/savevm.c b/migration/savevm.c index 96b5e5d639..7b7b64bd13 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -65,6 +65,7 @@ #include "qemu/bitmap.h" #include "net/announce.h" #include "qemu/yank.h" +#include "yank_functions.h" const unsigned int postcopy_ram_discard_version; @@ -2568,6 +2569,12 @@ static bool postcopy_pause_incoming(MigrationIncomingState *mis) /* Clear the triggered bit to allow one recovery */ mis->postcopy_recover_triggered = false; + /* + * Unregister yank with either from/to src would work, since ioc behind it + * is the same + */ + migration_ioc_unregister_yank_from_file(mis->from_src_file); + assert(mis->from_src_file); qemu_file_shutdown(mis->from_src_file); qemu_fclose(mis->from_src_file); diff --git a/migration/yank_functions.c b/migration/yank_functions.c index 23697173ae..8c08aef14a 100644 --- a/migration/yank_functions.c +++ b/migration/yank_functions.c @@ -14,6 +14,7 @@ #include "qemu/yank.h" #include "io/channel-socket.h" #include "io/channel-tls.h" +#include "qemu-file.h" void migration_yank_iochannel(void *opaque) { @@ -46,3 +47,16 @@ void migration_ioc_unregister_yank(QIOChannel *ioc) QIO_CHANNEL(ioc)); } } + +void migration_ioc_unregister_yank_from_file(QEMUFile *file) +{ + QIOChannel *ioc = qemu_file_get_ioc(file); + + if (ioc) { + /* + * For migration qemufiles, we'll always reach here. Though we'll skip + * calls from e.g. savevm/loadvm as they don't use yank. + */ + migration_ioc_unregister_yank(ioc); + } +} diff --git a/migration/yank_functions.h b/migration/yank_functions.h index 74c7f18c91..a7577955ed 100644 --- a/migration/yank_functions.h +++ b/migration/yank_functions.h @@ -17,3 +17,4 @@ void migration_yank_iochannel(void *opaque); void migration_ioc_register_yank(QIOChannel *ioc); void migration_ioc_unregister_yank(QIOChannel *ioc); +void migration_ioc_unregister_yank_from_file(QEMUFile *file);