From patchwork Fri Apr 26 14:20:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13644902 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1277FC4345F for ; Fri, 26 Apr 2024 14:21:59 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s0MRZ-00074O-H8; Fri, 26 Apr 2024 10:20:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s0MRY-000749-8W for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:20:52 -0400 Received: from smtp-out2.suse.de ([2a07:de40:b251:101:10:150:64:2]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s0MRW-0001j0-Lt for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:20:52 -0400 Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 7B1B75D1B4; Fri, 26 Apr 2024 14:20:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141249; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mQBVEPhYKVMyFrGSn4n311EcmSB4NFH4AoNPtuIqqss=; b=cjb26cV+RmFt6WNZCga0PwzbpUkjDBEsTKZbmkYQMfD8GgwJb/9Hq6QanVxRzKFG0VP7r1 O3HOJWD7EcPwbXN298UlvdEwaCBe6gBPO8ynXNzIdB1VAnnCPxC5WW2xh6giOulqWYbUlS ikT0Mr0DZhWIa2/LwlBCcAvcFQDsoY0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141249; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mQBVEPhYKVMyFrGSn4n311EcmSB4NFH4AoNPtuIqqss=; b=duMuWZPWOU+o9kQcv1dyNRWqsAcknkQXej9hzTp8SX619ahExHzNdRYiqgYCIkhCY0FIvQ kZ2/9kkgAr2adTBA== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141249; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mQBVEPhYKVMyFrGSn4n311EcmSB4NFH4AoNPtuIqqss=; b=cjb26cV+RmFt6WNZCga0PwzbpUkjDBEsTKZbmkYQMfD8GgwJb/9Hq6QanVxRzKFG0VP7r1 O3HOJWD7EcPwbXN298UlvdEwaCBe6gBPO8ynXNzIdB1VAnnCPxC5WW2xh6giOulqWYbUlS ikT0Mr0DZhWIa2/LwlBCcAvcFQDsoY0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141249; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mQBVEPhYKVMyFrGSn4n311EcmSB4NFH4AoNPtuIqqss=; b=duMuWZPWOU+o9kQcv1dyNRWqsAcknkQXej9hzTp8SX619ahExHzNdRYiqgYCIkhCY0FIvQ kZ2/9kkgAr2adTBA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id A3888139B1; Fri, 26 Apr 2024 14:20:47 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id KLsbGj+4K2YWcAAAD6G6ig (envelope-from ); Fri, 26 Apr 2024 14:20:47 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: berrange@redhat.com, armbru@redhat.com, Peter Xu , Claudio Fontana , Jim Fehlig Subject: [PATCH 1/9] monitor: Honor QMP request for fd removal immediately Date: Fri, 26 Apr 2024 11:20:34 -0300 Message-Id: <20240426142042.14573-2-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240426142042.14573-1-farosas@suse.de> References: <20240426142042.14573-1-farosas@suse.de> MIME-Version: 1.0 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.991]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:email,imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns]; RCVD_TLS_ALL(0.00)[] Received-SPF: pass client-ip=2a07:de40:b251:101:10:150:64:2; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org We're enabling using the fdset interface to pass file descriptors for use in the migration code. Since migrations can happen more than once during the VMs lifetime, we need a way to remove an fd from the fdset at the end of migration. The current code only removes an fd from the fdset if the VM is running. This causes a QMP call to "remove-fd" to not actually remove the fd if the VM happens to be stopped. While the fd would eventually be removed when monitor_fdset_cleanup() is called again, the user request should be honored and the fd actually removed. Calling remove-fd + query-fdset shows a recently removed fd still present. The runstate_is_running() check was introduced by commit ebe52b592d ("monitor: Prevent removing fd from set during init"), which by the shortlog indicates that they were trying to avoid removing an yet-unduplicated fd too early. I don't see why an fd explicitly removed with qmp_remove_fd() should be under runstate_is_running(). I'm assuming this was a mistake when adding the parenthesis around the expression. Move the runstate_is_running() check to apply only to the QLIST_EMPTY(dup_fds) side of the expression and ignore it when mon_fdset_fd->removed has been explicitly set. Signed-off-by: Fabiano Rosas --- monitor/fds.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/monitor/fds.c b/monitor/fds.c index d86c2c674c..4ec3b7eea9 100644 --- a/monitor/fds.c +++ b/monitor/fds.c @@ -173,9 +173,9 @@ static void monitor_fdset_cleanup(MonFdset *mon_fdset) MonFdsetFd *mon_fdset_fd_next; QLIST_FOREACH_SAFE(mon_fdset_fd, &mon_fdset->fds, next, mon_fdset_fd_next) { - if ((mon_fdset_fd->removed || - (QLIST_EMPTY(&mon_fdset->dup_fds) && mon_refcount == 0)) && - runstate_is_running()) { + if (mon_fdset_fd->removed || + (QLIST_EMPTY(&mon_fdset->dup_fds) && mon_refcount == 0 && + runstate_is_running())) { close(mon_fdset_fd->fd); g_free(mon_fdset_fd->opaque); QLIST_REMOVE(mon_fdset_fd, next); From patchwork Fri Apr 26 14:20:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13644898 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C86A0C4345F for ; Fri, 26 Apr 2024 14:21:17 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s0MRl-00076d-VN; Fri, 26 Apr 2024 10:21:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s0MRb-00074q-2X for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:21:00 -0400 Received: from smtp-out1.suse.de ([195.135.223.130]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s0MRZ-0001n5-Bb for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:20:54 -0400 Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id C55F9227DD; Fri, 26 Apr 2024 14:20:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141251; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c7pq969m/eSku4SA1kn+8rZjTBbc88kbtQ4svsW/Eqg=; b=UnOGFWXwhn58pwJAs2sANqUv1JGicl+2e/oYpJNUZwameGB64S2rN2OC4YNGxmm1O3wwjx 4v/MQBAEMA+gsM8JilBlfu8hXtDwLbpKso7vg7Lh7mZ2GiVRIOZFrJBHIUSMrDEAXi91wF 9t5Bny/eqPGwCCbe0Jo7iQUSP6VmnlE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141251; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c7pq969m/eSku4SA1kn+8rZjTBbc88kbtQ4svsW/Eqg=; b=jIJGFD5OHOz9kMy4sLgIQlJmxkVJdmndbFrhgxLuZPQNis8np+N9kGdqWZTWb5GDSbPukP 7W7N8pQP+HapNcDw== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141251; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c7pq969m/eSku4SA1kn+8rZjTBbc88kbtQ4svsW/Eqg=; b=UnOGFWXwhn58pwJAs2sANqUv1JGicl+2e/oYpJNUZwameGB64S2rN2OC4YNGxmm1O3wwjx 4v/MQBAEMA+gsM8JilBlfu8hXtDwLbpKso7vg7Lh7mZ2GiVRIOZFrJBHIUSMrDEAXi91wF 9t5Bny/eqPGwCCbe0Jo7iQUSP6VmnlE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141251; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c7pq969m/eSku4SA1kn+8rZjTBbc88kbtQ4svsW/Eqg=; b=jIJGFD5OHOz9kMy4sLgIQlJmxkVJdmndbFrhgxLuZPQNis8np+N9kGdqWZTWb5GDSbPukP 7W7N8pQP+HapNcDw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id ECFB1136DB; Fri, 26 Apr 2024 14:20:49 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id 6M13LEG4K2YWcAAAD6G6ig (envelope-from ); Fri, 26 Apr 2024 14:20:49 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: berrange@redhat.com, armbru@redhat.com, Peter Xu , Claudio Fontana , Jim Fehlig Subject: [PATCH 2/9] migration: Fix file migration with fdset Date: Fri, 26 Apr 2024 11:20:35 -0300 Message-Id: <20240426142042.14573-3-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240426142042.14573-1-farosas@suse.de> References: <20240426142042.14573-1-farosas@suse.de> MIME-Version: 1.0 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.991]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:email,imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns]; RCVD_TLS_ALL(0.00)[] Received-SPF: pass client-ip=195.135.223.130; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org When the migration using the "file:" URI was implemented, I don't think any of us noticed that if you pass in a file name with the format "/dev/fdset/N", this allows a file descriptor to be passed in to QEMU and that behaves just like the "fd:" URI. So the "file:" support has been added without regard for the fdset part and we got some things wrong. The first issue is that we should not truncate the migration file if we're allowing an fd + offset. We need to leave the file contents untouched. The second issue is that there's an expectation that QEMU removes the fd after the migration has finished. That's what the "fd:" code does. Otherwise a second migration on the same VM could attempt to provide an fdset with the same name and QEMU would reject it. We can fix the first issue by detecting when we're using the fdset vs. the plain file name. This requires storing the fdset_id somewhere. We can then use this stored fdset_id to do cleanup at the end and also fix the second issue. Fixes: 385f510df5 ("migration: file URI offset") Signed-off-by: Fabiano Rosas Signed-off-by: Peter Xu --- migration/file.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/migration/file.c b/migration/file.c index ab18ba505a..8f30999400 100644 --- a/migration/file.c +++ b/migration/file.c @@ -10,6 +10,7 @@ #include "qemu/cutils.h" #include "qemu/error-report.h" #include "qapi/error.h" +#include "qapi/qapi-commands-misc.h" #include "channel.h" #include "file.h" #include "migration.h" @@ -23,6 +24,7 @@ static struct FileOutgoingArgs { char *fname; + int64_t fdset_id; } outgoing_args; /* Remove the offset option from @filespec and return it in @offsetp. */ @@ -44,10 +46,39 @@ int file_parse_offset(char *filespec, uint64_t *offsetp, Error **errp) return 0; } +static void file_remove_fdset(void) +{ + if (outgoing_args.fdset_id != -1) { + qmp_remove_fd(outgoing_args.fdset_id, false, -1, NULL); + outgoing_args.fdset_id = -1; + } +} + +static bool file_parse_fdset(const char *filename, int64_t *fdset_id, + Error **errp) +{ + const char *fdset_id_str; + + *fdset_id = -1; + + if (!strstart(filename, "/dev/fdset/", &fdset_id_str)) { + return true; + } + + *fdset_id = qemu_parse_fd(fdset_id_str); + if (*fdset_id == -1) { + error_setg_errno(errp, EINVAL, "Could not parse fdset %s", fdset_id_str); + return false; + } + + return true; +} + void file_cleanup_outgoing_migration(void) { g_free(outgoing_args.fname); outgoing_args.fname = NULL; + file_remove_fdset(); } bool file_send_channel_create(gpointer opaque, Error **errp) @@ -81,11 +112,24 @@ void file_start_outgoing_migration(MigrationState *s, g_autofree char *filename = g_strdup(file_args->filename); uint64_t offset = file_args->offset; QIOChannel *ioc; + int flags = O_CREAT | O_WRONLY; trace_migration_file_outgoing(filename); - fioc = qio_channel_file_new_path(filename, O_CREAT | O_WRONLY | O_TRUNC, - 0600, errp); + if (!file_parse_fdset(filename, &outgoing_args.fdset_id, errp)) { + return; + } + + /* + * Only truncate if it's QEMU opening the file. If an fd has been + * passed in the file will already contain data written by the + * management layer. + */ + if (outgoing_args.fdset_id == -1) { + flags |= O_TRUNC; + } + + fioc = qio_channel_file_new_path(filename, flags, 0600, errp); if (!fioc) { return; } From patchwork Fri Apr 26 14:20:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13644901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 47C05C4345F for ; Fri, 26 Apr 2024 14:21:54 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s0MRp-0007A7-6k; Fri, 26 Apr 2024 10:21:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s0MRf-00075A-9K for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:21:00 -0400 Received: from smtp-out2.suse.de ([2a07:de40:b251:101:10:150:64:2]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s0MRb-0001qn-VK for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:20:57 -0400 Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id C7CB95D1B3; Fri, 26 Apr 2024 14:20:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141254; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IPWq7CqMuVlecbO8p1Wu+OhDjoi7ZLLNh5mPmBy6qzQ=; b=PRhlSdxjlGZHepYnXTO3LQ1d3H2TpWgNIIfpR/ebCSjSiu9AgPGO4hxQ23luyY9SGSlrep 612BRl3Ks5Sg4NLqBDGZVY5zHy/TTFPZ3EkZHJX8geDivyNkqry24/g1a29tCxdTKbR18V bN0xtjTCLZrOU+FzMuguZvOS8Fv70H0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141254; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IPWq7CqMuVlecbO8p1Wu+OhDjoi7ZLLNh5mPmBy6qzQ=; b=1E9HJKkziKB9913fmFmvGqmH3GWFbM2Ri0zGZsUdMoH4tEslCXzRrXZ8vmQbCU4yINPD60 DTl+VVPAb/r7drAA== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141254; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IPWq7CqMuVlecbO8p1Wu+OhDjoi7ZLLNh5mPmBy6qzQ=; b=PRhlSdxjlGZHepYnXTO3LQ1d3H2TpWgNIIfpR/ebCSjSiu9AgPGO4hxQ23luyY9SGSlrep 612BRl3Ks5Sg4NLqBDGZVY5zHy/TTFPZ3EkZHJX8geDivyNkqry24/g1a29tCxdTKbR18V bN0xtjTCLZrOU+FzMuguZvOS8Fv70H0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141254; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IPWq7CqMuVlecbO8p1Wu+OhDjoi7ZLLNh5mPmBy6qzQ=; b=1E9HJKkziKB9913fmFmvGqmH3GWFbM2Ri0zGZsUdMoH4tEslCXzRrXZ8vmQbCU4yINPD60 DTl+VVPAb/r7drAA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 4270A136DB; Fri, 26 Apr 2024 14:20:52 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id aPbQAkS4K2YWcAAAD6G6ig (envelope-from ); Fri, 26 Apr 2024 14:20:52 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: berrange@redhat.com, armbru@redhat.com, Peter Xu , Claudio Fontana , Jim Fehlig , Thomas Huth , Laurent Vivier , Paolo Bonzini Subject: [PATCH 3/9] tests/qtest/migration: Fix file migration offset check Date: Fri, 26 Apr 2024 11:20:36 -0300 Message-Id: <20240426142042.14573-4-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240426142042.14573-1-farosas@suse.de> References: <20240426142042.14573-1-farosas@suse.de> MIME-Version: 1.0 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.991]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; FUZZY_BLOCKED(0.00)[rspamd.com]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_SEVEN(0.00)[9]; RCVD_COUNT_TWO(0.00)[2]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo, imap1.dmz-prg2.suse.org:rdns, suse.de:email] Received-SPF: pass client-ip=2a07:de40:b251:101:10:150:64:2; envelope-from=farosas@suse.de; helo=smtp-out2.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org When doing file migration, QEMU accepts an offset that should be skipped when writing the migration stream to the file. The purpose of the offset is to allow the management layer to put its own metadata at the start of the file. We have tests for this in migration-test, but only testing that the migration stream starts at the correct offset and not that it actually leaves the data intact. Unsurprisingly, there's been a bug in that area that the tests didn't catch. Fix the tests to write some data to the offset region and check that it's actually there after the migration. Fixes: 3dc35470c8 ("tests/qtest: migration-test: Add tests for file-based migration") Signed-off-by: Fabiano Rosas --- tests/qtest/migration-test.c | 70 +++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 5 deletions(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 5d6d8cd634..7b177686b4 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -2081,6 +2081,63 @@ static void test_precopy_file(void) test_file_common(&args, true); } +#ifndef _WIN32 +static void file_dirty_offset_region(void) +{ +#if defined(__linux__) + g_autofree char *path = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); + size_t size = FILE_TEST_OFFSET; + uintptr_t *addr, *p; + int fd; + + fd = open(path, O_CREAT | O_RDWR, 0660); + g_assert(fd != -1); + + g_assert(!ftruncate(fd, size)); + + addr = mmap(NULL, size, PROT_WRITE, MAP_SHARED, fd, 0); + g_assert(addr != MAP_FAILED); + + /* ensure the skipped offset contains some data */ + p = addr; + while (p < addr + FILE_TEST_OFFSET / sizeof(uintptr_t)) { + *p = (unsigned long) FILE_TEST_FILENAME; + p++; + } + + munmap(addr, size); + fsync(fd); + close(fd); +#endif +} + +static void *file_offset_start_hook(QTestState *from, QTestState *to) +{ + g_autofree char *file = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); + int src_flags = O_WRONLY; + int dst_flags = O_RDONLY; + int fds[2]; + + file_dirty_offset_region(); + + fds[0] = open(file, src_flags, 0660); + assert(fds[0] != -1); + + fds[1] = open(file, dst_flags, 0660); + assert(fds[1] != -1); + + qtest_qmp_fds_assert_success(from, &fds[0], 1, "{'execute': 'add-fd', " + "'arguments': {'fdset-id': 1}}"); + + qtest_qmp_fds_assert_success(to, &fds[1], 1, "{'execute': 'add-fd', " + "'arguments': {'fdset-id': 1}}"); + + close(fds[0]); + close(fds[1]); + + return NULL; +} + static void file_offset_finish_hook(QTestState *from, QTestState *to, void *opaque) { @@ -2096,12 +2153,12 @@ static void file_offset_finish_hook(QTestState *from, QTestState *to, g_assert(addr != MAP_FAILED); /* - * Ensure the skipped offset contains zeros and the migration - * stream starts at the right place. + * Ensure the skipped offset region's data has not been touched + * and the migration stream starts at the right place. */ p = addr; while (p < addr + FILE_TEST_OFFSET / sizeof(uintptr_t)) { - g_assert(*p == 0); + g_assert_cmpstr((char *) *p, ==, FILE_TEST_FILENAME); p++; } g_assert_cmpint(cpu_to_be64(*p) >> 32, ==, QEMU_VM_FILE_MAGIC); @@ -2113,17 +2170,18 @@ static void file_offset_finish_hook(QTestState *from, QTestState *to, static void test_precopy_file_offset(void) { - g_autofree char *uri = g_strdup_printf("file:%s/%s,offset=%d", tmpfs, - FILE_TEST_FILENAME, + g_autofree char *uri = g_strdup_printf("file:/dev/fdset/1,offset=%d", FILE_TEST_OFFSET); MigrateCommon args = { .connect_uri = uri, .listen_uri = "defer", + .start_hook = file_offset_start_hook, .finish_hook = file_offset_finish_hook, }; test_file_common(&args, false); } +#endif static void test_precopy_file_offset_bad(void) { @@ -3636,8 +3694,10 @@ int main(int argc, char **argv) migration_test_add("/migration/precopy/file", test_precopy_file); +#ifndef _WIN32 migration_test_add("/migration/precopy/file/offset", test_precopy_file_offset); +#endif migration_test_add("/migration/precopy/file/offset/bad", test_precopy_file_offset_bad); From patchwork Fri Apr 26 14:20:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13644900 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A18C4C04FFE for ; Fri, 26 Apr 2024 14:21:54 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s0MRo-00079K-TV; Fri, 26 Apr 2024 10:21:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s0MRg-00075N-Qo for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:21:01 -0400 Received: from smtp-out1.suse.de ([195.135.223.130]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s0MRf-0001u1-1i for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:21:00 -0400 Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 59C7C22774; Fri, 26 Apr 2024 14:20:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141257; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+035CNw8GklI6SUjAuE0D7i3DayB3sEpp59Ii7Ebiuk=; b=GWClYEBHBo3L2iHt5dAdpAbD3l8LxmwBLYK+9PeQ95Nn81pqiqsa1eKW6u1A9VG1RpEGzy CqWeQpvIq0f0eqiiLTXf819Tmvz0NGECUIh7bNYYFennD4b9j2KZ2Yse+Wjc7O5RK9DqY0 k0wuC7T0qjGjXdXRJTh7qvup9MDDcIM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141257; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+035CNw8GklI6SUjAuE0D7i3DayB3sEpp59Ii7Ebiuk=; b=xWtbof2nS+LgqzIPU/bq/k/UMiKo8eE3L6tXTvKNJS4mGprPWhzfxzV3GfIL8q2rrJx+pL Pw+VuEW8EYUKZeDw== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141257; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+035CNw8GklI6SUjAuE0D7i3DayB3sEpp59Ii7Ebiuk=; b=GWClYEBHBo3L2iHt5dAdpAbD3l8LxmwBLYK+9PeQ95Nn81pqiqsa1eKW6u1A9VG1RpEGzy CqWeQpvIq0f0eqiiLTXf819Tmvz0NGECUIh7bNYYFennD4b9j2KZ2Yse+Wjc7O5RK9DqY0 k0wuC7T0qjGjXdXRJTh7qvup9MDDcIM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141257; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+035CNw8GklI6SUjAuE0D7i3DayB3sEpp59Ii7Ebiuk=; b=xWtbof2nS+LgqzIPU/bq/k/UMiKo8eE3L6tXTvKNJS4mGprPWhzfxzV3GfIL8q2rrJx+pL Pw+VuEW8EYUKZeDw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 4CA17136DB; Fri, 26 Apr 2024 14:20:55 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id QNNBBUe4K2YWcAAAD6G6ig (envelope-from ); Fri, 26 Apr 2024 14:20:55 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: berrange@redhat.com, armbru@redhat.com, Peter Xu , Claudio Fontana , Jim Fehlig , Eric Blake Subject: [PATCH 4/9] migration: Add direct-io parameter Date: Fri, 26 Apr 2024 11:20:37 -0300 Message-Id: <20240426142042.14573-5-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240426142042.14573-1-farosas@suse.de> References: <20240426142042.14573-1-farosas@suse.de> MIME-Version: 1.0 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.991]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; FUZZY_BLOCKED(0.00)[rspamd.com]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_COUNT_TWO(0.00)[2]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo, imap1.dmz-prg2.suse.org:rdns, suse.de:email] Received-SPF: pass client-ip=195.135.223.130; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add the direct-io migration parameter that tells the migration code to use O_DIRECT when opening the migration stream file whenever possible. This is currently only used with the mapped-ram migration that has a clear window guaranteed to perform aligned writes. Acked-by: Markus Armbruster Signed-off-by: Fabiano Rosas --- include/qemu/osdep.h | 2 ++ migration/migration-hmp-cmds.c | 11 +++++++++++ migration/options.c | 30 ++++++++++++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 18 +++++++++++++++--- util/osdep.c | 9 +++++++++ 6 files changed, 68 insertions(+), 3 deletions(-) diff --git a/include/qemu/osdep.h b/include/qemu/osdep.h index c7053cdc2b..645c14a65d 100644 --- a/include/qemu/osdep.h +++ b/include/qemu/osdep.h @@ -612,6 +612,8 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive); bool qemu_has_ofd_lock(void); #endif +bool qemu_has_direct_io(void); + #if defined(__HAIKU__) && defined(__i386__) #define FMT_pid "%ld" #elif defined(WIN64) diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index 7e96ae6ffd..8496a2b34e 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -397,6 +397,13 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) monitor_printf(mon, "%s: %s\n", MigrationParameter_str(MIGRATION_PARAMETER_MODE), qapi_enum_lookup(&MigMode_lookup, params->mode)); + + if (params->has_direct_io) { + monitor_printf(mon, "%s: %s\n", + MigrationParameter_str( + MIGRATION_PARAMETER_DIRECT_IO), + params->direct_io ? "on" : "off"); + } } qapi_free_MigrationParameters(params); @@ -690,6 +697,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_mode = true; visit_type_MigMode(v, param, &p->mode, &err); break; + case MIGRATION_PARAMETER_DIRECT_IO: + p->has_direct_io = true; + visit_type_bool(v, param, &p->direct_io, &err); + break; default: assert(0); } diff --git a/migration/options.c b/migration/options.c index 239f5ecfb4..ae464aa4f2 100644 --- a/migration/options.c +++ b/migration/options.c @@ -826,6 +826,22 @@ int migrate_decompress_threads(void) return s->parameters.decompress_threads; } +bool migrate_direct_io(void) +{ + MigrationState *s = migrate_get_current(); + + /* For now O_DIRECT is only supported with mapped-ram */ + if (!s->capabilities[MIGRATION_CAPABILITY_MAPPED_RAM]) { + return false; + } + + if (s->parameters.has_direct_io) { + return s->parameters.direct_io; + } + + return false; +} + uint64_t migrate_downtime_limit(void) { MigrationState *s = migrate_get_current(); @@ -1061,6 +1077,11 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->has_zero_page_detection = true; params->zero_page_detection = s->parameters.zero_page_detection; + if (s->parameters.has_direct_io) { + params->has_direct_io = true; + params->direct_io = s->parameters.direct_io; + } + return params; } @@ -1097,6 +1118,7 @@ void migrate_params_init(MigrationParameters *params) params->has_vcpu_dirty_limit = true; params->has_mode = true; params->has_zero_page_detection = true; + params->has_direct_io = qemu_has_direct_io(); } /* @@ -1416,6 +1438,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params, if (params->has_zero_page_detection) { dest->zero_page_detection = params->zero_page_detection; } + + if (params->has_direct_io) { + dest->direct_io = params->direct_io; + } } static void migrate_params_apply(MigrateSetParameters *params, Error **errp) @@ -1570,6 +1596,10 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp) if (params->has_zero_page_detection) { s->parameters.zero_page_detection = params->zero_page_detection; } + + if (params->has_direct_io) { + s->parameters.direct_io = params->direct_io; + } } void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp) diff --git a/migration/options.h b/migration/options.h index ab8199e207..aa5509cd2a 100644 --- a/migration/options.h +++ b/migration/options.h @@ -76,6 +76,7 @@ uint8_t migrate_cpu_throttle_increment(void); uint8_t migrate_cpu_throttle_initial(void); bool migrate_cpu_throttle_tailslow(void); int migrate_decompress_threads(void); +bool migrate_direct_io(void); uint64_t migrate_downtime_limit(void); uint8_t migrate_max_cpu_throttle(void); uint64_t migrate_max_bandwidth(void); diff --git a/qapi/migration.json b/qapi/migration.json index 8c65b90328..1a8a4b114c 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -914,6 +914,9 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# requires that the @mapped-ram capability is enabled. (since 9.1) +# # Features: # # @deprecated: Member @block-incremental is deprecated. Use @@ -948,7 +951,8 @@ { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] }, 'vcpu-dirty-limit', 'mode', - 'zero-page-detection'] } + 'zero-page-detection', + 'direct-io'] } ## # @MigrateSetParameters: @@ -1122,6 +1126,9 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# requires that the @mapped-ram capability is enabled. (since 9.1) +# # Features: # # @deprecated: Member @block-incremental is deprecated. Use @@ -1176,7 +1183,8 @@ 'features': [ 'unstable' ] }, '*vcpu-dirty-limit': 'uint64', '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection'} } + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool' } } ## # @migrate-set-parameters: @@ -1354,6 +1362,9 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# requires that the @mapped-ram capability is enabled. (since 9.1) +# # Features: # # @deprecated: Member @block-incremental is deprecated. Use @@ -1405,7 +1416,8 @@ 'features': [ 'unstable' ] }, '*vcpu-dirty-limit': 'uint64', '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection'} } + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool' } } ## # @query-migrate-parameters: diff --git a/util/osdep.c b/util/osdep.c index e996c4744a..d0227a60ab 100644 --- a/util/osdep.c +++ b/util/osdep.c @@ -277,6 +277,15 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive) } #endif +bool qemu_has_direct_io(void) +{ +#ifdef O_DIRECT + return true; +#else + return false; +#endif +} + static int qemu_open_cloexec(const char *name, int flags, mode_t mode) { int ret; From patchwork Fri Apr 26 14:20:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13644903 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 77302C4345F for ; Fri, 26 Apr 2024 14:22:35 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s0MRy-0007Jt-PJ; Fri, 26 Apr 2024 10:21:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s0MRw-0007FO-M6 for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:21:16 -0400 Received: from smtp-out1.suse.de ([195.135.223.130]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s0MRi-0001xe-Ih for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:21:16 -0400 Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id A3A9534E8F; Fri, 26 Apr 2024 14:20:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141259; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rqQbG+3aTKLy7AA7zqRDf4iFroAaNZ18tQCzSkYhB0k=; b=Whtd0i1bZ2iwTkl/xqbvamBrzfoIs0CZomrjEyPyJobMuwlMg5u+Mx9oo5S5Js/BkJLG+P 4j4eL5wPB78yM4OyBhozdMFBoreN5xXihrBJ0SN5aPM7diyaSYY84joqL2i8yXASTpeQEs vrl1nOjUR5E/2fFAcPn48AGJIu1lqyY= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141259; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rqQbG+3aTKLy7AA7zqRDf4iFroAaNZ18tQCzSkYhB0k=; b=l4AKAjCJWgaV706zcSX091DaHxSx0H7BGeGHg+5Q62N/IOUsCGtqw33Rz3/UsLuUuYGW+j 520iLmA/maT3hcAQ== Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=Whtd0i1b; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=l4AKAjCJ DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141259; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rqQbG+3aTKLy7AA7zqRDf4iFroAaNZ18tQCzSkYhB0k=; b=Whtd0i1bZ2iwTkl/xqbvamBrzfoIs0CZomrjEyPyJobMuwlMg5u+Mx9oo5S5Js/BkJLG+P 4j4eL5wPB78yM4OyBhozdMFBoreN5xXihrBJ0SN5aPM7diyaSYY84joqL2i8yXASTpeQEs vrl1nOjUR5E/2fFAcPn48AGJIu1lqyY= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141259; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rqQbG+3aTKLy7AA7zqRDf4iFroAaNZ18tQCzSkYhB0k=; b=l4AKAjCJWgaV706zcSX091DaHxSx0H7BGeGHg+5Q62N/IOUsCGtqw33Rz3/UsLuUuYGW+j 520iLmA/maT3hcAQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id CD25E136DB; Fri, 26 Apr 2024 14:20:57 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id uIG1JEm4K2YWcAAAD6G6ig (envelope-from ); Fri, 26 Apr 2024 14:20:57 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: berrange@redhat.com, armbru@redhat.com, Peter Xu , Claudio Fontana , Jim Fehlig Subject: [PATCH 5/9] migration/multifd: Add direct-io support Date: Fri, 26 Apr 2024 11:20:38 -0300 Message-Id: <20240426142042.14573-6-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240426142042.14573-1-farosas@suse.de> References: <20240426142042.14573-1-farosas@suse.de> MIME-Version: 1.0 X-Rspamd-Action: no action X-Rspamd-Queue-Id: A3A9534E8F X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; NEURAL_HAM_LONG(-1.00)[-1.000]; MID_CONTAINS_FROM(1.00)[]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; RBL_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:104:10:150:64:97:from]; SPAMHAUS_XBL(0.00)[2a07:de40:b281:104:10:150:64:97:from]; RCVD_COUNT_TWO(0.00)[2]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns,suse.de:dkim,suse.de:email]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DNSWL_BLOCKED(0.00)[2a07:de40:b281:104:10:150:64:97:from]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCPT_COUNT_FIVE(0.00)[6]; DKIM_TRACE(0.00)[suse.de:+] Received-SPF: pass client-ip=195.135.223.130; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org When multifd is used along with mapped-ram, we can take benefit of a filesystem that supports the O_DIRECT flag and perform direct I/O in the multifd threads. This brings a significant performance improvement because direct-io writes bypass the page cache which would otherwise be thrashed by the multifd data which is unlikely to be needed again in a short period of time. To be able to use a multifd channel opened with O_DIRECT, we must ensure that a certain aligment is used. Filesystems usually require a block-size alignment for direct I/O. The way to achieve this is by enabling the mapped-ram feature, which already aligns its I/O properly (see MAPPED_RAM_FILE_OFFSET_ALIGNMENT at ram.c). By setting O_DIRECT on the multifd channels, all writes to the same file descriptor need to be aligned as well, even the ones that come from outside multifd, such as the QEMUFile I/O from the main migration code. This makes it impossible to use the same file descriptor for the QEMUFile and for the multifd channels. The various flags and metadata written by the main migration code will always be unaligned by virtue of their small size. To workaround this issue, we'll require a second file descriptor to be used exclusively for direct I/O. The second file descriptor can be obtained by QEMU by re-opening the migration file (already possible), or by being provided by the user or management application (support to be added in future patches). Signed-off-by: Fabiano Rosas --- migration/file.c | 22 +++++++++++++++++++--- migration/migration.c | 23 +++++++++++++++++++++++ 2 files changed, 42 insertions(+), 3 deletions(-) diff --git a/migration/file.c b/migration/file.c index 8f30999400..b9265b14dd 100644 --- a/migration/file.c +++ b/migration/file.c @@ -83,17 +83,33 @@ void file_cleanup_outgoing_migration(void) bool file_send_channel_create(gpointer opaque, Error **errp) { - QIOChannelFile *ioc; + QIOChannelFile *ioc = NULL; int flags = O_WRONLY; - bool ret = true; + bool ret = false; + + if (migrate_direct_io()) { +#ifdef O_DIRECT + /* + * Enable O_DIRECT for the secondary channels. These are used + * for sending ram pages and writes should be guaranteed to be + * aligned to at least page size. + */ + flags |= O_DIRECT; +#else + error_setg(errp, "System does not support O_DIRECT"); + error_append_hint(errp, + "Try disabling direct-io migration capability\n"); + goto out; +#endif + } ioc = qio_channel_file_new_path(outgoing_args.fname, flags, 0, errp); if (!ioc) { - ret = false; goto out; } multifd_channel_connect(opaque, QIO_CHANNEL(ioc)); + ret = true; out: /* diff --git a/migration/migration.c b/migration/migration.c index b5af6b5105..cb923a3f62 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -155,6 +155,16 @@ static bool migration_needs_seekable_channel(void) return migrate_mapped_ram(); } +static bool migration_needs_multiple_fds(void) +{ + /* + * When doing direct-io, multifd requires two different, + * non-duplicated file descriptors so we can use one of them for + * unaligned IO. + */ + return migrate_multifd() && migrate_direct_io(); +} + static bool transport_supports_seeking(MigrationAddress *addr) { if (addr->transport == MIGRATION_ADDRESS_TYPE_FILE) { @@ -164,6 +174,12 @@ static bool transport_supports_seeking(MigrationAddress *addr) return false; } +static bool transport_supports_multiple_fds(MigrationAddress *addr) +{ + /* file: works because QEMU can open it multiple times */ + return addr->transport == MIGRATION_ADDRESS_TYPE_FILE; +} + static bool migration_channels_and_transport_compatible(MigrationAddress *addr, Error **errp) @@ -180,6 +196,13 @@ migration_channels_and_transport_compatible(MigrationAddress *addr, return false; } + if (migration_needs_multiple_fds() && + !transport_supports_multiple_fds(addr)) { + error_setg(errp, + "Migration requires a transport that allows for multiple fds (e.g. file)"); + return false; + } + return true; } From patchwork Fri Apr 26 14:20:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13644899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8912AC4345F for ; Fri, 26 Apr 2024 14:21:34 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s0MRv-0007Ev-Ta; Fri, 26 Apr 2024 10:21:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s0MRt-0007CV-Cv for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:21:14 -0400 Received: from smtp-out1.suse.de ([195.135.223.130]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s0MRk-0001zp-2H for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:21:10 -0400 Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id A450A22774; Fri, 26 Apr 2024 14:21:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141262; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bk81xWisVj9HyM5b9IOLBvd2q2sfczVnt0abhSE4FBo=; b=UVEaWagAqahNa0+cSvYFk8o+z360fowV+2oDgR8vC/lxUSbR++tbxphjAiNLQfr2HspQwg Uu16fWMllxIdjY9Bi25TX7oRKxiNnsVlWHc5rPF3duXAbEyP0djCvCDSFygVTCbkuojTD6 tpktBw2eYhgD4SJ6m4fK8LT7j2OVVdk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141262; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bk81xWisVj9HyM5b9IOLBvd2q2sfczVnt0abhSE4FBo=; b=VxTvDommt8so9Egvh/iZplwewE4y+eTXoRj9YiuQnXlRSOfjnTCg9YH8Pn3L5kSgOpyggW nEK7Gzl/KTHWUaAg== Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=UVEaWagA; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=VxTvDomm DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141262; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bk81xWisVj9HyM5b9IOLBvd2q2sfczVnt0abhSE4FBo=; b=UVEaWagAqahNa0+cSvYFk8o+z360fowV+2oDgR8vC/lxUSbR++tbxphjAiNLQfr2HspQwg Uu16fWMllxIdjY9Bi25TX7oRKxiNnsVlWHc5rPF3duXAbEyP0djCvCDSFygVTCbkuojTD6 tpktBw2eYhgD4SJ6m4fK8LT7j2OVVdk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141262; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bk81xWisVj9HyM5b9IOLBvd2q2sfczVnt0abhSE4FBo=; b=VxTvDommt8so9Egvh/iZplwewE4y+eTXoRj9YiuQnXlRSOfjnTCg9YH8Pn3L5kSgOpyggW nEK7Gzl/KTHWUaAg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 241DB136DB; Fri, 26 Apr 2024 14:20:59 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id aDgFN0u4K2YWcAAAD6G6ig (envelope-from ); Fri, 26 Apr 2024 14:20:59 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: berrange@redhat.com, armbru@redhat.com, Peter Xu , Claudio Fontana , Jim Fehlig , Thomas Huth , Laurent Vivier , Paolo Bonzini Subject: [PATCH 6/9] tests/qtest/migration: Add tests for file migration with direct-io Date: Fri, 26 Apr 2024 11:20:39 -0300 Message-Id: <20240426142042.14573-7-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240426142042.14573-1-farosas@suse.de> References: <20240426142042.14573-1-farosas@suse.de> MIME-Version: 1.0 X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; MIME_TRACE(0.00)[0:+]; FUZZY_BLOCKED(0.00)[rspamd.com]; TO_DN_SOME(0.00)[]; RBL_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:104:10:150:64:97:from]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_SEVEN(0.00)[9]; DNSWL_BLOCKED(0.00)[2a07:de40:b281:104:10:150:64:97:from]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; DKIM_TRACE(0.00)[suse.de:+]; SPAMHAUS_XBL(0.00)[2a07:de40:b281:104:10:150:64:97:from]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:dkim, suse.de:email, imap1.dmz-prg2.suse.org:helo, imap1.dmz-prg2.suse.org:rdns] X-Rspamd-Action: no action X-Rspamd-Queue-Id: A450A22774 X-Rspamd-Server: rspamd1.dmz-prg2.suse.org Received-SPF: pass client-ip=195.135.223.130; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The tests are only allowed to run in systems that know about the O_DIRECT flag and in filesystems which support it. Signed-off-by: Fabiano Rosas Reviewed-by: Peter Xu --- tests/qtest/migration-helpers.c | 42 +++++++++++++++++++++++++++++++++ tests/qtest/migration-helpers.h | 1 + tests/qtest/migration-test.c | 42 +++++++++++++++++++++++++++++++++ 3 files changed, 85 insertions(+) diff --git a/tests/qtest/migration-helpers.c b/tests/qtest/migration-helpers.c index ce6d6615b5..356cd4fa8c 100644 --- a/tests/qtest/migration-helpers.c +++ b/tests/qtest/migration-helpers.c @@ -473,3 +473,45 @@ void migration_test_add(const char *path, void (*fn)(void)) qtest_add_data_func_full(path, test, migration_test_wrapper, migration_test_destroy); } + +#ifdef O_DIRECT +/* + * Probe for O_DIRECT support on the filesystem. Since this is used + * for tests, be conservative, if anything fails, assume it's + * unsupported. + */ +bool probe_o_direct_support(const char *tmpfs) +{ + g_autofree char *filename = g_strdup_printf("%s/probe-o-direct", tmpfs); + int fd, flags = O_CREAT | O_RDWR | O_TRUNC | O_DIRECT; + void *buf; + ssize_t ret, len; + uint64_t offset; + + fd = open(filename, flags, 0660); + if (fd < 0) { + unlink(filename); + return false; + } + + /* + * Assuming 4k should be enough to satisfy O_DIRECT alignment + * requirements. The migration code uses 1M to be conservative. + */ + len = 0x100000; + offset = 0x100000; + + buf = aligned_alloc(len, len); + g_assert(buf); + + ret = pwrite(fd, buf, len, offset); + unlink(filename); + g_free(buf); + + if (ret < 0) { + return false; + } + + return true; +} +#endif diff --git a/tests/qtest/migration-helpers.h b/tests/qtest/migration-helpers.h index 1339835698..d827e16145 100644 --- a/tests/qtest/migration-helpers.h +++ b/tests/qtest/migration-helpers.h @@ -54,5 +54,6 @@ char *find_common_machine_version(const char *mtype, const char *var1, const char *var2); char *resolve_machine_version(const char *alias, const char *var1, const char *var2); +bool probe_o_direct_support(const char *tmpfs); void migration_test_add(const char *path, void (*fn)(void)); #endif /* MIGRATION_HELPERS_H */ diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 7b177686b4..512b7ede8b 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -2295,6 +2295,43 @@ static void test_multifd_file_mapped_ram(void) test_file_common(&args, true); } +#ifdef O_DIRECT +static void *migrate_mapped_ram_dio_start(QTestState *from, + QTestState *to) +{ + migrate_mapped_ram_start(from, to); + migrate_set_parameter_bool(from, "direct-io", true); + migrate_set_parameter_bool(to, "direct-io", true); + + return NULL; +} + +static void *migrate_multifd_mapped_ram_dio_start(QTestState *from, + QTestState *to) +{ + migrate_multifd_mapped_ram_start(from, to); + return migrate_mapped_ram_dio_start(from, to); +} + +static void test_multifd_file_mapped_ram_dio(void) +{ + g_autofree char *uri = g_strdup_printf("file:%s/%s", tmpfs, + FILE_TEST_FILENAME); + MigrateCommon args = { + .connect_uri = uri, + .listen_uri = "defer", + .start_hook = migrate_multifd_mapped_ram_dio_start, + }; + + if (!probe_o_direct_support(tmpfs)) { + g_test_skip("Filesystem does not support O_DIRECT"); + return; + } + + test_file_common(&args, true); +} + +#endif /* O_DIRECT */ static void test_precopy_tcp_plain(void) { @@ -3719,6 +3756,11 @@ int main(int argc, char **argv) migration_test_add("/migration/multifd/file/mapped-ram/live", test_multifd_file_mapped_ram_live); +#ifdef O_DIRECT + migration_test_add("/migration/multifd/file/mapped-ram/dio", + test_multifd_file_mapped_ram_dio); +#endif + #ifdef CONFIG_GNUTLS migration_test_add("/migration/precopy/unix/tls/psk", test_precopy_unix_tls_psk); From patchwork Fri Apr 26 14:20:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13644905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4034FC4345F for ; Fri, 26 Apr 2024 14:22:55 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s0MRx-0007I0-Gd; Fri, 26 Apr 2024 10:21:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s0MRt-0007Ca-Dg for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:21:14 -0400 Received: from smtp-out1.suse.de ([195.135.223.130]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s0MRm-00021Y-CE for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:21:10 -0400 Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 033A234E8A; Fri, 26 Apr 2024 14:21:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141265; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J8CCKXkL/YLC5/eb7TlLoS0uyJKrJifQBuci848gQpg=; b=E+HhrQW8fBQr8obf6ejtw6fpan1AdJqyTRymSU7XaYs/PXFbQI8rdKQzJaFQkH1Uz7+/Nl AHMoKRRkaC4qbd7HZgZ8wHksEiD/Vq2MtH9WUNVAtIGlhepnmXtaFQ3mnC6yPaFYEv7PuQ CiY/pc3RybERlp3G98fy4CmEy5R+wPM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141265; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J8CCKXkL/YLC5/eb7TlLoS0uyJKrJifQBuci848gQpg=; b=6mk2RUgH59ovmjXNvpaVMRA97IicTzmL3A1olkGB3treKcDHQU144b9bPPaPqjyRSBE+Xw T39J7qyCqIqwK1BA== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141265; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J8CCKXkL/YLC5/eb7TlLoS0uyJKrJifQBuci848gQpg=; b=E+HhrQW8fBQr8obf6ejtw6fpan1AdJqyTRymSU7XaYs/PXFbQI8rdKQzJaFQkH1Uz7+/Nl AHMoKRRkaC4qbd7HZgZ8wHksEiD/Vq2MtH9WUNVAtIGlhepnmXtaFQ3mnC6yPaFYEv7PuQ CiY/pc3RybERlp3G98fy4CmEy5R+wPM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141265; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J8CCKXkL/YLC5/eb7TlLoS0uyJKrJifQBuci848gQpg=; b=6mk2RUgH59ovmjXNvpaVMRA97IicTzmL3A1olkGB3treKcDHQU144b9bPPaPqjyRSBE+Xw T39J7qyCqIqwK1BA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 23303136DB; Fri, 26 Apr 2024 14:21:02 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id EFvRNk64K2YWcAAAD6G6ig (envelope-from ); Fri, 26 Apr 2024 14:21:02 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: berrange@redhat.com, armbru@redhat.com, Peter Xu , Claudio Fontana , Jim Fehlig Subject: [PATCH 7/9] monitor: fdset: Match against O_DIRECT Date: Fri, 26 Apr 2024 11:20:40 -0300 Message-Id: <20240426142042.14573-8-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240426142042.14573-1-farosas@suse.de> References: <20240426142042.14573-1-farosas@suse.de> MIME-Version: 1.0 X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-0.991]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:email, imap1.dmz-prg2.suse.org:helo, imap1.dmz-prg2.suse.org:rdns] Received-SPF: pass client-ip=195.135.223.130; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org We're about to enable the use of O_DIRECT in the migration code and due to the alignment restrictions imposed by filesystems we need to make sure the flag is only used when doing aligned IO. The migration will do parallel IO to different regions of a file, so we need to use more than one file descriptor. Those cannot be obtained by duplicating (dup()) since duplicated file descriptors share the file status flags, including O_DIRECT. If one migration channel does unaligned IO while another sets O_DIRECT to do aligned IO, the filesystem would fail the unaligned operation. The add-fd QMP command along with the fdset code are specifically designed to allow the user to pass a set of file descriptors with different access flags into QEMU to be later fetched by code that needs to alternate between those flags when doing IO. Extend the fdset matching to behave the same with the O_DIRECT flag. Signed-off-by: Fabiano Rosas --- monitor/fds.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/monitor/fds.c b/monitor/fds.c index 4ec3b7eea9..62e324fcec 100644 --- a/monitor/fds.c +++ b/monitor/fds.c @@ -420,6 +420,11 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags) int fd = -1; int dup_fd; int mon_fd_flags; + int mask = O_ACCMODE; + +#ifdef O_DIRECT + mask |= O_DIRECT; +#endif if (mon_fdset->id != fdset_id) { continue; @@ -431,7 +436,7 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags) return -1; } - if ((flags & O_ACCMODE) == (mon_fd_flags & O_ACCMODE)) { + if ((flags & mask) == (mon_fd_flags & mask)) { fd = mon_fdset_fd->fd; break; } From patchwork Fri Apr 26 14:20:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13644906 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A2C08C4345F for ; Fri, 26 Apr 2024 14:22:59 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s0MRw-0007FM-M5; Fri, 26 Apr 2024 10:21:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s0MRt-0007CW-D0 for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:21:14 -0400 Received: from smtp-out1.suse.de ([195.135.223.130]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s0MRp-000245-BP for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:21:10 -0400 Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 472A922DF8; Fri, 26 Apr 2024 14:21:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141267; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/eMGgmiQ5JFvoP97b8KdNqSKLJOCLLrmNvHFvzlrZEc=; b=DReSgi2quHvJIGDkjhDq7bXYAiPX5b/MC37ca4vx/RL4neTKoRiip3pEfF7VrsIoNi+Bb2 xNPZW/+qGnlhkC3nx0/AsqlHm0RifCARlAN8p/vcTQXrn8HLRfYbmafE3QT3lJ2j7tWGTb 3bIHv0hy1NDhAbjv8eYlcciHw70C3A4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141267; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/eMGgmiQ5JFvoP97b8KdNqSKLJOCLLrmNvHFvzlrZEc=; b=8mHonLPVENdjOJeBO3sKUbU3SXD8ISghZ2IDqUPP6xhJ6ROaUTMWQfnVI9EuIbl9yP52I7 D1L1XNNYh1UndiAw== Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=DReSgi2q; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=8mHonLPV DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141267; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/eMGgmiQ5JFvoP97b8KdNqSKLJOCLLrmNvHFvzlrZEc=; b=DReSgi2quHvJIGDkjhDq7bXYAiPX5b/MC37ca4vx/RL4neTKoRiip3pEfF7VrsIoNi+Bb2 xNPZW/+qGnlhkC3nx0/AsqlHm0RifCARlAN8p/vcTQXrn8HLRfYbmafE3QT3lJ2j7tWGTb 3bIHv0hy1NDhAbjv8eYlcciHw70C3A4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141267; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/eMGgmiQ5JFvoP97b8KdNqSKLJOCLLrmNvHFvzlrZEc=; b=8mHonLPVENdjOJeBO3sKUbU3SXD8ISghZ2IDqUPP6xhJ6ROaUTMWQfnVI9EuIbl9yP52I7 D1L1XNNYh1UndiAw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 6DFF4136DB; Fri, 26 Apr 2024 14:21:05 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id iAEyDVG4K2YWcAAAD6G6ig (envelope-from ); Fri, 26 Apr 2024 14:21:05 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: berrange@redhat.com, armbru@redhat.com, Peter Xu , Claudio Fontana , Jim Fehlig Subject: [PATCH 8/9] migration: Add support for fdset with multifd + file Date: Fri, 26 Apr 2024 11:20:41 -0300 Message-Id: <20240426142042.14573-9-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240426142042.14573-1-farosas@suse.de> References: <20240426142042.14573-1-farosas@suse.de> MIME-Version: 1.0 X-Rspamd-Action: no action X-Rspamd-Queue-Id: 472A922DF8 X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; NEURAL_HAM_LONG(-1.00)[-1.000]; MID_CONTAINS_FROM(1.00)[]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; RBL_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:104:10:150:64:97:from]; SPAMHAUS_XBL(0.00)[2a07:de40:b281:104:10:150:64:97:from]; RCVD_COUNT_TWO(0.00)[2]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:dkim,suse.de:email,imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DNSWL_BLOCKED(0.00)[2a07:de40:b281:104:10:150:64:97:from]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCPT_COUNT_FIVE(0.00)[6]; DKIM_TRACE(0.00)[suse.de:+] Received-SPF: pass client-ip=195.135.223.130; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Allow multifd to use an fdset when migrating to a file. This is useful for the scenario where the management layer wants to have control over the migration file. By receiving the file descriptors directly, QEMU can delegate some high level operating system operations to the management layer (such as mandatory access control). The management layer might also want to add its own headers before the migration stream. Enable the "file:/dev/fdset/#" syntax for the multifd migration with mapped-ram. The requirements for the fdset mechanism are: On the migration source side: - the fdset must contain two fds that are not duplicates between themselves; - if direct-io is to be used, exactly one of the fds must have the O_DIRECT flag set; - the file must be opened with WRONLY both times. On the migration destination side: - the fdset must contain one fd; - the file must be opened with RDONLY. Signed-off-by: Fabiano Rosas Signed-off-by: Fabiano Rosas Reviewed-by: Daniel P. Berrangé --- docs/devel/migration/main.rst | 18 ++++++++++++++ docs/devel/migration/mapped-ram.rst | 6 ++++- migration/file.c | 38 ++++++++++++++++++++++++++++- 3 files changed, 60 insertions(+), 2 deletions(-) diff --git a/docs/devel/migration/main.rst b/docs/devel/migration/main.rst index 54385a23e5..50f6096470 100644 --- a/docs/devel/migration/main.rst +++ b/docs/devel/migration/main.rst @@ -47,6 +47,24 @@ over any transport. QEMU interference. Note that QEMU does not flush cached file data/metadata at the end of migration. + The file migration also supports using a file that has already been + opened. A set of file descriptors is passed to QEMU via an "fdset" + (see add-fd QMP command documentation). This method allows a + management application to have control over the migration file + opening operation. There are, however, strict requirements to this + interface: + + On the migration source side: + - if the multifd capability is to be used, the fdset must contain + two file descriptors that are not duplicates between themselves; + - if the direct-io capability is to be used, exactly one of the + file descriptors must have the O_DIRECT flag set; + - the file must be opened with WRONLY. + + On the migration destination side: + - the fdset must contain one file descriptor; + - the file must be opened with RDONLY. + In addition, support is included for migration using RDMA, which transports the page data using ``RDMA``, where the hardware takes care of transporting the pages, and the load on the CPU is much lower. While the diff --git a/docs/devel/migration/mapped-ram.rst b/docs/devel/migration/mapped-ram.rst index fa4cefd9fc..e6505511f0 100644 --- a/docs/devel/migration/mapped-ram.rst +++ b/docs/devel/migration/mapped-ram.rst @@ -16,7 +16,7 @@ location in the file, rather than constantly being added to a sequential stream. Having the pages at fixed offsets also allows the usage of O_DIRECT for save/restore of the migration stream as the pages are ensured to be written respecting O_DIRECT alignment -restrictions (direct-io support not yet implemented). +restrictions. Usage ----- @@ -35,6 +35,10 @@ Use a ``file:`` URL for migration: Mapped-ram migration is best done non-live, i.e. by stopping the VM on the source side before migrating. +For best performance enable the ``direct-io`` capability as well: + + ``migrate_set_capability direct-io on`` + Use-cases --------- diff --git a/migration/file.c b/migration/file.c index b9265b14dd..3bc8bc7463 100644 --- a/migration/file.c +++ b/migration/file.c @@ -17,6 +17,7 @@ #include "io/channel-file.h" #include "io/channel-socket.h" #include "io/channel-util.h" +#include "monitor/monitor.h" #include "options.h" #include "trace.h" @@ -54,10 +55,18 @@ static void file_remove_fdset(void) } } +/* + * With multifd, due to the behavior of the dup() system call, we need + * the fdset to have two non-duplicate fds so we can enable direct IO + * in the secondary channels without affecting the main channel. + */ static bool file_parse_fdset(const char *filename, int64_t *fdset_id, Error **errp) { + FdsetInfoList *fds_info; + FdsetFdInfoList *fd_info; const char *fdset_id_str; + int nfds = 0; *fdset_id = -1; @@ -71,6 +80,32 @@ static bool file_parse_fdset(const char *filename, int64_t *fdset_id, return false; } + if (!migrate_multifd() || !migrate_direct_io()) { + return true; + } + + for (fds_info = qmp_query_fdsets(NULL); fds_info; + fds_info = fds_info->next) { + + if (*fdset_id != fds_info->value->fdset_id) { + continue; + } + + for (fd_info = fds_info->value->fds; fd_info; fd_info = fd_info->next) { + if (nfds++ > 2) { + break; + } + } + } + + if (nfds != 2) { + error_setg(errp, "Outgoing migration needs two fds in the fdset, " + "got %d", nfds); + qmp_remove_fd(*fdset_id, false, -1, NULL); + *fdset_id = -1; + return false; + } + return true; } @@ -209,10 +244,11 @@ void file_start_incoming_migration(FileMigrationArgs *file_args, Error **errp) g_autofree char *filename = g_strdup(file_args->filename); QIOChannelFile *fioc = NULL; uint64_t offset = file_args->offset; + int flags = O_RDONLY; trace_migration_file_incoming(filename); - fioc = qio_channel_file_new_path(filename, O_RDONLY, 0, errp); + fioc = qio_channel_file_new_path(filename, flags, 0, errp); if (!fioc) { return; } From patchwork Fri Apr 26 14:20:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fabiano Rosas X-Patchwork-Id: 13644904 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 24CA3C4345F for ; Fri, 26 Apr 2024 14:22:45 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s0MRw-0007F2-Gl; Fri, 26 Apr 2024 10:21:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s0MRv-0007EI-1D for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:21:15 -0400 Received: from smtp-out1.suse.de ([195.135.223.130]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s0MRt-00024W-4o for qemu-devel@nongnu.org; Fri, 26 Apr 2024 10:21:14 -0400 Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 547F434E8C; Fri, 26 Apr 2024 14:21:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141270; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g8aiSWTRniGHc9b86c5xny7TEU/klyh6y5vk0UkaIyc=; b=qq0eE0n26PiAfZsOHJ0s3GYQCH/QjV1c/haZm07OXXL7KeO3B7XX3Svf+LpOLLpppTc3Sz I8cxGtwroQJENuEFTYtxI0whslVKSVfmtJCt87weqAz7Lw/AfrG9GHjZvyAHfdoIxIGw8f Rh9F+fLA/SIJeQkrOOHq/Mj4VHR9FkU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141270; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g8aiSWTRniGHc9b86c5xny7TEU/klyh6y5vk0UkaIyc=; b=bpqV2eMBP2ZxJRfWZ4mhAcXVZoYdFgmIinOG4VDNd9KbF7F8iyLzxGrq5E/e9+/ON24Jf1 rKtDFvQuZ4ThG7Dg== Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=qq0eE0n2; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=bpqV2eMB DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1714141270; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g8aiSWTRniGHc9b86c5xny7TEU/klyh6y5vk0UkaIyc=; b=qq0eE0n26PiAfZsOHJ0s3GYQCH/QjV1c/haZm07OXXL7KeO3B7XX3Svf+LpOLLpppTc3Sz I8cxGtwroQJENuEFTYtxI0whslVKSVfmtJCt87weqAz7Lw/AfrG9GHjZvyAHfdoIxIGw8f Rh9F+fLA/SIJeQkrOOHq/Mj4VHR9FkU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1714141270; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g8aiSWTRniGHc9b86c5xny7TEU/klyh6y5vk0UkaIyc=; b=bpqV2eMBP2ZxJRfWZ4mhAcXVZoYdFgmIinOG4VDNd9KbF7F8iyLzxGrq5E/e9+/ON24Jf1 rKtDFvQuZ4ThG7Dg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id BC19E136DB; Fri, 26 Apr 2024 14:21:07 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id UEN5IFO4K2YWcAAAD6G6ig (envelope-from ); Fri, 26 Apr 2024 14:21:07 +0000 From: Fabiano Rosas To: qemu-devel@nongnu.org Cc: berrange@redhat.com, armbru@redhat.com, Peter Xu , Claudio Fontana , Jim Fehlig , Thomas Huth , Laurent Vivier , Paolo Bonzini Subject: [PATCH 9/9] tests/qtest/migration: Add a test for mapped-ram with passing of fds Date: Fri, 26 Apr 2024 11:20:42 -0300 Message-Id: <20240426142042.14573-10-farosas@suse.de> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240426142042.14573-1-farosas@suse.de> References: <20240426142042.14573-1-farosas@suse.de> MIME-Version: 1.0 X-Rspamd-Action: no action X-Rspamd-Queue-Id: 547F434E8C X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; MIME_TRACE(0.00)[0:+]; FUZZY_BLOCKED(0.00)[rspamd.com]; TO_DN_SOME(0.00)[]; RBL_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:104:10:150:64:97:from]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_SEVEN(0.00)[9]; DNSWL_BLOCKED(0.00)[2a07:de40:b281:104:10:150:64:97:from]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; DKIM_TRACE(0.00)[suse.de:+]; SPAMHAUS_XBL(0.00)[2a07:de40:b281:104:10:150:64:97:from]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:dkim, suse.de:email, imap1.dmz-prg2.suse.org:helo, imap1.dmz-prg2.suse.org:rdns] Received-SPF: pass client-ip=195.135.223.130; envelope-from=farosas@suse.de; helo=smtp-out1.suse.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add a multifd test for mapped-ram with passing of fds into QEMU. This is how libvirt will consume the feature. There are a couple of details to the fdset mechanism: - multifd needs two distinct file descriptors (not duplicated with dup()) on the outgoing side so it can enable O_DIRECT only on the channels that write with alignment. The dup() system call creates file descriptors that share status flags, of which O_DIRECT is one. the incoming side doesn't set O_DIRECT, so it can dup() fds and therefore can receive only one in the fdset. - the open() access mode flags used for the fds passed into QEMU need to match the flags QEMU uses to open the file. Currently O_WRONLY for src and O_RDONLY for dst. O_DIRECT is not supported on all systems/filesystems, so run the fdset test without O_DIRECT if that's the case. The migration code should still work in that scenario. Signed-off-by: Fabiano Rosas --- tests/qtest/migration-test.c | 90 ++++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 512b7ede8b..d83f1bdd4f 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -2331,8 +2331,93 @@ static void test_multifd_file_mapped_ram_dio(void) test_file_common(&args, true); } +static void migrate_multifd_mapped_ram_fdset_dio_end(QTestState *from, + QTestState *to, + void *opaque) +{ + QDict *resp; + QList *fdsets; + + file_offset_finish_hook(from, to, opaque); + + /* + * Check that we removed the fdsets after migration, otherwise a + * second migration would fail due to too many fdsets. + */ + + resp = qtest_qmp(from, "{'execute': 'query-fdsets', " + "'arguments': {}}"); + g_assert(qdict_haskey(resp, "return")); + fdsets = qdict_get_qlist(resp, "return"); + g_assert(fdsets && qlist_empty(fdsets)); +} #endif /* O_DIRECT */ +#ifndef _WIN32 +static void *migrate_multifd_mapped_ram_fdset(QTestState *from, QTestState *to) +{ + g_autofree char *file = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); + int fds[3]; + int src_flags = O_WRONLY; + + file_dirty_offset_region(); + + /* main outgoing channel: no O_DIRECT */ + fds[0] = open(file, src_flags, 0660); + assert(fds[0] != -1); + + qtest_qmp_fds_assert_success(from, &fds[0], 1, "{'execute': 'add-fd', " + "'arguments': {'fdset-id': 1}}"); + +#ifdef O_DIRECT + src_flags |= O_DIRECT; + + /* secondary outgoing channels */ + fds[1] = open(file, src_flags, 0660); + assert(fds[1] != -1); + + qtest_qmp_fds_assert_success(from, &fds[1], 1, "{'execute': 'add-fd', " + "'arguments': {'fdset-id': 1}}"); + + /* incoming channel */ + fds[2] = open(file, O_CREAT | O_RDONLY, 0660); + assert(fds[2] != -1); + + qtest_qmp_fds_assert_success(to, &fds[2], 1, "{'execute': 'add-fd', " + "'arguments': {'fdset-id': 1}}"); + + migrate_multifd_mapped_ram_dio_start(from, to); +#else + migrate_multifd_mapped_ram_start(from, to); +#endif + + return NULL; +} + +static void test_multifd_file_mapped_ram_fdset(void) +{ + g_autofree char *uri = g_strdup_printf("file:/dev/fdset/1,offset=%d", + FILE_TEST_OFFSET); + MigrateCommon args = { + .connect_uri = uri, + .listen_uri = "defer", + .start_hook = migrate_multifd_mapped_ram_fdset, +#ifdef O_DIRECT + .finish_hook = migrate_multifd_mapped_ram_fdset_dio_end, +#endif + }; + +#ifdef O_DIRECT + if (!probe_o_direct_support(tmpfs)) { + g_test_skip("Filesystem does not support O_DIRECT"); + return; + } +#endif + + test_file_common(&args, true); +} +#endif /* _WIN32 */ + static void test_precopy_tcp_plain(void) { MigrateCommon args = { @@ -3761,6 +3846,11 @@ int main(int argc, char **argv) test_multifd_file_mapped_ram_dio); #endif +#ifndef _WIN32 + qtest_add_func("/migration/multifd/file/mapped-ram/fdset", + test_multifd_file_mapped_ram_fdset); +#endif + #ifdef CONFIG_GNUTLS migration_test_add("/migration/precopy/unix/tls/psk", test_precopy_unix_tls_psk);