| Message ID | 20191218020119.3776-2-quintela@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Multifd Migration Compression | expand |
* Juan Quintela (quintela@redhat.com) wrote: > We can scale much better with 16, so we can scale to higher numbers. > > Signed-off-by: Juan Quintela <quintela@redhat.com> > --- > migration/migration.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/migration/migration.c b/migration/migration.c > index 354ad072fa..e7f707e033 100644 > --- a/migration/migration.c > +++ b/migration/migration.c > @@ -86,7 +86,7 @@ > > /* The delay time (in ms) between two COLO checkpoints */ > #define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY (200 * 100) > -#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2 > +#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 16 OK, I suspect the management apps will set it anyway; some could be a bit surprised chewing 16 cores/threads of CPU in some cases. Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com> > /* Background transfer rate for postcopy, 0 means unlimited, note > * that page requests can still exceed this limit. > -- > 2.23.0 > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
On Wed, Dec 18, 2019 at 03:01:10AM +0100, Juan Quintela wrote:
> We can scale much better with 16, so we can scale to higher numbers.
What was the test scenario showing such scaling ?
In the real world I'm sceptical that virt hosts will have
16 otherwise idle CPU cores available that are permissible
to use for migration, or indeed whether they'll have network
bandwidth available to allow 16 cores to saturate the link.
Regards,
Daniel
* Daniel P. Berrangé (berrange@redhat.com) wrote: > On Wed, Dec 18, 2019 at 03:01:10AM +0100, Juan Quintela wrote: > > We can scale much better with 16, so we can scale to higher numbers. > > What was the test scenario showing such scaling ? > > In the real world I'm sceptical that virt hosts will have > 16 otherwise idle CPU cores available that are permissible > to use for migration, or indeed whether they'll have network > bandwidth available to allow 16 cores to saturate the link. With TLS or compression, the network bandwidth could easily be there. Dave > > Regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
On Fri, Jan 03, 2020 at 05:01:14PM +0000, Dr. David Alan Gilbert wrote: > * Daniel P. Berrangé (berrange@redhat.com) wrote: > > On Wed, Dec 18, 2019 at 03:01:10AM +0100, Juan Quintela wrote: > > > We can scale much better with 16, so we can scale to higher numbers. > > > > What was the test scenario showing such scaling ? > > > > In the real world I'm sceptical that virt hosts will have > > 16 otherwise idle CPU cores available that are permissible > > to use for migration, or indeed whether they'll have network > > bandwidth available to allow 16 cores to saturate the link. > > With TLS or compression, the network bandwidth could easily be there. Yes, but this constant is setting a default that applies regardless of whether TLS / compression is enabled and/or whether CPU cores are idle. IOW, there can be cases where using 16 threads will be a perf win, I'm just questioning the suitability as a global default out of the box. I feel like what's really lacking with migration is documentation around the usefulness of the very many parameters, and the various interesting combinations & tradeoffs around enabling them. So instead of changing the default threads, can we focusing on improving documentation so that mgmt apps have good information on which to make the decision about whether & when to use 2 or 16 or $NNN migration threads. Regards, Daniel
* Daniel P. Berrangé (berrange@redhat.com) wrote: > On Fri, Jan 03, 2020 at 05:01:14PM +0000, Dr. David Alan Gilbert wrote: > > * Daniel P. Berrangé (berrange@redhat.com) wrote: > > > On Wed, Dec 18, 2019 at 03:01:10AM +0100, Juan Quintela wrote: > > > > We can scale much better with 16, so we can scale to higher numbers. > > > > > > What was the test scenario showing such scaling ? > > > > > > In the real world I'm sceptical that virt hosts will have > > > 16 otherwise idle CPU cores available that are permissible > > > to use for migration, or indeed whether they'll have network > > > bandwidth available to allow 16 cores to saturate the link. > > > > With TLS or compression, the network bandwidth could easily be there. > > Yes, but this constant is setting a default that applies regardless of > whether TLS / compression is enabled and/or whether CPU cores are idle. > IOW, there can be cases where using 16 threads will be a perf win, I'm > just questioning the suitability as a global default out of the box. > > I feel like what's really lacking with migration is documentation > around the usefulness of the very many parameters, and the various > interesting combinations & tradeoffs around enabling them. So instead > of changing the default threads, can we focusing on improving > documentation so that mgmt apps have good information on which to > make the decision about whether & when to use 2 or 16 or $NNN migration > threads. Yes, although the short answer is; increase it if you find your migration threads are saturated, either due to a very fast network connection, or a CPU heavy setting (such as TLS or compression). The answer to that might also vary if you have compression/encryption offload engines (which I'd like to try). Given that this series is for compression, I guess that's the use Juan is using here. On a 100Gbps NIC (which are readily available these days), I managed to squeeze 70Gbps out of an earlier multifd version with 8 channels, which beat the RDMA code in throughput (albeit eating CPU). Dave > Regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
On Wed, Dec 18, 2019 at 03:01:10AM +0100, Juan Quintela wrote: > We can scale much better with 16, so we can scale to higher numbers. > > Signed-off-by: Juan Quintela <quintela@redhat.com> > --- > migration/migration.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/migration/migration.c b/migration/migration.c > index 354ad072fa..e7f707e033 100644 > --- a/migration/migration.c > +++ b/migration/migration.c > @@ -86,7 +86,7 @@ > > /* The delay time (in ms) between two COLO checkpoints */ > #define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY (200 * 100) > -#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2 > +#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 16 I think this can break compatibility for migration between new and old QEMU. Consider a deployment has enabled multifd, but not given an explicit number of channels, and now try to start the migration. In the method multifd_recv_initial_packet() there is a check if (msg.id > migrate_multifd_channels()) { error_setg(errp, "multifd: received channel version %d " "expected %d", msg.version, MULTIFD_VERSION); return -1; } which will fail if migrating from new QEMU to old QEMU because migrate_multifd_channels() will be 2 for old QEMU and new QEMU will be trying to open 16 channels. Regards, Daniel
Daniel P. Berrangé <berrange@redhat.com> wrote: > On Wed, Dec 18, 2019 at 03:01:10AM +0100, Juan Quintela wrote: >> We can scale much better with 16, so we can scale to higher numbers. > > What was the test scenario showing such scaling ? On my test hardware, with 2 channels we can saturate around 8Gigabit max, more than that, and the migration thread is not fast enough to fill the network bandwidth. With 8 that is enough to fill whatever we can find. We used to have a bug where we were getting trouble with more channels than cores. That was the initial reason why the default was so low. So, pros/cons are: - have low value (2). We are backwards compatible, but we are not using all bandwith. Notice that we will dectect the error before 5.0 is out and print a good error message. - have high value (I tested 8 and 16). Found no performance loss when moving to lower bandwidth limits, and clearly we were able to saturate the higher speeds (I tested on localhost, so I had big enough bandwidth) > In the real world I'm sceptical that virt hosts will have > 16 otherwise idle CPU cores available that are permissible > to use for migration, or indeed whether they'll have network > bandwidth available to allow 16 cores to saturate the link. The problem here is that if you have such a host, and you want to have high speed migration, you need to configure it. My measumermets are that high number of channels don't affect performance with low bandwidth, but low number of channels affect performance with high bandwidth speed. So, if we want to have something that works "automatically" everywhere, we need to put it to at least 8. Or we can trust that management app will do the right thing. If you are using a low value of bandwidth, the only difference with 16 channels is that you are using a bit more memory (just the space for the stacks) and that you are having less contention for the locks (but with low bandwidth you are not having contention anyways). So, I think that the question is: - What does libvirt prefferes - What does ovirt/openstack preffer - Do we really want that the user "have" to configure that value I don't really care one way or another. Thanks, Juan. PD. On next patch submission I will make it be 2 for old machine types, it is not difficult and makes the backward compatibility problem go away.
On Fri, Jan 03, 2020 at 07:25:08PM +0100, Juan Quintela wrote: > Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Wed, Dec 18, 2019 at 03:01:10AM +0100, Juan Quintela wrote: > >> We can scale much better with 16, so we can scale to higher numbers. > > > > What was the test scenario showing such scaling ? > > On my test hardware, with 2 channels we can saturate around 8Gigabit max, > more than that, and the migration thread is not fast enough to fill the > network bandwidth. > > With 8 that is enough to fill whatever we can find. > We used to have a bug where we were getting trouble with more channels > than cores. That was the initial reason why the default was so low. > > So, pros/cons are: > - have low value (2). We are backwards compatible, but we are not using > all bandwith. Notice that we will dectect the error before 5.0 is > out and print a good error message. > > - have high value (I tested 8 and 16). Found no performance loss when > moving to lower bandwidth limits, and clearly we were able to saturate > the higher speeds (I tested on localhost, so I had big enough bandwidth) > > > > In the real world I'm sceptical that virt hosts will have > > 16 otherwise idle CPU cores available that are permissible > > to use for migration, or indeed whether they'll have network > > bandwidth available to allow 16 cores to saturate the link. > > The problem here is that if you have such a host, and you want to have > high speed migration, you need to configure it. My measumermets are > that high number of channels don't affect performance with low > bandwidth, but low number of channels affect performance with high > bandwidth speed. I'm not concerned about impact on performance of migration on a low bandwidth link, rather I'm concerned about impact on performance of other guests on the host. It will cause migration to contend with other guest's vCPUs and network traffic. > So, if we want to have something that works "automatically" everywhere, > we need to put it to at least 8. Or we can trust that management app > will do the right thing. Aren't we still setting the bandwidth limit to MB bandwidth out of the box, so we already require mgmt app to change settings to use more bandwidth ? > If you are using a low value of bandwidth, the only difference with 16 > channels is that you are using a bit more memory (just the space for the > stacks) and that you are having less contention for the locks (but with > low bandwidth you are not having contention anyways). > > So, I think that the question is: > - What does libvirt prefferes Libvirt doesn't really have an opinion in this case. I believe we'll always set the number of channels on both src & dst, so we don't see the defaults. > - What does ovirt/openstack preffer Libvirt should insulate them from any change in defaults in QEMU in this case, but always explicitly setting channels on src & dst to match. > - Do we really want that the user "have" to configure that value Right so this is the key quesiton - for a user not using libvirt or a libvirt based mgmt app, what we do want out out of the box migration to be tuned for ? If we want to maximise migration performance, at cost of anything else, then we can change the migration channels count, but probably also ought to remove the 32MB bandwidth cap as no useful guest with active apps will succeed migration with a 32MB cap. Regards, Daniel
Daniel P. Berrangé <berrange@redhat.com> wrote: > On Fri, Jan 03, 2020 at 07:25:08PM +0100, Juan Quintela wrote: >> Daniel P. Berrangé <berrange@redhat.com> wrote: >> > On Wed, Dec 18, 2019 at 03:01:10AM +0100, Juan Quintela wrote: >> >> We can scale much better with 16, so we can scale to higher numbers. >> > >> > What was the test scenario showing such scaling ? >> >> On my test hardware, with 2 channels we can saturate around 8Gigabit max, >> more than that, and the migration thread is not fast enough to fill the >> network bandwidth. >> >> With 8 that is enough to fill whatever we can find. >> We used to have a bug where we were getting trouble with more channels >> than cores. That was the initial reason why the default was so low. >> >> So, pros/cons are: >> - have low value (2). We are backwards compatible, but we are not using >> all bandwith. Notice that we will dectect the error before 5.0 is >> out and print a good error message. >> >> - have high value (I tested 8 and 16). Found no performance loss when >> moving to lower bandwidth limits, and clearly we were able to saturate >> the higher speeds (I tested on localhost, so I had big enough bandwidth) >> >> >> > In the real world I'm sceptical that virt hosts will have >> > 16 otherwise idle CPU cores available that are permissible >> > to use for migration, or indeed whether they'll have network >> > bandwidth available to allow 16 cores to saturate the link. >> >> The problem here is that if you have such a host, and you want to have >> high speed migration, you need to configure it. My measumermets are >> that high number of channels don't affect performance with low >> bandwidth, but low number of channels affect performance with high >> bandwidth speed. > > I'm not concerned about impact on performance of migration on a > low bandwidth link, rather I'm concerned about impact on performance > of other guests on the host. It will cause migration to contend with > other guest's vCPUs and network traffic. Two things here: - vcpus: If you want migration to consume all the bandwidth, you are happy with it using more vcpus. - bandwidth: It will only consume only the one that the guest has assigned, split (we hope evenly) between all the channels. My main reason to have a higher number of channels is: - test better the code with more than one channel - work "magically" well in all scenarios. With a low number of channels, we are not going to be able to saturate a big network pipe. > >> So, if we want to have something that works "automatically" everywhere, >> we need to put it to at least 8. Or we can trust that management app >> will do the right thing. > > Aren't we still setting the bandwidth limit to MB bandwidth out of the > box, so we already require mgmt app to change settings to use more > bandwidth ? Yeap. This is the default bandwidth. #define MAX_THROTTLE (32 << 20) >> If you are using a low value of bandwidth, the only difference with 16 >> channels is that you are using a bit more memory (just the space for the >> stacks) and that you are having less contention for the locks (but with >> low bandwidth you are not having contention anyways). >> >> So, I think that the question is: Note that my idea is to make multifd "default" in the near future (5.1 timeframe or so). >> - What does libvirt prefferes > > Libvirt doesn't really have an opinion in this case. I believe we'll > always set the number of channels on both src & dst, so we don't > see the defaults. What does libvirt does today for this value? >> - What does ovirt/openstack preffer > > Libvirt should insulate them from any change in defaults in QEMU > in this case, but always explicitly setting channels on src & dst > to match. I agree here, they should don't care by default. >> - Do we really want that the user "have" to configure that value > > Right so this is the key quesiton - for a user not using libvirt > or a libvirt based mgmt app, what we do want out out of the box > migration to be tuned for ? In my opinion, we should have something like: - multifd: enabled by default - max downtime: 300 ms (current) looks right to me - max bandwidth: 32MB/s (current) seems a bit low. 100MB/s (i.e. almost full gigabit ethernet) seems reasonable to me. Having a default for 10Gigabit ethernet or similar seems too high. > If we want to maximise migration performance, at cost of anything > else, then we can change the migration channels count, but probably > also ought to remove the 32MB bandwidth cap as no useful guest with > active apps will succeed migration with a 32MB cap. Will start another series with the current values to discuss all the defaults, ok? thanks for the comments, Juan.
On Tue, Jan 07, 2020 at 02:32:24PM +0100, Juan Quintela wrote: > Daniel P. Berrangé <berrange@redhat.com> wrote: > >> - What does libvirt prefferes > > > > Libvirt doesn't really have an opinion in this case. I believe we'll > > always set the number of channels on both src & dst, so we don't > > see the defaults. > > What does libvirt does today for this value? Libvirt doesn't have any default value. By default migration uses a single connection. The mgmt app can request multiple connections for the migration, which will cause libvirt to turn on multifd with the request channel count. Essentially we avoid the problem by not having a separate tunable for "multifd enable" & "multifd channels" - we only have one tunable "connection count". Thus the idea of enabling multifd with a default channel count doesn't exist. Regards, Daniel
diff --git a/migration/migration.c b/migration/migration.c index 354ad072fa..e7f707e033 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -86,7 +86,7 @@ /* The delay time (in ms) between two COLO checkpoints */ #define DEFAULT_MIGRATE_X_CHECKPOINT_DELAY (200 * 100) -#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2 +#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 16 /* Background transfer rate for postcopy, 0 means unlimited, note * that page requests can still exceed this limit.
We can scale much better with 16, so we can scale to higher numbers. Signed-off-by: Juan Quintela <quintela@redhat.com> --- migration/migration.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)