diff mbox

vhost: get 2% performance improved by reducing spin_lock race in vhost_work_queue

Message ID 5872DA217C2FF7488B20897D84F904E7338FD2C5@nkgeml511-mbx.china.huawei.com (mailing list archive)
State New, archived
Headers show

Commit Message

Qinchuanyu May 20, 2013, 4:22 a.m. UTC 
The patch below is base on 
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/tree/drivers/vhost/vhost.c?id=refs/tags/next-20130517

Signed-off-by: Chuanyu Qin <qinchuanyu@huawei.com>


I did the test by using iperf in 10G environment, the test num as below:
                 orignal                   modified
thread_num  tp(Gbps)   vhost(%)  |  tp(Gbps)     vhost(%)
1           9.59         28.82   |      9.59        27.49
8            9.61        32.92   |      9.62        26.77
64            9.58        46.48  |     9.55        38.99
256            9.6        63.7   |      9.6         52.59

The cost of vhost reduced while the throughput is almost unchanged.

On 05/20/2013 11:06 AM, Qinchuanyu wrote:
> Right now the wake_up_process func is included in spin_lock/unlock, but it could be done outside the spin_lock.

> I have test it with kernel 3.0.27 and guest suse11-sp2, it provide 2%-3% net performance improved.

>

> Signed-off-by: Chuanyu Qin <qinchuanyu@huawei.com>


Make sense to me but need generate a patch against net-next.git or
vhost.git in git.kernel.org.

Btw. How did you test this? Care to share the perf numbers?

Thanks
> mu

> --- a/drivers/vhost/vhost.c     2013-05-20 10:36:30.000000000 +0800

> +++ b/drivers/vhost/vhost.c     2013-05-20 10:36:54.000000000 +0800

> @@ -144,9 +144,10 @@

>         if (list_empty(&work->node)) {

>                 list_add_tail(&work->node, &dev->work_list);

>                 work->queue_seq++;

> +               spin_unlock_irqrestore(&dev->work_lock, flags);

>                 wake_up_process(dev->worker);

> -       }

> -       spin_unlock_irqrestore(&dev->work_lock, flags);

> +       } else

> +               spin_unlock_irqrestore(&dev->work_lock, flags);

>  }

>  

>  void vhost_poll_queue(struct vhost_poll *poll)

> --

> To unsubscribe from this list: send the line "unsubscribe netdev" in

> the body of a message to majordomo@vger.kernel.org

> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Comments

Jason Wang May 20, 2013, 4:46 a.m. UTC | #1 
On 05/20/2013 12:22 PM, Qinchuanyu wrote:
> The patch below is base on 
> https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/tree/drivers/vhost/vhost.c?id=refs/tags/next-20130517
>
> Signed-off-by: Chuanyu Qin <qinchuanyu@huawei.com>
> --- a/drivers/vhost/vhost.c     2013-05-20 11:47:05.000000000 +0800
> +++ b/drivers/vhost/vhost.c     2013-05-20 11:48:24.000000000 +0800
> @@ -154,9 +154,10 @@
>         if (list_empty(&work->node)) {
>                 list_add_tail(&work->node, &dev->work_list);
>                 work->queue_seq++;
> +               spin_unlock_irqrestore(&dev->work_lock, flags);
>                 wake_up_process(dev->worker);
> -       }
> -       spin_unlock_irqrestore(&dev->work_lock, flags);
> +       } else
> +               spin_unlock_irqrestore(&dev->work_lock, flags);
>  }
>  
>  void vhost_poll_queue(struct vhost_poll *poll)
>
> I did the test by using iperf in 10G environment, the test num as below:
>                  orignal                   modified
> thread_num  tp(Gbps)   vhost(%)  |  tp(Gbps)     vhost(%)
> 1           9.59         28.82   |      9.59        27.49
> 8            9.61        32.92   |      9.62        26.77
> 64            9.58        46.48  |     9.55        38.99
> 256            9.6        63.7   |      9.6         52.59
>
> The cost of vhost reduced while the throughput is almost unchanged.

Thanks, and please generate a formal patch based on
Documentation/SubmittingPatches (put the description and perf numbers in
the commit log). Then resubmit it to let the maintainer apply it.

>
> On 05/20/2013 11:06 AM, Qinchuanyu wrote:
>> Right now the wake_up_process func is included in spin_lock/unlock, but it could be done outside the spin_lock.
>> I have test it with kernel 3.0.27 and guest suse11-sp2, it provide 2%-3% net performance improved.
>>
>> Signed-off-by: Chuanyu Qin <qinchuanyu@huawei.com>
> Make sense to me but need generate a patch against net-next.git or
> vhost.git in git.kernel.org.
>
> Btw. How did you test this? Care to share the perf numbers?
>
> Thanks
>> mu
>> --- a/drivers/vhost/vhost.c     2013-05-20 10:36:30.000000000 +0800
>> +++ b/drivers/vhost/vhost.c     2013-05-20 10:36:54.000000000 +0800
>> @@ -144,9 +144,10 @@
>>         if (list_empty(&work->node)) {
>>                 list_add_tail(&work->node, &dev->work_list);
>>                 work->queue_seq++;
>> +               spin_unlock_irqrestore(&dev->work_lock, flags);
>>                 wake_up_process(dev->worker);
>> -       }
>> -       spin_unlock_irqrestore(&dev->work_lock, flags);
>> +       } else
>> +               spin_unlock_irqrestore(&dev->work_lock, flags);
>>  }
>>  
>>  void vhost_poll_queue(struct vhost_poll *poll)
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> N?????r??y???b?X???v?^?)?{.n?+???z?^?)???w*jg?????????j/???z????2?????&?)??a?????G???h??j:+v???w??

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Qinchuanyu May 21, 2013, 2:40 a.m. UTC | #2 
RnJvbTogQ2h1YW55dSBRaW4gPHFpbmNodWFueXVAaHVhd2VpLmNvbT4NClN1YmplY3Q6IFtQQVRD
SF0gZ2V0IDIlIG9yIG1vcmUgcGVyZm9ybWFuY2UgaW1wcm92ZWQgYnkgcmVkdWNpbmcgc3Bpbl9s
b2NrIHJhY2UgaW4gdmhvc3Rfd29ya19xdWV1ZQ0KDQp0aGUgd2FrZV91cF9wcm9jZXNzIGZ1bmMg
aXMgaW5jbHVkZWQgYnkgc3Bpbl9sb2NrL3VubG9jayBpbiB2aG9zdF93b3JrX3F1ZXVlLCANCmJ1
dCBpdCBjb3VsZCBiZSBkb25lIG91dHNpZGUgdGhlIHNwaW5fbG9jay4gDQpJIGhhdmUgdGVzdCBp
dCB3aXRoIGtlcm5lbCAzLjAuMjcgYW5kIGd1ZXN0IHN1c2UxMS1zcDIgdXNpbmcgaXBlcmYsIHRo
ZSBudW0gYXMgYmVsb3cuDQogICAgICAgICAgICAgICAgIG9yaWduYWwgICAgICAgICAgICAgICAg
ICAgbW9kaWZpZWQNCnRocmVhZF9udW0gIHRwKEdicHMpICAgdmhvc3QoJSkgIHwgIHRwKEdicHMp
ICAgICB2aG9zdCglKQ0KMSAgICAgICAgICAgOS41OSAgICAgICAgIDI4LjgyICAgfCAgICAgIDku
NTkgICAgICAgIDI3LjQ5DQo4ICAgICAgICAgICAgOS42MSAgICAgICAgMzIuOTIgICB8ICAgICAg
OS42MiAgICAgICAgMjYuNzcNCjY0ICAgICAgICAgICAgOS41OCAgICAgICAgNDYuNDggIHwgICAg
IDkuNTUgICAgICAgIDM4Ljk5DQoyNTYgICAgICAgICAgICA5LjYgICAgICAgIDYzLjcgICB8ICAg
ICAgOS42ICAgICAgICAgNTIuNTkNCg0KU2lnbmVkLW9mZi1ieTogQ2h1YW55dSBRaW4gPHFpbmNo
dWFueXVAaHVhd2VpLmNvbT4NCi0tLQ0KIGRyaXZlcnMvdmhvc3Qvdmhvc3QuYyB8ICAgIDUgKysr
LS0NCiAxIGZpbGVzIGNoYW5nZWQsIDMgaW5zZXJ0aW9ucygrKSwgMiBkZWxldGlvbnMoLSkNCg0K
ZGlmZiAtLWdpdCBhL2RyaXZlcnMvdmhvc3Qvdmhvc3QuYyBiL2RyaXZlcnMvdmhvc3Qvdmhvc3Qu
Yw0KaW5kZXggOTRkYmQyNS4uOGJlZTEwOSAxMDA2NDQNCi0tLSBhL2RyaXZlcnMvdmhvc3Qvdmhv
c3QuYw0KKysrIGIvZHJpdmVycy92aG9zdC92aG9zdC5jDQpAQCAtMTQ2LDkgKzE0NiwxMCBAQCBz
dGF0aWMgaW5saW5lIHZvaWQgdmhvc3Rfd29ya19xdWV1ZShzdHJ1Y3Qgdmhvc3RfZGV2ICpkZXYs
DQogCWlmIChsaXN0X2VtcHR5KCZ3b3JrLT5ub2RlKSkgew0KIAkJbGlzdF9hZGRfdGFpbCgmd29y
ay0+bm9kZSwgJmRldi0+d29ya19saXN0KTsNCiAJCXdvcmstPnF1ZXVlX3NlcSsrOw0KKwkJc3Bp
bl91bmxvY2tfaXJxcmVzdG9yZSgmZGV2LT53b3JrX2xvY2ssIGZsYWdzKTsNCiAJCXdha2VfdXBf
cHJvY2VzcyhkZXYtPndvcmtlcik7DQotCX0NCi0Jc3Bpbl91bmxvY2tfaXJxcmVzdG9yZSgmZGV2
LT53b3JrX2xvY2ssIGZsYWdzKTsNCisJfSBlbHNlDQorCQlzcGluX3VubG9ja19pcnFyZXN0b3Jl
KCZkZXYtPndvcmtfbG9jaywgZmxhZ3MpOw0KIH0NCiANCiB2b2lkIHZob3N0X3BvbGxfcXVldWUo
c3RydWN0IHZob3N0X3BvbGwgKnBvbGwpDQotLSANCjEuNy4zLjEubXN5c2dpdC4wDQoNCg0KPiBP
biAwNS8yMC8yMDEzIDEyOjIyIFBNLCBRaW5jaHVhbnl1IHdyb3RlOg0KPiA+IFRoZSBwYXRjaCBi
ZWxvdyBpcyBiYXNlIG9uDQo+ID4gaHR0cHM6Ly9naXQua2VybmVsLm9yZy9jZ2l0L2xpbnV4L2tl
cm5lbC9naXQvbmV4dC9saW51eC0NCj4gbmV4dC5naXQvdHJlZS9kcml2ZXJzL3Zob3N0L3Zob3N0
LmM/aWQ9cmVmcy90YWdzL25leHQtMjAxMzA1MTcNCj4gPg0KPiA+IFNpZ25lZC1vZmYtYnk6IENo
dWFueXUgUWluIDxxaW5jaHVhbnl1QGh1YXdlaS5jb20+DQo+ID4gLS0tIGEvZHJpdmVycy92aG9z
dC92aG9zdC5jICAgICAyMDEzLTA1LTIwIDExOjQ3OjA1LjAwMDAwMDAwMCArMDgwMA0KPiA+ICsr
KyBiL2RyaXZlcnMvdmhvc3Qvdmhvc3QuYyAgICAgMjAxMy0wNS0yMCAxMTo0ODoyNC4wMDAwMDAw
MDAgKzA4MDANCj4gPiBAQCAtMTU0LDkgKzE1NCwxMCBAQA0KPiA+ICAgICAgICAgaWYgKGxpc3Rf
ZW1wdHkoJndvcmstPm5vZGUpKSB7DQo+ID4gICAgICAgICAgICAgICAgIGxpc3RfYWRkX3RhaWwo
JndvcmstPm5vZGUsICZkZXYtPndvcmtfbGlzdCk7DQo+ID4gICAgICAgICAgICAgICAgIHdvcmst
PnF1ZXVlX3NlcSsrOw0KPiA+ICsgICAgICAgICAgICAgICBzcGluX3VubG9ja19pcnFyZXN0b3Jl
KCZkZXYtPndvcmtfbG9jaywgZmxhZ3MpOw0KPiA+ICAgICAgICAgICAgICAgICB3YWtlX3VwX3By
b2Nlc3MoZGV2LT53b3JrZXIpOw0KPiA+IC0gICAgICAgfQ0KPiA+IC0gICAgICAgc3Bpbl91bmxv
Y2tfaXJxcmVzdG9yZSgmZGV2LT53b3JrX2xvY2ssIGZsYWdzKTsNCj4gPiArICAgICAgIH0gZWxz
ZQ0KPiA+ICsgICAgICAgICAgICAgICBzcGluX3VubG9ja19pcnFyZXN0b3JlKCZkZXYtPndvcmtf
bG9jaywgZmxhZ3MpOw0KPiA+ICB9DQo+ID4NCj4gPiAgdm9pZCB2aG9zdF9wb2xsX3F1ZXVlKHN0
cnVjdCB2aG9zdF9wb2xsICpwb2xsKQ0KPiA+DQo+ID4gSSBkaWQgdGhlIHRlc3QgYnkgdXNpbmcg
aXBlcmYgaW4gMTBHIGVudmlyb25tZW50LCB0aGUgdGVzdCBudW0gYXMNCj4gYmVsb3c6DQo+ID4g
ICAgICAgICAgICAgICAgICBvcmlnbmFsICAgICAgICAgICAgICAgICAgIG1vZGlmaWVkDQo+ID4g
dGhyZWFkX251bSAgdHAoR2JwcykgICB2aG9zdCglKSAgfCAgdHAoR2JwcykgICAgIHZob3N0KCUp
DQo+ID4gMSAgICAgICAgICAgOS41OSAgICAgICAgIDI4LjgyICAgfCAgICAgIDkuNTkgICAgICAg
IDI3LjQ5DQo+ID4gOCAgICAgICAgICAgIDkuNjEgICAgICAgIDMyLjkyICAgfCAgICAgIDkuNjIg
ICAgICAgIDI2Ljc3DQo+ID4gNjQgICAgICAgICAgICA5LjU4ICAgICAgICA0Ni40OCAgfCAgICAg
OS41NSAgICAgICAgMzguOTkNCj4gPiAyNTYgICAgICAgICAgICA5LjYgICAgICAgIDYzLjcgICB8
ICAgICAgOS42ICAgICAgICAgNTIuNTkNCj4gPg0KPiA+IFRoZSBjb3N0IG9mIHZob3N0IHJlZHVj
ZWQgd2hpbGUgdGhlIHRocm91Z2hwdXQgaXMgYWxtb3N0IHVuY2hhbmdlZC4NCj4gDQo+IFRoYW5r
cywgYW5kIHBsZWFzZSBnZW5lcmF0ZSBhIGZvcm1hbCBwYXRjaCBiYXNlZCBvbg0KPiBEb2N1bWVu
dGF0aW9uL1N1Ym1pdHRpbmdQYXRjaGVzIChwdXQgdGhlIGRlc2NyaXB0aW9uIGFuZCBwZXJmIG51
bWJlcnMNCj4gaW4gdGhlIGNvbW1pdCBsb2cpLiBUaGVuIHJlc3VibWl0IGl0IHRvIGxldCB0aGUg
bWFpbnRhaW5lciBhcHBseSBpdC4NCg0K
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

--- a/drivers/vhost/vhost.c     2013-05-20 11:47:05.000000000 +0800
+++ b/drivers/vhost/vhost.c     2013-05-20 11:48:24.000000000 +0800
@@ -154,9 +154,10 @@ 
        if (list_empty(&work->node)) {
                list_add_tail(&work->node, &dev->work_list);
                work->queue_seq++;
+               spin_unlock_irqrestore(&dev->work_lock, flags);
                wake_up_process(dev->worker);
-       }
-       spin_unlock_irqrestore(&dev->work_lock, flags);
+       } else
+               spin_unlock_irqrestore(&dev->work_lock, flags);
 }
 
 void vhost_poll_queue(struct vhost_poll *poll)