From patchwork Fri Mar 7 05:28:27 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 3787671 Return-Path: X-Original-To: patchwork-kvm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id C00DABF540 for ; Fri, 7 Mar 2014 05:29:30 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id C0BB520272 for ; Fri, 7 Mar 2014 05:29:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C15D820270 for ; Fri, 7 Mar 2014 05:29:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751003AbaCGF25 (ORCPT ); Fri, 7 Mar 2014 00:28:57 -0500 Received: from mx1.redhat.com ([209.132.183.28]:12504 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750905AbaCGF2z (ORCPT ); Fri, 7 Mar 2014 00:28:55 -0500 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s275Snxm007772 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 7 Mar 2014 00:28:49 -0500 Received: from jason-ThinkPad-T430s.nay.redhat.com ([10.66.4.174]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s275SiJ0023604; Fri, 7 Mar 2014 00:28:45 -0500 From: Jason Wang To: mst@redhat.com, kvm@vger.kernel.org, virtio-dev@lists.oasis-open.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jason Wang , Qin Chuanyu Subject: [PATCH net V2] vhost: net: switch to use data copy if pending DMAs exceed the limit Date: Fri, 7 Mar 2014 13:28:27 +0800 Message-Id: <1394170107-12018-1-git-send-email-jasowang@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We used to stop the handling of tx when the number of pending DMAs exceeds VHOST_MAX_PEND. This is used to reduce the memory occupation of both host and guest. But it was too aggressive in some cases, since any delay or blocking of a single packet may delay or block the guest transmission. Consider the following setup: +-----+ +-----+ | VM1 | | VM2 | +--+--+ +--+--+ | | +--+--+ +--+--+ | tap0| | tap1| +--+--+ +--+--+ | | pfifo_fast htb(10Mbit/s) | | +--+--------------+---+ | bridge | +--+------------------+ | pfifo_fast | +-----+ | eth0|(100Mbit/s) +-----+ - start two VMs and connect them to a bridge - add an physical card (100Mbit/s) to that bridge - setup htb on tap1 and limit its throughput to 10Mbit/s - run two netperfs in the same time, one is from VM1 to VM2. Another is from VM1 to an external host through eth0. - result shows that not only the VM1 to VM2 traffic were throttled but also the VM1 to external host through eth0 is also throttled somehow. This is because the delay added by htb may lead the delay the finish of DMAs and cause the pending DMAs for tap0 exceeds the limit (VHOST_MAX_PEND). In this case vhost stop handling tx request until htb send some packets. The problem here is all of the packets transmission were blocked even if it does not go to VM2. We can solve this issue by relaxing it a little bit: switching to use data copy instead of stopping tx when the number of pending DMAs exceed half of the vq size. This is safe because: - The number of pending DMAs were still limited (half of the vq size) - The out of order completion during mode switch can make sure that most of the tx buffers were freed in time in guest. So even if about 50% packets were delayed in zero-copy case, vhost could continue to do the transmission through data copy in this case. Test result: Before this patch: VM1 to VM2 throughput is 9.3Mbit/s VM1 to External throughput is 40Mbit/s CPU utilization is 7% After this patch: VM1 to VM2 throughput is 9.3Mbit/s Vm1 to External throughput is 93Mbit/s CPU utilization is 16% Completed performance test on 40gbe shows no obvious changes in both throughput and cpu utilization with this patch. The patch only solve this issue when unlimited sndbuf. We still need a solution for limited sndbuf. Cc: Michael S. Tsirkin Cc: Qin Chuanyu Signed-off-by: Jason Wang Reviewed-by: Qin chuanyu --- Changes from V1: - Remove VHOST_MAX_PEND and switch to use half of the vq size as the limit - Add cpu utilization in commit log --- drivers/vhost/net.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index a0fa5de..2925e9a 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -38,8 +38,6 @@ MODULE_PARM_DESC(experimental_zcopytx, "Enable Zero Copy TX;" * Using this limit prevents one virtqueue from starving others. */ #define VHOST_NET_WEIGHT 0x80000 -/* MAX number of TX used buffers for outstanding zerocopy */ -#define VHOST_MAX_PEND 128 #define VHOST_GOODCOPY_LEN 256 /* @@ -345,7 +343,7 @@ static void handle_tx(struct vhost_net *net) .msg_flags = MSG_DONTWAIT, }; size_t len, total_len = 0; - int err; + int err, num_pends; size_t hdr_size; struct socket *sock; struct vhost_net_ubuf_ref *uninitialized_var(ubufs); @@ -366,13 +364,6 @@ static void handle_tx(struct vhost_net *net) if (zcopy) vhost_zerocopy_signal_used(net, vq); - /* If more outstanding DMAs, queue the work. - * Handle upend_idx wrap around - */ - if (unlikely((nvq->upend_idx + vq->num - VHOST_MAX_PEND) - % UIO_MAXIOV == nvq->done_idx)) - break; - head = vhost_get_vq_desc(&net->dev, vq, vq->iov, ARRAY_SIZE(vq->iov), &out, &in, @@ -405,9 +396,13 @@ static void handle_tx(struct vhost_net *net) break; } + num_pends = likely(nvq->upend_idx >= nvq->done_idx) ? + (nvq->upend_idx - nvq->done_idx) : + (nvq->upend_idx + UIO_MAXIOV - + nvq->done_idx); + zcopy_used = zcopy && len >= VHOST_GOODCOPY_LEN - && (nvq->upend_idx + 1) % UIO_MAXIOV != - nvq->done_idx + && num_pends <= vq->num >> 1 && vhost_net_tx_select_zcopy(net); /* use msg_control to pass vhost zerocopy ubuf info to skb */