From patchwork Fri Mar 8 06:24:10 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 2235191 Return-Path: X-Original-To: patchwork-kvm@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork1.kernel.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by patchwork1.kernel.org (Postfix) with ESMTP id 3E16A3FCF6 for ; Fri, 8 Mar 2013 06:24:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751864Ab3CHGYU (ORCPT ); Fri, 8 Mar 2013 01:24:20 -0500 Received: from mx1.redhat.com ([209.132.183.28]:8023 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750757Ab3CHGYT (ORCPT ); Fri, 8 Mar 2013 01:24:19 -0500 Received: from int-mx02.intmail.prod.int.phx2.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id r286OGQP022255 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 8 Mar 2013 01:24:16 -0500 Received: from [10.66.113.134] (vpn1-113-134.nay.redhat.com [10.66.113.134]) by int-mx02.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id r286OB9i024980; Fri, 8 Mar 2013 01:24:12 -0500 Message-ID: <5139840A.6020204@redhat.com> Date: Fri, 08 Mar 2013 14:24:10 +0800 From: Jason Wang User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130106 Thunderbird/17.0.2 MIME-Version: 1.0 To: netdev@vger.kernel.org CC: "Michael S. Tsirkin" , KVM , David Miller , Eric Dumazet , Stephen Hemminger , jpirko@redhat.com Subject: TCP small packets throughput and multiqueue virtio-net X-Scanned-By: MIMEDefang 2.67 on 10.5.11.12 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Hello all: I meet an issue when testing multiqueue virtio-net. When I testing guest small packets stream sending performance with netperf. I find an regression of multiqueue. When I run 2 sessions of TCP_STREAM test with 1024 byte from guest to local host, I get following result: 1q result: 3457.64 2q result: 7781.45 Statistics shows that: compared with one queue, multiqueue tends to send much more but smaller packets. Tcpdump shows single queue has a much higher possibility to produce a 64K gso packet compared to multiqueue. More but smaller packets will cause more vmexits and interrupts which lead a degradation on throughput. Then problem only exist for small packets sending. When I test with larger size, multiqueue will gradually outperfrom single queue. And multiqueue also outperfrom in both TCP_RR and pktgen test (even with small packets). The problem disappear when I turn off both gso and tso. I'm not sure whether it's a bug or expected since anyway we get improvement on latency. And if it's a bug, I suspect it was related to TCP GSO batching algorithm who tends to batch less in this situation. ( Jiri Pirko suspect it was the defect of virtio-net driver, but I didn't find any obvious clue on this). After some experiments, I find the it maybe related to tcp_tso_should_defer(), after 1) change the tcp_tso_win_divisor to 1 2) the following changes the throughput were almost the same (but still a little worse) as single queue: already? */ if ((skb != tcp_write_queue_tail(sk)) && (limit >= skb->len)) Git history shows this check were added for both westwood and fasttcp, I'm not familiar with tcp but looks like we can easily hit this check under when multiqueue is enabled for virtio-net. Maybe I was wrong but I wonder whether we can still do some batching here. Any comments, thoughts are welcomed. Thanks --- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index fd0cea1..dedd2a6 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -1777,10 +1777,12 @@ static bool tcp_tso_should_defer(struct sock *sk, struct sk_buff *skb) limit = min(send_win, cong_win); +#if 0 /* If a full-sized TSO skb can be sent, do it. */ if (limit >= min_t(unsigned int, sk->sk_gso_max_size, sk->sk_gso_max_segs * tp->mss_cache)) goto send_now; +#endif /* Middle in queue won't get any more data, full sendable