From patchwork Mon Nov 14 16:25:05 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 9427891 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id CD2F96047D for ; Mon, 14 Nov 2016 16:25:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C8F6F2862E for ; Mon, 14 Nov 2016 16:25:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BDCFD28A2F; Mon, 14 Nov 2016 16:25:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id D68222862E for ; Mon, 14 Nov 2016 16:25:52 +0000 (UTC) Received: from localhost ([::1]:41203 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c6K4t-0004Qo-Q1 for patchwork-qemu-devel@patchwork.kernel.org; Mon, 14 Nov 2016 11:25:51 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46484) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c6K4K-0004P3-IS for qemu-devel@nongnu.org; Mon, 14 Nov 2016 11:25:17 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1c6K4F-0001Vb-0l for qemu-devel@nongnu.org; Mon, 14 Nov 2016 11:25:16 -0500 Received: from mail-wm0-x242.google.com ([2a00:1450:400c:c09::242]:36190) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1c6K4E-0001Sh-Kj for qemu-devel@nongnu.org; Mon, 14 Nov 2016 11:25:10 -0500 Received: by mail-wm0-x242.google.com with SMTP id m203so16691914wma.3 for ; Mon, 14 Nov 2016 08:25:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=T7FJKNTGnW8/1GRCXTo2h3BR+F5ewjd8J261dpdC4fA=; b=B6uVtHqqsJ63WSRhC/D5XmvujXusNayHu5JL4UatkrFqrDiJWiGKZyFyBzZvFoB56U DhDqGTDaGZ61r7aPYkZHI1kF6CI33DGBqgXv6EmyzIS8Fpc2P6tirvM8tKyrfg4zSVVi vBnlgGge8n8JpdQGT382FZNtHXRyb0iz7OL+gnl68QzX8VZdNp0Lxd3jMdnxvk9IAKns KopfcWOU150FjdUACel6so6sAiSw6bDM6SmTepl1W3cTvRGvK+qNWFHJDYU8Ujje4J1j GoNRGnfQpptZBJv2rlwServXWLSagEAD1N9CBlLLDa3iqGwrVXXdAbghfYGpC+BQayci yV7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=T7FJKNTGnW8/1GRCXTo2h3BR+F5ewjd8J261dpdC4fA=; b=TqdyUzqEKvZLKKccLhQ1VsapCz7bHolOLGQUOxJheDJjntgt90C682YnSiCtRLaWJO 5DoPkIdFUZt1r/gUWKFaizxWCtHF+228WC0Ql8Emy4UHglSI8kWfyvYRzo8Z6zgKzLtx YERw18EYRBj5uK661z5Lpw5TWAeHS3Imu9yfYXirzpFTJ7dxgOmlVG9CVupwIjbrKUwg 6ztjbwO0M9kvjSobpfidI580Q5aXla/dCd7zigXTph5Z0b1A6wSg9ni6Ja4H++IcSHPc 2P9hDxIO8qI8Kn3woYSL5y0luOGtCBz7QaLRVDEWstiJJNqv68BjfpdOANes0vhHD4hc GGhw== X-Gm-Message-State: ABUngveaL7HfKZidK7zDg+U6q+rsqtoQP3yDhXx552bL+xYH0Qvu7YeQ+NYKNk0CWHxXpA== X-Received: by 10.194.158.100 with SMTP id wt4mr25415594wjb.148.1479140707773; Mon, 14 Nov 2016 08:25:07 -0800 (PST) Received: from localhost (host86-163-92-203.range86-163.btcentralplus.com. [86.163.92.203]) by smtp.gmail.com with ESMTPSA id vr9sm29525384wjc.35.2016.11.14.08.25.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 Nov 2016 08:25:06 -0800 (PST) Date: Mon, 14 Nov 2016 16:25:05 +0000 From: Stefan Hajnoczi To: Russell King - ARM Linux , David Woodhouse Message-ID: <20161114162505.GD26664@stefanha-x1.localdomain> References: <20161111210500.GE1041@n2100.armlinux.org.uk> <1478899423.3892.7.camel@infradead.org> <20161111224427.GG1041@n2100.armlinux.org.uk> <20161114092947.GB2031@work-vm> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20161114092947.GB2031@work-vm> User-Agent: Mutt/1.7.1 (2016-10-04) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a00:1450:400c:c09::242 Subject: Re: [Qemu-devel] TCP performance problems - GSO/TSO, MSS, 8139cp related X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: vyasevic@redhat.com, netdev@vger.kernel.org, jasowang@redhat.com, qemu-devel@nongnu.org, dgilbert@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP On Mon, Nov 14, 2016 at 09:29:48AM +0000, Dr. David Alan Gilbert wrote: > * Russell King - ARM Linux (linux@armlinux.org.uk) wrote: > > On Fri, Nov 11, 2016 at 09:23:43PM +0000, David Woodhouse wrote: > > > It's also *fairly* unlikely that the kernel in the guest has developed > > > a bug and isn't setting gso_size sanely. I'm more inclined to suspect > > > that qemu isn't properly emulating those bits. But at first glance at > > > the code, it looks like *that's* been there for the last decade too... > > > > I take issue with that, having looked at the qemu rtl8139 code: > > > > if ((txdw0 & CP_TX_LGSEN) && ip_protocol == IP_PROTO_TCP) > > { > > int large_send_mss = (txdw0 >> 16) & CP_TC_LGSEN_MSS_MASK; > > > > DPRINTF("+++ C+ mode offloaded task TSO MTU=%d IP data %d " > > "frame data %d specified MSS=%d\n", ETH_MTU, > > ip_data_len, saved_size - ETH_HLEN, large_send_mss); > > > > That's the only reference to "large_send_mss" there, other than that, > > the MSS value that gets stuck into the field by 8139cp.c is completely > > unused. Instead, qemu does this: > > > > eth_payload_data = saved_buffer + ETH_HLEN; > > eth_payload_len = saved_size - ETH_HLEN; > > > > ip = (ip_header*)eth_payload_data; > > > > hlen = IP_HEADER_LENGTH(ip); > > ip_data_len = be16_to_cpu(ip->ip_len) - hlen; > > > > tcp_header *p_tcp_hdr = (tcp_header*)(eth_payload_data + hlen); > > int tcp_hlen = TCP_HEADER_DATA_OFFSET(p_tcp_hdr); > > > > /* ETH_MTU = ip header len + tcp header len + payload */ > > int tcp_data_len = ip_data_len - tcp_hlen; > > int tcp_chunk_size = ETH_MTU - hlen - tcp_hlen; > > > > for (tcp_send_offset = 0; tcp_send_offset < tcp_data_len; tcp_send_offset += tcp_chunk_size) > > { > > > > It uses a fixed value of ETH_MTU to calculate the size of the TCP > > data chunks, and this is not surprisingly the well known: > > > > #define ETH_MTU 1500 > > > > Qemu seems to be buggy - it ignores the MSS value, and always tries to > > send 1500 byte frames. > > cc'ing in Stefan who last touched that code and Jason and Vlad who > know the net code. CCing Igor Kovalenko who implemented "fixed for TCP segmentation offloading - removed dependency on slirp.h" in 2006. I don't actually expect him to remember this from 10 years ago though :). Looking at the history the large_send_mss variable was never used for anything beyond the debug printf. The datasheet for this NIC is here: http://realtek.info/pdf/rtl8139cp.pdf. See 9.2.1 Transmit. Does this untested patch work for you? diff --git a/hw/net/rtl8139.c b/hw/net/rtl8139.c index f05e59c..a3f1af5 100644 --- a/hw/net/rtl8139.c +++ b/hw/net/rtl8139.c @@ -2167,9 +2167,13 @@ static int rtl8139_cplus_transmit_one(RTL8139State *s) goto skip_offload; } - /* ETH_MTU = ip header len + tcp header len + payload */ + /* MSS too small */ + if (tcp_hlen + hlen >= large_send_mss) { + goto skip_offload; + } + int tcp_data_len = ip_data_len - tcp_hlen; - int tcp_chunk_size = ETH_MTU - hlen - tcp_hlen; + int tcp_chunk_size = large_send_mss - hlen - tcp_hlen; DPRINTF("+++ C+ mode TSO IP data len %d TCP hlen %d TCP " "data len %d TCP chunk size %d\n", ip_data_len,