From patchwork Wed Apr 17 19:10:29 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 2455961 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork1.kernel.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by patchwork1.kernel.org (Postfix) with ESMTP id 7A6E43FC64 for ; Wed, 17 Apr 2013 19:10:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754719Ab3DQTKc (ORCPT ); Wed, 17 Apr 2013 15:10:32 -0400 Received: from mail-ob0-f175.google.com ([209.85.214.175]:56445 "EHLO mail-ob0-f175.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757341Ab3DQTKa (ORCPT ); Wed, 17 Apr 2013 15:10:30 -0400 Received: by mail-ob0-f175.google.com with SMTP id va7so1745759obc.6 for ; Wed, 17 Apr 2013 12:10:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=5BHVPkeULOxaah2auu/ubWfcfBx+kqjf71407hH9IuA=; b=CcGObLSIl4pPPkCBOdUk2pjRfOIYTnayD9kngeYBQxiR9VhN7TWvFHNWQDmhz4EuO1 AuglDmQ8Np9WT4BGsny+Esydza0TQkAVIvM7Y0BpNa1D3URNVm4sHQEoV5v0qrGytEzc AODzWnnJu9EcyT2upMtaOVD+Umr3EmjfDs4E5AqJFjmQYDJaPTRc0LAsLTp9UfuN8ntf EjU7JhIwKRlXyxHZa0+DSoklzLqe+p2pmQyx7GYfhnAoq6OuWyAictgaClTdqk5i5GeB NTuCcsduI2nikzP+onr3qpb5t6oN2c6RpyiXh8XDcMZHymGTy64JJR7vb8HfYs4Lr2Uk Recg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:x-gm-message-state; bh=5BHVPkeULOxaah2auu/ubWfcfBx+kqjf71407hH9IuA=; b=HotkANl/G/CzdgaMrpj/Y+sfi0HZp9Nsab6/enFWV9ou/cl1EOfE0YHL+8TVT9h00f 8Hjz4DljRt95b+F4v/WXZuEat7xwSGP31vwvodPCuUZo2bulBiL5hE/knitJGJn/4jPi XtYqWeWN+iNRGQTau+EdNmi1O6q5H6CNWj4nt+n+NoBOW1aaaXNcv0smDEF5p8jbWi4k Ucwy4llJYNYwYzY2NKI8imrMrYsJDTmOpA+w7hu8agKtEAnrg66JIhdQ0yfKs7rVaKPK wYXuaJe63VgJUBjc2xyyYn+z4wsoEWv/4XMEm3OP89ZhUFzGIWmOvlKYdDeIch/+JeBk Dawg== MIME-Version: 1.0 X-Received: by 10.182.165.105 with SMTP id yx9mr3326495obb.86.1366225829710; Wed, 17 Apr 2013 12:10:29 -0700 (PDT) Received: by 10.182.61.81 with HTTP; Wed, 17 Apr 2013 12:10:29 -0700 (PDT) In-Reply-To: References: Date: Wed, 17 Apr 2013 12:10:29 -0700 Message-ID: Subject: Re: IPoIB header prefetch for fragmented SKB From: Eric Dumazet To: Markus Stockhausen Cc: Roland Dreier , "linux-rdma@vger.kernel.org" X-Gm-Message-State: ALoCoQlhEmm1OslIt5mzr4OvHyMUCn/E6cQp1IDnIabrBYioSc9KhKLuiITzvcksRQ3t2tzAfF1tjNRwzM3LiwlzJNxfz4bO/7xcJkfuBGwrKISvZ4yIbAkHPTD+PaPtAc/VqJXzK2c18DCmlzTHbdRJhB+F5wGSyqcty0yeQiLUHhRP6aySNMtcN9fL/mngT2al+CWca1sfk4SuEv5GpFw1mnFPkGNs2A== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Please try : * first buf size is IPOIB_UD_HEAD_SIZE On Wed, Apr 17, 2013 at 11:38 AM, Markus Stockhausen wrote: >> >> >> >>That's probably because of a cache line miss. >> >>The thing I don't really understand is that normally, the first cache >>line (64 bytes) contains both the Ethernet header and IPv4 header. >> >>So what does this adapter in this respect ? >> >>I guess you should try to use IPOIB_UD_HEAD_SIZE=64 to use the whole >>cache line. >> >>Many drivers use prefetch() to make sure cpu starts to bring this >>cache line into cache as soon as possible. >> >>A single prefetch() call at the right place might help a lot. > > Hello, > > @Eric: Thanks for the tip. > > In the 4K MAX MTU IPoIB driver path ipoib_ib_handle_rx_wc() will > produce an empty skb linear part with the whole data placed into > the first fragment. napi_gro_receive() finally pulls the IP > header out of the fragment into the linear part. > > As far as I understand the pull out of the fragment should come > without additional cost when one calls a prefetch "long" before > the skb_pull(). > > I'm willing to check this out but I'm unsure if the IP header > is aligned to a cache line of 64 bytes. As a simple guess I > would implement the prefetch here: > > static void ipoib_ib_handle_rx_wc(); > ... > skb_pull (skb, IB_GRH_BYTES); > > skb->protocol = ((struct ipoib_header *) skb->data)->proto; > skb_reset_mac_header(skb); > skb_pull(skb, IPOIB_ENCAP_LEN); > + > + if (ipoib_ud_need_sg(priv->max_ib_mtu)) > + prefetch(); > ... > > Can you give me a hint what address one should put into the call? > > Thanks in advance. > > Markus > > --- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c index 2cfa76f..9bdff21 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c @@ -112,6 +112,8 @@ static void ipoib_ud_skb_put_frags(struct ipoib_dev_priv *priv, if (ipoib_ud_need_sg(priv->max_ib_mtu)) { skb_frag_t *frag = &skb_shinfo(skb)->frags[0]; unsigned int size; + + prefetch(page_address(frag->page.p)); /* * There is only two buffers needed for max_payload = 4K,