From patchwork Tue Nov 14 13:47:07 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 10057699 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 017CA6023A for ; Tue, 14 Nov 2017 13:58:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D889C297BA for ; Tue, 14 Nov 2017 13:58:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CD79D297C2; Tue, 14 Nov 2017 13:58:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 76929297BF for ; Tue, 14 Nov 2017 13:58:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755375AbdKNN5q (ORCPT ); Tue, 14 Nov 2017 08:57:46 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52306 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753507AbdKNN5p (ORCPT ); Tue, 14 Nov 2017 08:57:45 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 14 Nov 2017 15:47:40 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id vAEDlejd011430; Tue, 14 Nov 2017 15:47:40 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id vAEDleC0018763; Tue, 14 Nov 2017 15:47:40 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id vAEDleJJ018762; Tue, 14 Nov 2017 15:47:40 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, noaos@mellanox.com, majd@mellanox.com, dledford@redhat.com Subject: [PATCH rdma-core 1/2] verbs: Report the device's PCI write end paddding capability Date: Tue, 14 Nov 2017 15:47:07 +0200 Message-Id: <1510667228-18579-2-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1510667228-18579-1-git-send-email-yishaih@mellanox.com> References: <1510667228-18579-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Noa Osherovich There are PCIe root complex that are able to optimize their performance when incoming data is multiple full cache lines. Expose the device capability to report whether the device supports padding the ending of incoming packets to full cache line, such that the last upstream write generated by the incoming packet will be a full cache line. User should consider several factors before activating this feature: - In case of high CPU memory load (which may cause PCI backpressure in turn), if a large percent of the writes are partial cache line, this feature should be checked as an optional solution. - This feature might reduce performance if most packets are between one and two cache lines and PCIe throughput has reached its maximum capacity. E.g. 65B packet from the network port will lead to 128B write on PCIe, which may cause trafiic on PCIe to reach high throughput. Signed-off-by: Noa Osherovich Reviewed-by: Yishai Hadas --- libibverbs/examples/devinfo.c | 5 ++++- libibverbs/verbs.h | 1 + 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/libibverbs/examples/devinfo.c b/libibverbs/examples/devinfo.c index 169da2e..d02952e 100644 --- a/libibverbs/examples/devinfo.c +++ b/libibverbs/examples/devinfo.c @@ -331,10 +331,13 @@ static void print_odp_caps(const struct ibv_odp_caps *caps) static void print_device_cap_flags_ex(uint64_t device_cap_flags_ex) { uint64_t ex_flags = device_cap_flags_ex & 0xffffffff00000000ULL; - uint64_t unknown_flags = ~(IBV_DEVICE_RAW_SCATTER_FCS); + uint64_t unknown_flags = ~(IBV_DEVICE_RAW_SCATTER_FCS | + IBV_DEVICE_PCI_WRITE_END_PADDING); if (ex_flags & IBV_DEVICE_RAW_SCATTER_FCS) printf("\t\t\t\t\tRAW_SCATTER_FCS\n"); + if (ex_flags & IBV_DEVICE_PCI_WRITE_END_PADDING) + printf("\t\t\t\t\tPCI_WRITE_END_PADDING\n"); if (ex_flags & unknown_flags) printf("\t\t\t\t\tUnknown flags: 0x%" PRIX64 "\n", ex_flags & unknown_flags); diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h index 3e543cb..025e321 100644 --- a/libibverbs/verbs.h +++ b/libibverbs/verbs.h @@ -142,6 +142,7 @@ enum ibv_device_cap_flags { * enum range is limited to 4 bytes. */ #define IBV_DEVICE_RAW_SCATTER_FCS (1ULL << 34) +#define IBV_DEVICE_PCI_WRITE_END_PADDING (1ULL << 36) enum ibv_atomic_cap { IBV_ATOMIC_NONE,