From patchwork Thu Aug 11 12:29:13 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 9275215 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 9069060231 for ; Thu, 11 Aug 2016 12:30:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7FDCE2861C for ; Thu, 11 Aug 2016 12:30:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 740AD28626; Thu, 11 Aug 2016 12:30:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, T_DKIM_INVALID, T_TVD_MIME_EPI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D63682861C for ; Thu, 11 Aug 2016 12:30:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751725AbcHKMae (ORCPT ); Thu, 11 Aug 2016 08:30:34 -0400 Received: from mail-oi0-f52.google.com ([209.85.218.52]:34175 "EHLO mail-oi0-f52.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751266AbcHKMac (ORCPT ); Thu, 11 Aug 2016 08:30:32 -0400 Received: by mail-oi0-f52.google.com with SMTP id l203so2756022oib.1 for ; Thu, 11 Aug 2016 05:29:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=profitbricks-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=jdbNG5vu26exq/ngFVX0Xofo+2RTllR+aYLz7iqpFyI=; b=JmEljy0RITeRWtxEovpCzXY6996Daf1dSYbCxEANTBA/fNxqWHRDbC5BA56dwnk/sr 6uDSuvhg5FdGL79I+YRshombS8ZjgpDL/dRBmUJAmlfMqvW6mz6ACslBtjNAylIfSnyB KbAZuaZgmmc5KmMepcuro6Nkd60dbr2BK3O0hR3+qBTQ7A7/4UDUG+2IW5teg8iGl0q0 GpxNZ1mKVN/a9P/H/C3DNNjUh62r+1YCFYAdkwyeCjfbAcOGABv4BNzCFvDo4KIkfrsI YMwEX28nFd9ljWM549FruMkHcZDreKKJPwuT+CSK+zsWZlfWCwxiJzIJ+XaeiXCdfj9o 1AUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=jdbNG5vu26exq/ngFVX0Xofo+2RTllR+aYLz7iqpFyI=; b=e3LejVC6SX2tw6iwV+EyR3C8uFKSrz+azOG9k+kbBcA1dE+GtCDp46v/R5a/3022j6 uAc57Zk2FJYSRmRLaTCo/wyrvssRZxPWzJ2CmFpnQehpte1BaqYyCrKF7cpuYUoMJJqC 1NFpg+OupaoYeGpfz1CBVY+DnEuC9ZZqNvH5wzDsMNZHvcob+9UAHYYjDc69iPa+/DnO V69ZsLLaK65AYD/mSJuq5ch6WXVkZytuLpoBXz+gLS7wgw4Aq8bgSSETpn8U6NjGpXCf tjp8/A4xUfqJqUcPusm0v5cNzeeTEDfhX7bwfuy90s/JN9e3bwtO61yrs7jyq1JpIYt4 ycxg== X-Gm-Message-State: AEkoouv/w08+1JFGzOxjllAaztedDv1F9YUzPy7b2cRWaOgZOJzPogdwQJHGLXDNVNHZRGJraXmbjfzBEe1pYeLN X-Received: by 10.157.43.114 with SMTP id f47mr1787177otd.106.1470918574226; Thu, 11 Aug 2016 05:29:34 -0700 (PDT) MIME-Version: 1.0 Received: by 10.182.86.200 with HTTP; Thu, 11 Aug 2016 05:29:13 -0700 (PDT) In-Reply-To: <170f9d79-2351-d95f-9ed1-eddedc467d68@dev.mellanox.co.il> References: <170f9d79-2351-d95f-9ed1-eddedc467d68@dev.mellanox.co.il> From: Jinpu Wang Date: Thu, 11 Aug 2016 14:29:13 +0200 Message-ID: Subject: Re: [RFI] ucmatose: No effect to set service type for QoS To: Hal Rosenstock Cc: Sean Hefty , "linux-rdma@vger.kernel.org" Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Wed, Aug 10, 2016 at 8:52 PM, Hal Rosenstock wrote: > On 8/9/2016 12:26 PM, Jinpu Wang wrote: >> Hi Sean, >> >> I'm testing QoS support for IB. I notice ucmatose has equally >> performance when set different service type, but set SL in ib_send_bw >> works well (different SL show different performance base on opensm >> settings) >> >> I capature packats using ibdump, it shows in in LRH the service level >> fields are all 0 when running traffic with ucmatose. >> >> When running ib_send_bw, it carries the right service level I set. >> >> Seems in rdma_set_service_type, it sets to tos to id_priv->tos, and >> lter set to path_rec->qos_class or traffic_class but not to sl >> directly, what's the consideration here? >> code snip: >> switch (cma_family(id_priv)) { >> case AF_INET: >> path_rec->qos_class = cpu_to_be16((u16) id_priv->tos); >> comp_mask |= IB_SA_PATH_REC_QOS_CLASS; >> break; >> case AF_INET6: >> sin6 = (struct sockaddr_in6 *) cma_src_addr(id_priv); >> path_rec->traffic_class = (u8) >> (be32_to_cpu(sin6->sin6_flowinfo) >> 20); >> comp_mask |= IB_SA_PATH_REC_TRAFFIC_CLASS; >> break; >> case AF_IB: >> sib = (struct sockaddr_ib *) cma_src_addr(id_priv); >> path_rec->traffic_class = (u8) >> (be32_to_cpu(sib->sib_flowinfo) >> 20); >> >> >> Does it make sense we also set sl here, or service type for ucmatose >> is totally different with SL for ib_send_bw? > > I think this is an OpenSM configuration issue. QoS policy needs to be > setup to return the proper SL to use for QoS class or TClass in the > PathRecord response. > > -- Hal > Thanks Hal, Configure extra QoS policy seems quite complex. Do you think patch attached make sense? From beec3a2fc25d37d4f735c55231f4e0fbe3f180ac Mon Sep 17 00:00:00 2001 From: Jack Wang Date: Wed, 10 Aug 2016 10:50:53 +0200 Subject: [PATCH] cma: export function to set service level We want this for isolating network traffic with storage traffic. So extend cma to allow us to do it for QoS. Signed-off-by: Jack Wang --- drivers/infiniband/core/cma.c | 14 +++++++++++++- include/rdma/rdma_cm.h | 13 +++++++++++++ 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c index 66e8516..c464aa7 100644 --- a/drivers/infiniband/core/cma.c +++ b/drivers/infiniband/core/cma.c @@ -225,6 +225,7 @@ struct rdma_id_private { u32 options; u8 srq; u8 tos; + u8 sl; u8 reuseaddr; u8 afonly; enum ib_gid_type gid_type; @@ -2752,6 +2753,16 @@ static void cma_listen_on_all(struct rdma_id_private *id_priv) mutex_unlock(&lock); } +void rdma_set_service_level(struct rdma_cm_id *id, u8 sl) +{ + struct rdma_id_private *id_priv; + + id_priv = container_of(id, struct rdma_id_private, id); + id_priv->sl = sl; +} +EXPORT_SYMBOL(rdma_set_service_level); + + void rdma_set_service_type(struct rdma_cm_id *id, int tos) { struct rdma_id_private *id_priv; @@ -2838,9 +2849,10 @@ static int cma_query_ib_route(struct rdma_id_private *id_priv, int timeout_ms, path_rec->pkey = cpu_to_be16(ib_addr_get_pkey(&addr->dev_addr)); path_rec->numb_path = 1; path_rec->reversible = 1; + path_rec->sl = id_priv->sl; path_rec->service_id = rdma_get_service_id(&id_priv->id, cma_dst_addr(id_priv)); - comp_mask |= IB_SA_PATH_REC_PKEY | IB_SA_PATH_REC_NUMB_PATH | + comp_mask |= IB_SA_PATH_REC_PKEY | IB_SA_PATH_REC_NUMB_PATH | IB_SA_PATH_REC_SL | IB_SA_PATH_REC_REVERSIBLE | IB_SA_PATH_REC_SERVICE_ID; switch (cma_family(id_priv)) { diff --git a/include/rdma/rdma_cm.h b/include/rdma/rdma_cm.h index b34ee4e..df7030e 100644 --- a/include/rdma/rdma_cm.h +++ b/include/rdma/rdma_cm.h @@ -374,6 +374,19 @@ int rdma_join_multicast(struct rdma_cm_id *id, struct sockaddr *addr, void rdma_leave_multicast(struct rdma_cm_id *id, struct sockaddr *addr); /** + * rdma_set_service_level - Set the level of service associated with a + * connection identifier. + * @id: Communication identifier to associated with service type. + * @sl: service level. + * + * The service level should be specified before + * performing route resolution, as existing communication on the + * connection identifier may be unaffected. The level of service + * requested may not be supported by the network to all destinations. + */ +void rdma_set_service_level(struct rdma_cm_id *id, u8 sl); + +/** * rdma_set_service_type - Set the type of service associated with a * connection identifier. * @id: Communication identifier to associated with service type. -- 2.7.4