From patchwork Sun Jan 21 11:25:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Rosenbaum X-Patchwork-Id: 10176893 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 440EE601E7 for ; Sun, 21 Jan 2018 11:25:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2EE2922376 for ; Sun, 21 Jan 2018 11:25:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2361F26220; Sun, 21 Jan 2018 11:25:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7896A22376 for ; Sun, 21 Jan 2018 11:25:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750982AbeAULZw (ORCPT ); Sun, 21 Jan 2018 06:25:52 -0500 Received: from mail-io0-f195.google.com ([209.85.223.195]:40606 "EHLO mail-io0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750961AbeAULZv (ORCPT ); Sun, 21 Jan 2018 06:25:51 -0500 Received: by mail-io0-f195.google.com with SMTP id t22so6580284ioa.7 for ; Sun, 21 Jan 2018 03:25:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc; bh=UqkQPplgVUZo5INlhzvP+2WrisDsas2gsqCCG1M/+rI=; b=S4RmUX1p32fDY56+Q7lh/+TZ5xhHmtXiHobtzcJcWGHIpdTE9rq98+SmzDDSet0hZv YWSU0GlAw5mPb6ZEc164zenYQpGfWAinEA1BQTTpjTCwvPtXZS1jRogkXHfjUHE9V+fN HMFd5RAknBV9xK0996O39h+j7Ln6nqJ0tFnYllvOdZ9BZ8n9N9XUxAMSMRrN7U1MxGpy i01FyWx2rDq8KWAt+I8VJUoi0QS1ZO/skr6s7MnktZVCRgDlrdP4JUzRzLd7stT0wgae rYekjdhgyn6870HQVQJi10eblWeDsmQoSQjxAqXZpm7FCvqE72/ed1qF1J6pdxET3lDF L9fA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=UqkQPplgVUZo5INlhzvP+2WrisDsas2gsqCCG1M/+rI=; b=YeLk1X94GrCp0KojXqy20laAsz260+n0y9BeyMts16L2AD9oPM0lrpZUFMqJxkTxk8 ry29MNbBPJaQOs9fknxnMVh9K8OX9IJMBsddXqH6YvtIF5QLC6bY3OG6gbpGA6pOP+Uw ieOr84VV3qHj4bDEPuRYGl8hW/LhH1J/f0Mp/Sp+rvTdUtnHj2N/AjqFml9/pE5sokXk I3PcLqqRoDGkydKI2KzsNGvLimEnE8qcBnAogRnMWq27mRRkuFo/ZA1MCBIafQ61r1JE MGSunA+gebwswNHdU4p/ZiWszCajih936YDLRShTH3HAuCNa9wlqgDGaqUvh0qSWxAcW haZw== X-Gm-Message-State: AKwxyteGwp0ovpGexabSXkc3BuA+fuYnhpeNLhGH32CuhrSqN8jf77tp WfPaUwaPZFQmJ+VeRe3uZ+uRxUTxx2oAcQmH9XuBduH+ X-Google-Smtp-Source: AH8x224K9UewPigI640SAZoSvgSTWN0kGgR/55KSAbUzQ6WBDOJO43RAOOX/C2HFXpT/ORbkzKM60v5cm8t6h5eFiHU= X-Received: by 10.107.175.103 with SMTP id y100mr4804601ioe.45.1516533950318; Sun, 21 Jan 2018 03:25:50 -0800 (PST) MIME-Version: 1.0 Received: by 10.2.177.158 with HTTP; Sun, 21 Jan 2018 03:25:49 -0800 (PST) From: Alex Rosenbaum Date: Sun, 21 Jan 2018 13:25:49 +0200 Message-ID: Subject: [PATCH RFC] Extend verbs packet pacing QP rate limiting with burst info To: linux-rdma@vger.kernel.org Cc: Yishai Hadas , Bodong Wang , Leon Romanovsky , Majd Dibbiny Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Allow user space verbs application finer grain control of QP send rate limit by setting a max_burst_sz [bytes] and typical_pkt_sz [bytes]. Expose new ibv_modify_qp_rate_limit() user space verbs API with dedicated attribute struct. Fail call if requested configuration is not supported on QP or device. The standard kernel modify qp command will be used to pass the new values to be modified by the driver. Signed-off-by: Alex Rosenbaum --- libibverbs/man/ibv_modify_qp_rate_limit.3 | 65 +++++++++++++++++++++++++++++++ libibverbs/verbs.h | 26 +++++++++++++ 2 files changed, 91 insertions(+) create mode 100644 libibverbs/man/ibv_modify_qp_rate_limit.3 int (*dealloc_td)(struct ibv_td *td); @@ -2314,6 +2322,24 @@ int ibv_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr, int attr_mask); /** + * ibv_modify_qp_rate_limit - Modify a queue pair rate limit values + * @qp - QP object to modify + * @attr - Attributes to configure the rate limiting values of the QP + */ +static inline +int ibv_modify_qp_rate_limit(struct ibv_qp *qp, + struct ibv_qp_rate_limit_attr *attr) +{ + struct verbs_context *vctx; + + vctx = verbs_get_ctx_op(qp->context, modify_qp_rate_limit); + if (!vctx) + return ENOSYS; + + return vctx->modify_qp_rate_limit(qp, attr); +} + +/** * ibv_query_qp - Returns the attribute list and current values for the * specified QP. * @qp: The QP to query. diff --git a/libibverbs/man/ibv_modify_qp_rate_limit.3 b/libibverbs/man/ibv_modify_qp_rate_limit.3 new file mode 100644 index 0000000..232aaaf --- /dev/null +++ b/libibverbs/man/ibv_modify_qp_rate_limit.3 @@ -0,0 +1,65 @@ +.\" -*- nroff -*- +.\" Licensed under the OpenIB.org BSD license (FreeBSD Variant) - See COPYING.md +.\" +.TH IBV_MODIFY_QP_RATE_LIMIT 3 2018-01-09 libibverbs "Libibverbs Programmer's Manual" +.SH "NAME" +ibv_modify_qp_rate_limit \- modify the send rate limits attributes of a queue pair (QP) +.SH "SYNOPSIS" +.nf +.B #include +.sp +.BI "int ibv_modify_qp_rate_limit(struct ibv_qp " "*qp" ", struct ibv_qp_rate_limit_attr " "*attr"); +.fi +.SH "DESCRIPTION" +.B ibv_modify_qp_rate_limit() +modifies the send rate limiting packet pacing attributes of QP +.I qp +with the attributes in +.I attr\fR. +The argument \fIattr\fR is an ibv_qp_rate_limit_attr struct, as defined in . +.PP +The +.I rate_limit +defines the MAX send rate this QP will send at as long as the link in not blocked and there are work requestes in send queue. +.PP +Finer control for shaping the rate limit of a QP is achieved by defining the +.I max_burst_sz\fR, +single burst max bytes size and the +.I typical_pkt_sz\fR, +typical packet bytes size. These allow the device to adjust the inter-burst gap delay required to correctly shape the scheduling of sends to the wire in order to reach for requested application requirements. +.PP +Setting a value of 0 for +.I max_burst_sz +or +.I typical_pkt_sz +will use the devices defaults. +.I typical_pkt_sz +will default to the port's MTU value. +.PP +.nf +struct ibv_qp_rate_limit_attr { +.in +8 +uint32_t rate_limit; /* kbps */ +uint32_t max_burst_sz; +uint16_t typical_pkt_sz; +.in -8 +}; +.fi +.PP +.SH "RETURN VALUE" +.B ibv_modify_qp_rate_limit() +returns 0 on success, or the value of errno on failure (which indicates the failure reason). +.SH "ERRORS" +.SS EINVAL +Invalid arguments. +.SS ENOSYS +Function is not implemented for this device. +.PP +.SH "SEE ALSO" +.BR ibv_create_qp (3), +.BR ibv_destroy_qp (3), +.BR ibv_modify_qp (3), +.BR ibv_query_qp (3) +.SH "AUTHORS" +.TP +Alex Rosenbaum diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h index 997597a..cb73576 100644 --- a/libibverbs/verbs.h +++ b/libibverbs/verbs.h @@ -966,6 +966,13 @@ struct ibv_qp_attr { uint32_t rate_limit; }; +struct ibv_qp_rate_limit_attr { + uint32_t rate_limit; /* in kbps */ + uint32_t max_burst_sz; /* total burst size in bytes */ + uint16_t typical_pkt_sz; /* typical packet bytes send size */ + uint32_t comp_mask; +}; + enum ibv_wr_opcode { IBV_WR_RDMA_WRITE, IBV_WR_RDMA_WRITE_WITH_IMM, @@ -1650,6 +1657,7 @@ struct ibv_values_ex { struct verbs_context { /* "grows up" - new fields go here */ + int (*modify_qp_rate_limit)(struct ibv_qp *qp, struct ibv_qp_rate_limit_attr *attr); struct ibv_pd *(*alloc_parent_domain)(struct ibv_context *context, struct ibv_parent_domain_init_attr *attr);