From patchwork Tue Oct 18 08:04:00 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 9381533 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A08E7607D0 for ; Tue, 18 Oct 2016 08:04:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 946EF294B1 for ; Tue, 18 Oct 2016 08:04:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 88498294DD; Tue, 18 Oct 2016 08:04:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.4 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 785B6294B1 for ; Tue, 18 Oct 2016 08:04:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758963AbcJRIEM (ORCPT ); Tue, 18 Oct 2016 04:04:12 -0400 Received: from mail-lf0-f45.google.com ([209.85.215.45]:35556 "EHLO mail-lf0-f45.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752395AbcJRIEI (ORCPT ); Tue, 18 Oct 2016 04:04:08 -0400 Received: by mail-lf0-f45.google.com with SMTP id l131so14296572lfl.2; Tue, 18 Oct 2016 01:04:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=bxhM00D+viTq2Ts5qnYznBIiyYEgu0y2Nos1+sV/ozA=; b=HtQNPJ3dTAbohX+ih/HIWrWL1yH2JWz5WuI0vpx1yGFmhU6z+cZiJsWV5kfdjLwvs2 hAmnYthNnwBAL60mXR7p3wGs1pI1y3zdlssYEd6YbWurg64kktKdi/N1OpBc7NrrUdhO xFISxPO33q0Qb03JEeA5feBpxRSr4VCnq6KGGyt0g+NSDkHrqLRtUBKEjDOpG1M9YgTu Cv9KwM8rw2k+dmu0LP+UWrvY1P8Zs51kTOmNPXdR+2OUQWRABuG8agEO7IfkTwQdYzck DVtukYBNu0ljM77EPJ1I40b+L4kDsJfFrRg3GwSauyhaX/BGDeW0UkIbkQ5TIBWpHS/O uJvQ== X-Gm-Message-State: AA6/9RnGXuF+qs3sV8P0jge5/o8wy2aDDxZLKOsEBTpUQB858P9QqT1I+85zyH1vDX72Xw== X-Received: by 10.25.207.131 with SMTP id f125mr16374564lfg.181.1476777843410; Tue, 18 Oct 2016 01:04:03 -0700 (PDT) Received: from [192.168.1.177] (bzq-82-81-101-184.red.bezeqint.net. [82.81.101.184]) by smtp.gmail.com with ESMTPSA id d135sm380597lfg.7.2016.10.18.01.04.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Oct 2016 01:04:02 -0700 (PDT) Subject: Re: RQ overflow seen running isert traffic To: Steve Wise , 'Potnuri Bharat Teja' References: <20160927070157.GA13140@chelsio.com> <672d8b05-5537-d45a-ba3f-cdd5f054a4ab@grimberg.me> <20161017111655.GA21245@chelsio.com> <021001d228a4$6cd6a6c0$4683f440$@opengridcomputing.com> Cc: target-devel@vger.kernel.org, nab@linux-iscsi.org, linux-rdma@vger.kernel.org From: Sagi Grimberg Message-ID: Date: Tue, 18 Oct 2016 11:04:00 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 MIME-Version: 1.0 In-Reply-To: <021001d228a4$6cd6a6c0$4683f440$@opengridcomputing.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hey Steve and Baharat, > Hey Sagi, I'm looking at isert_create_qp() and it appears to not be correctly > sizing the SQ: > ... > #define ISERT_QP_MAX_REQ_DTOS (ISCSI_DEF_XMIT_CMDS_MAX + \ > ISERT_MAX_TX_MISC_PDUS + \ > ISERT_MAX_RX_MISC_PDUS) > ... > attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS + 1; > attr.cap.max_recv_wr = ISERT_QP_MAX_RECV_DTOS + 1; > ... > > I think above snipit assumes a DTO consumes exactly one WR/WQE in the SQ. But > the DTO can be broken into multiple WRs to handle REG_MRs, multiple WRITE or > READ WRs due to limits on local sge depths target sge depths, etc. Yes? Or am > I all wet? Or perhaps isert doesn't require the SQ to be the max possible > because it flow controls the DTO submissions? I think you are correct. On my test devices, I didn't see that multiple WRs has had any effect becuase: 1. My test devices usually give next power of 2 (256) 2. workloads that involved multiple rdma operations never stressed the system enough to get the queues full. Now, in iWARP for non-immediate writes we'll need more than a single wr per IO (I think the SQ size is expanded with the new rdma RW API which implicitly increases with attr.cap.max_rdma_ctxs). But I do agree that we need to take into account that each IO needs at least 2 WRs (one for rdma and one for send). So a temp bandage would be: --- -- But we do need to track the SQ overflow and queue a retransmit work when we don't have enough available SQ slots.. Thoughts? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h index fc791efe3a10..81afb95aeea9 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.h +++ b/drivers/infiniband/ulp/isert/ib_isert.h @@ -54,8 +54,14 @@ #define ISERT_MIN_POSTED_RX (ISCSI_DEF_XMIT_CMDS_MAX >> 2) -#define ISERT_QP_MAX_REQ_DTOS (ISCSI_DEF_XMIT_CMDS_MAX + \ - ISERT_MAX_TX_MISC_PDUS + \ +/* + * Max QP send work requests consist of: + * - RDMA + SEND for each iscsi IO + * - iscsi misc TX pdus + * - iscsi misc RX response pdus + */ +#define ISERT_QP_MAX_REQ_DTOS ((ISCSI_DEF_XMIT_CMDS_MAX * 2 ) + \ + ISERT_MAX_TX_MISC_PDUS + \ ISERT_MAX_RX_MISC_PDUS) #define ISER_RX_PAD_SIZE (ISCSI_DEF_MAX_RECV_SEG_LEN + 4096 - \