From patchwork Sun May 29 12:33:47 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 9139763 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id F0C8C60755 for ; Sun, 29 May 2016 12:34:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D6E3920202 for ; Sun, 29 May 2016 12:34:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CAA4A28183; Sun, 29 May 2016 12:34:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 584AE20202 for ; Sun, 29 May 2016 12:34:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932161AbcE2Mdw (ORCPT ); Sun, 29 May 2016 08:33:52 -0400 Received: from mail-wm0-f67.google.com ([74.125.82.67]:36650 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932154AbcE2Mdv (ORCPT ); Sun, 29 May 2016 08:33:51 -0400 Received: by mail-wm0-f67.google.com with SMTP id q62so14158827wmg.3 for ; Sun, 29 May 2016 05:33:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=7i7rHRdsVqpQIdHIszlWkbT6gvm41jFOhD36SMYuuYA=; b=WekRs5UOaMb+jhBhLpnoY61MA+2Ksp7375wcY+SIUmgjKfusZMVquVYRgB8kdGCwDc Yt0NG1Ix1owrcZsPRuJoK8wKCWcCfEiupsd5B/v/47VRtBC6da4q9bJc+uTob90VU9zj eDEud8OAj3zN8fPEFXA7A+iu31A4OwZ5WzK3z8zJvKyNgqYC7V/ROzanQLt3dTs/iHuw FKbNQFjJTI96BcNcf9tazmsFulr+mA7kEnDdfSQ57ucb+XizP8z3KQlxpmyC4bNqwjoS vPaY/8hpIPJZnLm/nbkmBT5ZhzuEMFlhY87Fj9dB9/1KJmwIPwcn3J/k5mo86ZlJYHQK qorQ== X-Gm-Message-State: ALyK8tIA8wGeZ/ImGF/hQT9iEU9ASp6BxIa/UtLB2UmakRoIwO3Px/HsTNV70RuyUKzFPg== X-Received: by 10.28.168.86 with SMTP id r83mr6838461wme.9.1464525230088; Sun, 29 May 2016 05:33:50 -0700 (PDT) Received: from [192.168.1.24] (bzq-82-81-101-184.red.bezeqint.net. [82.81.101.184]) by smtp.gmail.com with ESMTPSA id xt9sm29005974wjb.17.2016.05.29.05.33.48 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 29 May 2016 05:33:49 -0700 (PDT) Subject: Re: MLX5 iSER issue on 4.6? To: Max Gurtovoy , Robert LeBlanc , linux-rdma@vger.kernel.org, Matan Barak , "leon@leon.nu" References: <574A930E.6030507@grimberg.me> <574ADB25.8080903@mellanox.com> From: Sagi Grimberg Message-ID: <574AE1AB.9050501@grimberg.me> Date: Sun, 29 May 2016 15:33:47 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.8.0 MIME-Version: 1.0 In-Reply-To: <574ADB25.8080903@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP >>> I've been running iSER on 4.1/4.4 with a Connect-IB and ConnectX-3 and >>> everything works fine. When running the 4.6 kernel, the Connect-IB >>> card aren't able to login to iSER even though the ConnectX-3 card does >>> just fine. Hopefully, someone here has an idea of what the issue may >>> be. For this test, we are making 12 sessions to the same host (three >>> ports on each host, four sessions per port). Only four sessions become >>> established which correlate to the ConnectX-3 card. It doesn't seem to >>> matter which kernel the target is on, only the initiator. >>> >>> 02:00.0 Network controller: Mellanox Technologies MT27500 Family >>> [ConnectX-3] >>> 81:00.0 Infiniband controller: Mellanox Technologies MT27600 >>> [Connect-IB] >> >> Hi Robert, thanks for reporting. >> >> The difference in iSER in 4.6 is the arbitrary SG registration support >> which works with a capable device (IB_DEVICE_SG_GAPS_REG). The CIB does >> not support this feature but CX4 does, however, I wander if your CIB >> falsely reports it does support this feature. > > Sagi your'e right. > I repro this bug and saw that we alloc mr with: > mr_type = IB_MR_TYPE_SG_GAPS (means that IB_DEVICE_SG_GAPS_REG cap is on). > We also never call blk_queue_virt_boundary(sdev->request_queue, > ~MASK_4K); because of this false cap. Hey Max, Thanks for looking into this! >> >> Can you share the output of ibv_devinfo -v on the CIB device? I'm >> specifically interested in knowing what the max_mw is? it should >> be 0, if its not I suspect this is a FW bug. > > ibv_devinfo -v | grep max_mw > max_mw: 0 OK, this is weird, lets dive into the the mlx5 driver code: if (MLX5_CAP_GEN(mdev, imaicl)) { props->device_cap_flags |= IB_DEVICE_MEM_WINDOW | IB_DEVICE_MEM_WINDOW_TYPE_2B; props->max_mw = 1 << MLX5_CAP_GEN(mdev, log_max_mkey); /* We support 'Gappy' memory registration too */ props->device_cap_flags |= IB_DEVICE_SG_GAPS_REG; } We turn on SG_GAPS_REG in case the device supports memory windows (which allows upgrading local permissions to remote on a window). This is because in indirect registration, we effectively open a window on the local_dma_lkey. So the device seems to report it supports memory windows but it actually doesn't because the maximum windows you can allocate is 0 (which looks kind of silly to me). I'd say that the CIB FW needs to be fixed and not report imaicl altogether for CIB (instead of reporting it with 0 memory windows). Anyway, either this will be fixed, or the below patch should work around this (rather strange) behavior: --- -- Can you report if Mellanox plans to fix it in FW or should we work around in the driver? Thanks, Sagi. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index 6ad0489cb3c5..ae70fd1a6467 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -492,8 +492,9 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, props->device_cap_flags |= IB_DEVICE_MEM_WINDOW | IB_DEVICE_MEM_WINDOW_TYPE_2B; props->max_mw = 1 << MLX5_CAP_GEN(mdev, log_max_mkey); - /* We support 'Gappy' memory registration too */ - props->device_cap_flags |= IB_DEVICE_SG_GAPS_REG; + if (props->max_mw) + /* We support 'Gappy' memory registration too */ + props->device_cap_flags |= IB_DEVICE_SG_GAPS_REG; } props->device_cap_flags |= IB_DEVICE_MEM_MGT_EXTENSIONS; if (MLX5_CAP_GEN(mdev, sho)) {