From patchwork Mon Apr 30 09:14:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 10371199 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 2D28360234 for ; Mon, 30 Apr 2018 09:15:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1C8F4289F9 for ; Mon, 30 Apr 2018 09:15:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 11385289FB; Mon, 30 Apr 2018 09:15:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 733CF289F9 for ; Mon, 30 Apr 2018 09:15:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752936AbeD3JO5 (ORCPT ); Mon, 30 Apr 2018 05:14:57 -0400 Received: from mail-wm0-f65.google.com ([74.125.82.65]:40124 "EHLO mail-wm0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753166AbeD3JOy (ORCPT ); Mon, 30 Apr 2018 05:14:54 -0400 Received: by mail-wm0-f65.google.com with SMTP id j5so13102542wme.5 for ; Mon, 30 Apr 2018 02:14:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dev-mellanox-co-il.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=G1pze80XKIXKRvzSucA4OOvjwDEjmiALN4d0G0V+oTo=; b=dQfQZKtP7IRtDaL4unVKYR/E8QQPJP3RlvoLA5nN4Yvj4/848FPS9ODZFz6caKn03p PLnZmji+6Xnaf0IzxFWqfBVPb7jfD5on4eFecnSyD7zX0P9wWDPOPqrv/3uc9SktJrwJ zjoRGRIH0nL0GUzkDmrPp/UbrHrppfOi7qAtqMniMayC/WKWDwmpqVNDzcLQEV9BQaaB 0VvYn10r5xXYwp9/VjoLU+LDIn0hAI6d8nI9YLKkpICn9TqPgFVueB7q8H5hAIiPzn27 tTKylSwezFG/dBV9+5Okdzw7I7BKCP3JwSyV3IFTCpukLGBFC+MAnAC6TKPSCOnbJYji D5+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=G1pze80XKIXKRvzSucA4OOvjwDEjmiALN4d0G0V+oTo=; b=ungm5XKhAIOizIZO98l1Fadg3xpxMmPygAwHkeYyu518ZxaOqq9GsDRI7JE6QnrimF rY1GuFRnZMOkvwNGoRi9ofgz0Zbu+dkmQ0c+1lL3TqpwzANJR+akeTUBdQ00j7vLAsEY +l3VEyLixOOyjB0mkB7ZXptKpXlGVFnNJ3uaWYIXHmGnUET2rOobwde1cef0/IobSfoL +AzPb754HMdwuAovs3mH2GEaIoF2gW//+Pnm/93tCXgakPlO8YG3LZdFs1DZyNAGWa6P DIZdUccEdGjczdHIKym6hbxF3hWxLOij3ZnBumwh+dTSInlVvhUOkKTYb9njo3xyCkx/ OmmA== X-Gm-Message-State: ALQs6tB5XIXmzx8ASc5sDVVpobIWseErhW2FwW+FQvsCNkrU2oRAsvGI Jrv0gU9GSEKEBOntEK2We4ngew== X-Google-Smtp-Source: AB8JxZpQWJ6o6/kLh3wW3xchBpg5kXSjiuWkOKJfpFp29O2y1u0XpU/OCoArGANSNbgcUbShzy8rIw== X-Received: by 10.28.131.80 with SMTP id f77mr2391797wmd.1.1525079692962; Mon, 30 Apr 2018 02:14:52 -0700 (PDT) Received: from [10.8.1.82] ([193.47.165.251]) by smtp.googlemail.com with ESMTPSA id m134sm9125814wmg.4.2018.04.30.02.14.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 30 Apr 2018 02:14:52 -0700 (PDT) Subject: Re: [PATCH rdma-core 1/1] mlx5: Account for at least 1 medium bfreg in low_lat_uuars check To: Rohit Zambre Cc: RDMA mailing list , Eli Cohen , Yishai Hadas References: <1524506327-24417-1-git-send-email-rzambre@uci.edu> <2e6e4fb0-7b26-d584-78d2-a73ddd05fbc7@dev.mellanox.co.il> From: Yishai Hadas Message-ID: <751f8831-a7ff-6360-f437-917c04274843@dev.mellanox.co.il> Date: Mon, 30 Apr 2018 12:14:51 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 4/29/2018 8:34 PM, Rohit Zambre wrote: > On Sun, Apr 29, 2018 at 11:12 AM, Yishai Hadas > wrote: >> >> On 4/23/2018 8:58 PM, Rohit Zambre wrote: >>> >>> The calculation of need_uuar_lock rightly accounts for at least 1 medium >>> bfreg. >> >> >> The code should support even 0 medium bfregs comparing the suggested patch below. > > So when the user sets MLX5_TOTAL_UUARS=16 and > MLX5_NUM_LOW_LAT_UUARS=15, the first 15 QPs will map to the 15 low > latency bfregs. To which bfreg will the 16th QP map to? The 16th QP expects to get bfreg index 0 which is returned from the kernel once there are *no* low latency high class nor medium bfregs available. In this case DB is used in rdma-core instead of BF and no lock is needed. (see usage of bfs[bfi].buf_size) However, looking in the kernel code, it seems that there is some bug in 'first_med_bfreg()' which had to handle this non-default case when there are no medium bfregs. Below patch expects to fix that, can you please give it a try and confirm the fix ? this should come in parallel with the candidate patch that I have sent yesterday in rdma-core in need_uuar_lock(). --- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html --- a/drivers/infiniband/hw/mlx5/qp.c +++ b/drivers/infiniband/hw/mlx5/qp.c @@ -480,8 +480,12 @@ static int qp_has_rq(struct ib_qp_init_attr *attr) return 1; } -static int first_med_bfreg(void) +static int first_med_bfreg(struct mlx5_bfreg_info *bfregi) { + if (bfregi->total_num_bfregs - + (bfregi->num_low_latency_bfregs + bfregi->num_dyn_bfregs) == 1) + return -ENOMEM; + return 1; } @@ -537,10 +541,13 @@ static int alloc_high_class_bfreg(struct mlx5_ib_dev *dev, static int alloc_med_class_bfreg(struct mlx5_ib_dev *dev, struct mlx5_bfreg_info *bfregi) { - int minidx = first_med_bfreg(); + int minidx = first_med_bfreg(bfregi); int i; - for (i = first_med_bfreg(); i < first_hi_bfreg(dev, bfregi); i++) { + if (minidx < 0) + return minidx; + + for (i = minidx; i < first_hi_bfreg(dev, bfregi); i++) { if (bfregi->count[i] < bfregi->count[minidx]) minidx = i;