From patchwork Fri Sep 4 22:41:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758591 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A364C925 for ; Fri, 4 Sep 2020 22:42:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7574720FC3 for ; Fri, 4 Sep 2020 22:42:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="T43n2FfQ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726208AbgIDWmP (ORCPT ); Fri, 4 Sep 2020 18:42:15 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:44044 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728116AbgIDWmJ (ORCPT ); Fri, 4 Sep 2020 18:42:09 -0400 Received: from hkpgpgate101.nvidia.com (Not Verified[10.18.92.9]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:06 +0800 Received: from HKMAIL101.nvidia.com ([10.18.16.10]) by hkpgpgate101.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:06 -0700 X-PGP-Universal: processed; by hkpgpgate101.nvidia.com on Fri, 04 Sep 2020 15:42:06 -0700 Received: from HKMAIL104.nvidia.com (10.18.16.13) by HKMAIL101.nvidia.com (10.18.16.10) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:04 +0000 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.107) by HKMAIL104.nvidia.com (10.18.16.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:04 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JdtIpmvjtnsY3ZhYz2fu+KeeRLkvEfVl0e3OAyxmYqKO2QjgU1e2E6g0+EpaHYJn1RYNXo7O/YG1pQ39IXV9qqf0ea8HciCx/0hFCORTP/NzgBlM3oJDhFjcQK4mZ51qqpgscRPktbG4rhIB1jzVUXMAjonNxNV1yRdgRXwJ9IrUQYC3YsSSTsU8+vMH+tpxDVEa5soK9pf7dLIZEdHxKAtYdFNRciM5n+wqvGArfm3PJEoZymQcy8NJOWJnr/MyEypEqBvJ2rxFnw+Nb9LYAva8UG7VOUghAd1GxX19c4Tjn3gaF/IhXxq+ah28rvSI2Dz10YwIebxVNmfojcWvLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D/UbOXpXA302TPxzBEWcqkxR/LSBKG8cgcPd///qEBA=; b=hqC0vM1aBRffOUFT+4bPDYhdTKFb2mJLo5km+DmbOFItGylbRxMDW5oGaro/FTFILc6kMFCi/O1zNgegT+mWqj1RHO64uVF0IBaOtQWPYCLpztKzOl836sDBDSJsnmJJeqfWdkMGoxap8n/vWBQ6zRuvaoFb77QmC8FG4+Bv4dW71UiC0iGLwjyanG/fhnWlzC9KbtzivAEo+dfVtUWKYjbNhmGCZS3OyR+GA57GUlUb34g1JnF5wd98fJriNGS5GdgvVd0A0M30XKp0Qe1H3CG1dzfHmLtavUGPpRxzQpA/FL6LLOsJ9MMkzSOqUeJzhscCROmwM8HT6PWdxFJvzw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2806.namprd12.prod.outlook.com (2603:10b6:a03:70::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3326.22; Fri, 4 Sep 2020 22:42:02 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:02 +0000 From: Jason Gunthorpe To: Doug Ledford , CC: Leon Romanovsky , Shiraz Saleem Subject: [PATCH v2 01/17] RDMA/umem: Fix ib_umem_find_best_pgsz() for mappings that cross a page boundary Date: Fri, 4 Sep 2020 19:41:42 -0300 Message-ID: <1-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR13CA0026.namprd13.prod.outlook.com (2603:10b6:208:160::39) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by MN2PR13CA0026.namprd13.prod.outlook.com (2603:10b6:208:160::39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3370.9 via Frontend Transport; Fri, 4 Sep 2020 22:42:00 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP8-001vBz-Pn; Fri, 04 Sep 2020 19:41:58 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7fd7ce16-c789-4b4b-00eb-08d85123bc99 X-MS-TrafficTypeDiagnostic: BYAPR12MB2806: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2512; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: L6ZSEofuS03SLsv6hDouG6/g0oul3+ADjtbKPVbnLKG2daAvAYJQGvyIQH3N/Tx2bTmRM405Lv9odFFdXWfUJs3L278FnGGWcoiPDwUes4knHjwJVEIICgD+ZmihvrtJVSjuYN/WIFT35ZtY2YtauCvz9980fYbJuu1EtRjX4sp+7N1+BKJYUa4Ax/fdr8f1Ybf1Z8tPdkwNxpNz5mX/x6ak9OqZdZpQWjSEozZAX6X+ApxJhStBvuuFpU1RutCKmyhazOZLKczhCDyuMMgwMpSdWKwK48qko/lUJJ7ZxoAvwkbTF13oQqJQ9WvFenIH2e6jnP+a3qEIWHFTGEmCOtzReWTaNN09k5skzljLOvkShG/8PQEB/e7vix2r9L20 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(366004)(376002)(39860400002)(396003)(136003)(346002)(66556008)(26005)(54906003)(66476007)(6666004)(8676002)(186003)(316002)(83380400001)(426003)(2616005)(9786002)(478600001)(36756003)(9746002)(86362001)(8936002)(5660300002)(2906002)(4326008)(66946007)(4216001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: LdvpLq+FxSXWY8OLjrLqMNHsff1SpMmE8zXnNCowa8bnBVzuwS6IKzzGyr4RGOFZafPfMbH32Q3gX69xRTZewv46hVDLMenH81cFtQqVLf1HMXAE1at77l3k0CJ5GWzq57cmVbxtkLId5O+AK6wNIA7XV9rWZTC7xCZSjpRY3r5laRix57okrugckbIE0jhqbFiEsW9C2gnJZ5hPgs3n4cHIW4ALD5Rj06ArlM6GCbEMmu5XI8tjQI5ecZkIqDoUdeq0gJcX9+OBz2pO0rQZTpOTzASogg5qLQHQVsTefGJJqlB5EC6/VflAv6++scq6GZ7AqPzEb0pQ8+P5/SaALmmabQobensjduShlMhTwc50iDKjdD08mWY+Q3CPNa4E83/ICFhH9QG9tYSUrYRnSisAVDNyvdoOV27aaVsy7Dew2WOL5Hg2I39DGS3k9PnoS2j0fENF1opdB5OmWWpEogApQ9ZkDWVz1QXkInYAB28v6rac2J+2VnrqgHY1vrJXgZkJPjTM78WkQ3dVX3N4Ij/KHymLli8kqAt9I5GMoMRjpqJQLuKAx0SbukveZShUX0dgUeUm1dA2Xlxm0O7YCsna9fdYrrqJMA6ejp7Nz1pOd8B7VgT8W+QceZ4oJ8E4UuPGUmMHZnmos4TqeEDYwQ== X-MS-Exchange-CrossTenant-Network-Message-Id: 7fd7ce16-c789-4b4b-00eb-08d85123bc99 X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:01.0839 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: coCQD9+GWaHAQjcrccH3tVgBrJz9Uh39fVsc1hYkVhsI7fepi19NBgCiFNw5GqaM X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2806 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259326; bh=xm2DZjBst7h+iENmbRBi3dU+4re0mmT2hZqJzLUSw3E=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:CC: Subject:Date:Message-ID:In-Reply-To:References: Content-Transfer-Encoding:Content-Type:X-ClientProxiedBy: MIME-Version:X-MS-Exchange-MessageSentRepresentingType: X-Originating-IP:X-MS-PublicTrafficType: X-MS-Office365-Filtering-Correlation-Id:X-MS-TrafficTypeDiagnostic: X-MS-Exchange-Transport-Forked:X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers:X-MS-Exchange-SenderADCheck: X-Microsoft-Antispam:X-Microsoft-Antispam-Message-Info: X-Forefront-Antispam-Report:X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=T43n2FfQt1Xzaur5LO7VMJCkhhDCMbtToFdQqzjap+/JZeR2zsn/VVKI/zAeMR/om NcicEWmpX4OT1XChlcvqIHBGwFDNxwy/GgdF6hcuQgCv8YVOQIuS5hnI45V4ONrN6E h8uSDPk7dnsunMTxpp7PLuGqy6l2TibHk067IZPpDJ3yghwuCXP4iW3zBSMy92JdyC XKGr6RUjBnnE9crYhC3kn9rreltsou4DdDRqEB6zoS01lSTce+3QqJ6jz2bVxUMj/f 0nDpgEJkSq4PJ/mY+uG3RaQQByAgKGFEbNnPhu/q64LzX7wF8E+LQhjFha9/TJvEqX 7tPJpp0Plf7Mw== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org It is possible for a single SGL to span an aligned boundary, eg if the SGL is 61440 -> 90112 Then the length is 28672, which currently limits the block size to 32k. With a 32k page size the two covering blocks will be: 32768->65536 and 65536->98304 However, the correct answer is a 128K block size which will span the whole 28672 bytes in a single block. Instead of limiting based on length figure out which high IOVA bits don't change between the start and end addresses. That is the highest useful page size. Fixes: 4a35339958f1 ("RDMA/umem: Add API to find best driver supported page size in an MR") Signed-off-by: Jason Gunthorpe Reviewed-by: Leon Romanovsky Reviewed-by: Shiraz Saleem --- drivers/infiniband/core/umem.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index 831bff8d52e547..09539dd764ec05 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -156,8 +156,13 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, return 0; va = virt; - /* max page size not to exceed MR length */ - mask = roundup_pow_of_two(umem->length); + /* The best result is the smallest page size that results in the minimum + * number of required pages. Compute the largest page size that could + * work based on VA address bits that don't change. + */ + mask = pgsz_bitmap & + GENMASK(BITS_PER_LONG - 1, + bits_per((umem->length - 1 + virt) ^ virt)); /* offset into first SGL */ pgoff = umem->address & ~PAGE_MASK; From patchwork Fri Sep 4 22:41:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758599 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 96A4B925 for ; Fri, 4 Sep 2020 22:42:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7420F2083B for ; Fri, 4 Sep 2020 22:42:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="BREdMAx6" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726317AbgIDWmW (ORCPT ); Fri, 4 Sep 2020 18:42:22 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:48504 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726360AbgIDWmS (ORCPT ); Fri, 4 Sep 2020 18:42:18 -0400 Received: from hkpgpgate101.nvidia.com (Not Verified[10.18.92.100]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:08 +0800 Received: from HKMAIL102.nvidia.com ([10.18.16.11]) by hkpgpgate101.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:08 -0700 X-PGP-Universal: processed; by hkpgpgate101.nvidia.com on Fri, 04 Sep 2020 15:42:08 -0700 Received: from HKMAIL104.nvidia.com (10.18.16.13) by HKMAIL102.nvidia.com (10.18.16.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:03 +0000 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.107) by HKMAIL104.nvidia.com (10.18.16.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:03 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=U6hj4MJTnPYL5Mny0X2V7o0yljU0TqPHpSaIX4E9FHan2iPjYeZt3MuZfqIoqC6V+8w5Vht2vvDNT08GGNLyTGuYW670itiQfOVWr79lhf4LzPY9Yun8305p8gDUdF2YUVGgvrITuSVOFykVHmDtWDn1g6Q42M4jPf+cmK3fI0txcjwvmfPSyjTClCqXxBsrhfu/hg7Vor01qtCCFK5vCQCeSUgAxpKF6m74sJj5WOPYmBZtiz8eF58zXa4XT9MmRnL1+xubOxCttKG7S8Fla91GfXOkemL36z/u0zCmus5+NEqNUHDn3f4zFFkcU47W8kjOSkVNOQAsB2E6ch9NAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FutVtboNYTk/EFZt5ishUH41st+GLCvrWILRlcQNCIY=; b=bXLZEko0BdcL2yDSe6LxmZnUCoBYvHy1txmKYnATfyle1c91RiAI1vbbBwfhIHr0a9nz32GnS30UY033VMuE94SRzfy/jqFdNO9NcP3q5s/uN9C7JeYbuQl1NWi2e5zwGyPn44s/jCWMlD633HBxO7gBiX4S0zZZXybyu5Ue6bYkBQCd3Ix9jE5LVBnFwhpSlOZMf08U2P4RHSd6IOxEfWQLFzn8ZPnlsQRPHdu/84ddYsIet46/tr3hcLgCBBVFsDZHfTyVlbYM1RJ6q/IeK+Qh2ftqSyoH+ddal3Oez9/4R8d7sM55Nczir05rkFDIfAACIU01GtaA27lEtFgdaw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2806.namprd12.prod.outlook.com (2603:10b6:a03:70::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3326.22; Fri, 4 Sep 2020 22:42:01 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:01 +0000 From: Jason Gunthorpe To: Doug Ledford , CC: Leon Romanovsky , Shiraz Saleem Subject: [PATCH v2 02/17] RDMA/umem: Prevent small pages from being returned by ib_umem_find_best_pgsz() Date: Fri, 4 Sep 2020 19:41:43 -0300 Message-ID: <2-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR02CA0018.namprd02.prod.outlook.com (2603:10b6:207:3c::31) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by BL0PR02CA0018.namprd02.prod.outlook.com (2603:10b6:207:3c::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.16 via Frontend Transport; Fri, 4 Sep 2020 22:42:00 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP8-001vC4-Ql; Fri, 04 Sep 2020 19:41:58 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a4186296-c9f0-4578-0412-08d85123bca9 X-MS-TrafficTypeDiagnostic: BYAPR12MB2806: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:660; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: J4tgf/+gH910lfcDlboM2r7du5FzIWbpQsVocuqt+kgjc+ZILLWUodTQw6U7qMSnZrEhT0Q4cn74THl2HqcCQDwVdvOTWF0hJ0fmVdzN74Nh9nKqYwepNnqVdng23dyQ+kZ0m9sf0J61Tjo1wQXZprukjgog1u2zUfDMCu8DDS9+ab/bFagqXRKR8ueEUpyrvh+YG+VAvaT8J5TSNpvW1jlhcRXSMpriRknO/+lHZqI5yRDbWIOz+KmGLpMmQCxHnpCu+kQUZIJalaju4/wOdnZnlx2Do50fDtMcCKD/RYYN8LyQlu3JvgdmBPfAT/1F1Wdqcg7TeoVWdDkUPrB7aqpj5BL2gAuYJpZoT6JLXeHPHFp+pjpd5QM7knd1V49b X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(366004)(376002)(39860400002)(396003)(136003)(346002)(66556008)(26005)(54906003)(66476007)(6666004)(8676002)(186003)(316002)(83380400001)(426003)(2616005)(9786002)(478600001)(36756003)(9746002)(86362001)(8936002)(5660300002)(2906002)(4326008)(66946007)(4216001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 4Ovax45qFz4y49iszlqFn3fMOZaGLPFDIKiHojBWPgqyh+YvC0r+N8WGBvYi85F8KgBXEiFOgr5afMmIerYo+t5Gs6Z8cDpt4TFe2tVRNv4tpE+/KAOPiFFjebnPerGJ+WPTizJioZRP4FaiEPJI1Qv8AHn4D6Rp5CRQqERYVXDVAxspIAcy+Q9A+omtNuA07Xq9e86Kcmf9sDv66Z4hTG+7d+Y0DWbGzmY2nKaKNzOaIf+y7J1shG0B6/hls/Sk5RDKvNE49PxnprOyFGkCNTmC+95qvxOrJDicpC3LWcVBjJKESvCz5vJzOFMK1WDyB6qQN/qTn0V+Sd84oT7w2kche9QNXcboC6Ji7Eql4e8/Ns1VHPqTH1QZ4s+3ZNOXn6oLqQ78pUNGdGstwdhBtCr+JnOxSdsiSzalOnxwfXpbdxz3TB70td0x41z81Sp3UFv3hG4InyGiSkWVbW/a1b51m7L0VqwVfGPVlIdNu0l4lFXCtIeQqfyT9g+XjERi01KdqkLcc0J+NBqiAVSYTkKIlutp2kpcepBBoVweddJAKt2TtB0/xeXCYtlaIOP1fdMihAtSOyLoOXxGUqeBZyy4LdKiaF4h9AoOWCiWm7QJLt2ZR/vzEMtxYBahIQlQefl4w6AG0WVvTO3h8HnPMA== X-MS-Exchange-CrossTenant-Network-Message-Id: a4186296-c9f0-4578-0412-08d85123bca9 X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:01.0060 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cgDq651sBUjy/Qc7PbDmvTWtzWLJTZfhYGbbh7Q6xEA0sXTfLsWQ1SyCKwzmUuBH X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2806 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259328; bh=M5RRAXGa5FZ+Wn+kD8WKoopuG8Vz/6TxGxpInSvl6XQ=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:CC: Subject:Date:Message-ID:In-Reply-To:References: Content-Transfer-Encoding:Content-Type:X-ClientProxiedBy: MIME-Version:X-MS-Exchange-MessageSentRepresentingType: X-Originating-IP:X-MS-PublicTrafficType: X-MS-Office365-Filtering-Correlation-Id:X-MS-TrafficTypeDiagnostic: X-MS-Exchange-Transport-Forked:X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers:X-MS-Exchange-SenderADCheck: X-Microsoft-Antispam:X-Microsoft-Antispam-Message-Info: X-Forefront-Antispam-Report:X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=BREdMAx6rJGxVAuEQdiHtmYEEJg69C3I6yrNYdWkZwqXoMwEPPkIe1QXxfehwxsKq P85xvK1xRlHHAhETvIGK/KgPZip+2v+4e+GzrGMXKezoUKdKkBydk1lINUX06wZE69 JKiR+RdoLqHjxk01rXgzOlJWHWGYXLwAWhuZjvsVFpeu27DdFpyQ+xOci/OVJHYcUa B7NFkXqj6nVMGOtKfSAKjQXwnnEMtjld2Q+wgbVBP4iNNF61wSFBTHt1IbwHMNwtlG PmefaIdxhknrDI01NkeW6lmGw0GrDE3RFJgI+YlCBYDj0FxgKBjhYvZ+D+8oxG4vN2 TXKgTBr2L+7lA== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org rdma_for_each_block() makes assumptions about how the SGL is constructed that don't work if the block size is below the page size used to to build the SGL. The rules for umem SGL construction require that the SG's all be PAGE_SIZE aligned and we don't encode the actual byte offset of the VA range inside the SGL using offset and length. So rdma_for_each_block() has no idea where the actual starting/ending point is to compute the first/last block boundary if the starting address should be within a SGL. Fixing the SGL construction turns out to be really hard, and will be the subject of other patches. For now block smaller pages. Fixes: 4a35339958f1 ("RDMA/umem: Add API to find best driver supported page size in an MR") Signed-off-by: Jason Gunthorpe Reviewed-by: Leon Romanovsky Reviewed-by: Shiraz Saleem --- drivers/infiniband/core/umem.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index 09539dd764ec05..1d0599997d0fb5 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -151,6 +151,12 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, dma_addr_t mask; int i; + /* rdma_for_each_block() has a bug if the page size is smaller than the + * page size used to build the umem. For now prevent smaller page sizes + * from being returned. + */ + pgsz_bitmap &= GENMASK(BITS_PER_LONG - 1, PAGE_SHIFT); + /* At minimum, drivers must support PAGE_SIZE or smaller */ if (WARN_ON(!(pgsz_bitmap & GENMASK(PAGE_SHIFT, 0)))) return 0; From patchwork Fri Sep 4 22:41:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758589 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5232D925 for ; Fri, 4 Sep 2020 22:42:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 26F812083B for ; Fri, 4 Sep 2020 22:42:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="kfwdm7bZ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728141AbgIDWmJ (ORCPT ); Fri, 4 Sep 2020 18:42:09 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:65190 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728098AbgIDWmH (ORCPT ); Fri, 4 Sep 2020 18:42:07 -0400 Received: from hkpgpgate101.nvidia.com (Not Verified[10.18.92.9]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:04 +0800 Received: from HKMAIL104.nvidia.com ([10.18.16.13]) by hkpgpgate101.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:04 -0700 X-PGP-Universal: processed; by hkpgpgate101.nvidia.com on Fri, 04 Sep 2020 15:42:04 -0700 Received: from HKMAIL104.nvidia.com (10.18.16.13) by HKMAIL104.nvidia.com (10.18.16.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:04 +0000 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.107) by HKMAIL104.nvidia.com (10.18.16.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:04 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=APzj8n3k9m/betCMe1k5K2jlMik3FSPZDyPm0eB6hSdVAUUNjAOfjCXF3QhsDQWco7fDZqDELdSJ9VT9I8LxnWWlIdxSuh/UIcnD2yaQ+INVJ/K14mew2oG3Eg+YOBvuSci8cvI4fWeNvCyBl2Of+K3c+qaEz4UvP/t4rfn1T87mynamnU5Z7uTU/CRQF6ScBMr9hRIgWGvBFZLUKt1h3+xQtSB5901FYuI7iRmipkqeFxYx2cQEt/Su5xxqk7BSoH/qOA5iy/vHeJnE8Jc1Nf3NtKk6ympIneXyptRjAJbwXoudGLKYh9XtNZYNeDFgusT65guwpvhErVqE6FV0WA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=53zrpZr+2Bp1LtMts7gstfGPV5t4KE1YTqVw+rH3p3E=; b=ShQdki5z0M9QMF3mxYnxxC0DC0+LNbOvg4V7GOp5NKxN3JFWtHV9o82UhI72jhVGFJuu5/8VXoS4zfE7y8qM3emopDk0kHMXHLh77gPFy7PxdmgJ4J/PnV+pSBHkubRu1sO5u8K9vEZ0LkUu1jVR7N5KMl7YIyCNRRYwk/kfJLYFtnAnO+Bk67yv3/COlUWkD9sFCx9W3cYDmvFOBoy9TpmjyOnY0DIXXsAo9t/tXM/uAhorau5VEd0Gd45XvrokAZOCjWBI0B41so6V+kclOMhE+tIR9W2sORLPAcj4qQnO/t756WueLZqo3XwqtpB69EISpwFZ/v4fT4LKPBfusg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2806.namprd12.prod.outlook.com (2603:10b6:a03:70::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3326.22; Fri, 4 Sep 2020 22:42:02 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:02 +0000 From: Jason Gunthorpe To: Doug Ledford , Subject: [PATCH v2 03/17] RDMA/umem: Use simpler logic for ib_umem_find_best_pgsz() Date: Fri, 4 Sep 2020 19:41:44 -0300 Message-ID: <3-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR17CA0011.namprd17.prod.outlook.com (2603:10b6:208:15e::24) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by MN2PR17CA0011.namprd17.prod.outlook.com (2603:10b6:208:15e::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.16 via Frontend Transport; Fri, 4 Sep 2020 22:42:00 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP8-001vC7-Rs; Fri, 04 Sep 2020 19:41:58 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 37e65c17-f280-44ba-a18b-08d85123bc97 X-MS-TrafficTypeDiagnostic: BYAPR12MB2806: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5797; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6zAfP7FpjyMHmqJhVNJiOfzeRdyy1yWDX25ZzFbZ4R0C9sOiDula/W0+yF5mUle0nJeH0tO0Vji/yq0S3EMsWwhPhnlV8v0+OlmE/EIOTOUiWbXAZiZqwwej7YpXndym1Ygym7WJnpVbGsVt2izakyfC6K8U+T7NwHR3JdMwSeHOedvyBXWmdx5dk4E7bucpGh7A2Wf+w7RbZrOzv14A5R4CYwG423kRpe9dJPd5UkXTi85ukNz+XVjgR9I7BzcxGc/B7VAsZCNbEQU1aCFtOrPF/THybz/YLh2SNjJVDOvfLJcQgunl9DFtUxJv2PED7QqcAsmqFBem3ClTuSblQD4xQikM3mwcQiyXf//zeFDkefPGFMC4gHd7zriEp7af X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(366004)(376002)(39860400002)(396003)(136003)(346002)(66556008)(26005)(66476007)(6666004)(8676002)(186003)(316002)(83380400001)(426003)(2616005)(9786002)(478600001)(36756003)(9746002)(86362001)(8936002)(5660300002)(2906002)(66946007)(4216001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: /Yx6V/I1RX6vX/RZQwyUejzblCjHskDiyP49AAp17NxXQTzV8rEJJF6oggNdWFCpChPUV14f2Mo6dGkkiPlT0d0lomvRaWSm3Q3gabUh+yw/BuYCu5Q8E3zKjSY5roZ4vXwMiydRe5DJDsG9eM3C3ajR6RiEI6+uBu7WkgWqdgppjVzXTH5ZEdehbXTgLAxv2ovS8Kuc8/KqXPAmiH1AR2Fzy6NufGeCmcRewNtpPrPUl0YR6xbH1dTKvUHZa4z2/HbUTBx+mD8YxV1UWHEJP4fICeTs1BrokzIw462FbXJAfqA9uPvs9LtZNhQxQEytBdJQ9qMloRzw4k8vdVqnZ5r1VpVt6+qJuslTxHE0FPcVK9dBQT+jaNEvg6WQ7MR6L5a4daVMEFik1RoAXf/VejVrvo0LccwIEBP6kk7LzZB7zcHmMCx70ZKWMM+4QDn6EsYeHFK5NkqnieZRvVhsdV6nA8zYcKRlMMmAnXStkEuHCszlNS1Cmuhct1maNaIASkDvtLfTlp/F7wro+mj7Nyf4LnKqiMZVT4K5Xs9jj3MXUMsRJxyYCTJRKzfYV89u8nPS8sd3OBw4wY7YdHL4mzC4jQDefd1qAJYHOqBv5nq0kL7h6GmW5EVnDAeh92DKKSRWwzA2u9277PYvSj47pw== X-MS-Exchange-CrossTenant-Network-Message-Id: 37e65c17-f280-44ba-a18b-08d85123bc97 X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:01.0759 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: aJEPai8R+jgBs5E8LSWlxaLTsRlB6mglMMMlPVURDKeM2F4aijjPXU4RntS8nOHL X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2806 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259324; bh=1/TQv4WtqZ9tyx1b3Au+IIG4Xx6F+j09BTSiG0txYxw=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:Subject: Date:Message-ID:In-Reply-To:References:Content-Transfer-Encoding: Content-Type:X-ClientProxiedBy:MIME-Version: X-MS-Exchange-MessageSentRepresentingType:X-Originating-IP: X-MS-PublicTrafficType:X-MS-Office365-Filtering-Correlation-Id: X-MS-TrafficTypeDiagnostic:X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers:X-MS-Exchange-SenderADCheck: X-Microsoft-Antispam:X-Microsoft-Antispam-Message-Info: X-Forefront-Antispam-Report:X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=kfwdm7bZhAlgc+FYJrVFj2MyfQVo0ebbiPA5DJt2U7KvqvUHbP4+LKg6JCEFP5uxK F1Jwti4+IkDWjlIwpH5o7Jdo2v2rBpTeUTaAzNl7mzfiZXEcx4cn1HixjMjA6Pfa8S IDUs6TdSA0CqVm3dthzoVN598j5eoxnMM59hRBxMTKk/A9JBo/UjaJYUKhgGn9kHYN qfgNqsMJc/Tq0N39gjMiymgNGj8kEnflK19oXkNa0IXsEATmj0cJC3ZdlQ6Y1zlp/9 8ySlcAEbRZND72WyNz+2axtIlLgbHjmyXniymIYz7sCOvMGqFdX7sxRqxkU14+vSj8 GWBl925lEfZlQ== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The calculation in rdma_find_pg_bit() is fairly complicated, and the function is never called anywhere else. Inline a simpler version into ib_umem_find_best_pgsz() Signed-off-by: Jason Gunthorpe --- drivers/infiniband/core/umem.c | 11 ++++++++--- include/rdma/ib_verbs.h | 24 ------------------------ 2 files changed, 8 insertions(+), 27 deletions(-) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index 1d0599997d0fb5..fb7630e7aac3a7 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -39,6 +39,7 @@ #include #include #include +#include #include #include "uverbs.h" @@ -146,7 +147,6 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, unsigned long virt) { struct scatterlist *sg; - unsigned int best_pg_bit; unsigned long va, pgoff; dma_addr_t mask; int i; @@ -186,9 +186,14 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, mask |= va; pgoff = 0; } - best_pg_bit = rdma_find_pg_bit(mask, pgsz_bitmap); - return BIT_ULL(best_pg_bit); + /* The mask accumulates 1's in each position where the VA and physical + * address differ, thus the length of trailing 0 is the largest page + * size that can pass the VA through to the physical. + */ + if (mask) + pgsz_bitmap &= GENMASK(count_trailing_zeros(mask), 0); + return rounddown_pow_of_two(pgsz_bitmap); } EXPORT_SYMBOL(ib_umem_find_best_pgsz); diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index c868609a4ffaed..5dcbbb77cadb4f 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -3340,30 +3340,6 @@ static inline bool rdma_cap_read_inv(struct ib_device *dev, u32 port_num) return rdma_protocol_iwarp(dev, port_num); } -/** - * rdma_find_pg_bit - Find page bit given address and HW supported page sizes - * - * @addr: address - * @pgsz_bitmap: bitmap of HW supported page sizes - */ -static inline unsigned int rdma_find_pg_bit(unsigned long addr, - unsigned long pgsz_bitmap) -{ - unsigned long align; - unsigned long pgsz; - - align = addr & -addr; - - /* Find page bit such that addr is aligned to the highest supported - * HW page size - */ - pgsz = pgsz_bitmap & ~(-align << 1); - if (!pgsz) - return __ffs(pgsz_bitmap); - - return __fls(pgsz); -} - /** * rdma_core_cap_opa_port - Return whether the RDMA Port is OPA or not. * @device: Device From patchwork Fri Sep 4 22:41:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758623 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 356C2925 for ; Fri, 4 Sep 2020 22:42:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 13BA82100A for ; Fri, 4 Sep 2020 22:42:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="XP7svUz+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726265AbgIDWmm (ORCPT ); Fri, 4 Sep 2020 18:42:42 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:16388 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728116AbgIDWmh (ORCPT ); Fri, 4 Sep 2020 18:42:37 -0400 Received: from hkpgpgate101.nvidia.com (Not Verified[10.18.92.9]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:22 +0800 Received: from HKMAIL104.nvidia.com ([10.18.16.13]) by hkpgpgate101.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:22 -0700 X-PGP-Universal: processed; by hkpgpgate101.nvidia.com on Fri, 04 Sep 2020 15:42:22 -0700 Received: from HKMAIL104.nvidia.com (10.18.16.13) by HKMAIL104.nvidia.com (10.18.16.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:07 +0000 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.107) by HKMAIL104.nvidia.com (10.18.16.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:07 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FAcqFWLTkhkvzoJN6l1B0nmaUSmXqxHWuIh6hFHScIyGHD6E7vv8ajOrqfSQXYpXrLarTIAVvbs2qUeh+m8BE558/4EIO0bcnNeoaBw6joTmmLXzFDe7YbL6EtnY+3BJmtnoGs+N3GiHX7qtlugFkuTumAFtu7qRsrRWu6n7P8WBoQjzAZJNeFGo14lXBqNXBdRXA7tbim6dIfJchObnAonQKyukAt8lNx3PHCa35KMN8nHZ7dxDdLWHqNRaf+KIF44BoYphFsV8c67ncasdJj8VmqqIcioio8bgUm3ePzNKEDH4CkaecbIdBwTzvWoOH2F5hnzqO3zWJWEml0kdMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1qHp5GQyzKUZ7hOtgCWa/j+tQQo0rzLw72UdoReXYXE=; b=Oq91cMaJvMlUBTH7Dnc+pZrBgNeSj8N0CMl8GW+qeU1Zx6lwh3OFxJDj7zubzqR9xF32jxfpY/Wz5kDy5PIe3+D2kmG2O3fT77B3xXnJL6k0SO+XUVgUmcgiverMWRRjZBv+vBiB2Wh5aQ3liKNVj/LjxIew6zzDoo5l9MeS9ual+FFc1kjjvMz3pch+WQ0FQLOQ3KYYHWOAuHdxcvf46X1Qu9FBkSa63XDEmBN9EDiWUI/0kV85ICSBnOTJDc6SHt72pdvbvjcR7iJaDzM6XgzKWw3KQwrbuDB3MBPxppkipOzn3GAKPJ7essqYi5kchsAEqJAu50/2YrwK04gODw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: broadcom.com; dkim=none (message not signed) header.d=none;broadcom.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2806.namprd12.prod.outlook.com (2603:10b6:a03:70::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3326.22; Fri, 4 Sep 2020 22:42:05 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:05 +0000 From: Jason Gunthorpe To: Devesh Sharma , Doug Ledford , Faisal Latif , Gal Pressman , "Wei Hu(Xavier)" , , Weihang Li , "Naresh Kumar PBS" , Lijun Ou , Selvin Xavier , Yossi Leybovich , Somnath Kotur , "Sriharsha Basavapatna" CC: Miguel Ojeda , Shiraz Saleem Subject: [PATCH v2 04/17] RDMA/umem: Add rdma_umem_for_each_dma_block() Date: Fri, 4 Sep 2020 19:41:45 -0300 Message-ID: <4-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR02CA0010.namprd02.prod.outlook.com (2603:10b6:207:3c::23) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by BL0PR02CA0010.namprd02.prod.outlook.com (2603:10b6:207:3c::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3326.20 via Frontend Transport; Fri, 4 Sep 2020 22:42:02 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP8-001vCC-TC; Fri, 04 Sep 2020 19:41:58 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c7c940c8-5fce-4e8f-cf13-08d85123be1c X-MS-TrafficTypeDiagnostic: BYAPR12MB2806: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1060; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: VpEU0nbLbnuV6rq6l/cj3u1sI3gFymnqent61qPz5oAlaN2w9Vqm0p46wWFaHjCtpFoW9zZ2YLxOzdKKyXwORFvSW8py1VP2mA6ftwbfQq4E6ix4ZxQweANGSXYp1yxnRq4Mnml+LJz7kb//avndcD9lVu6YDrx0LotbfQ5ehkgG48HKnzSfn6Y0sGnKXSIsDkORIBYDUwfCflDxwirv4qHbyWkI9cP0NiQRnQQ/aH4HXEY6JiQ6laM20MsotcYMoPODOL+VX6WOMDSjtljWjY2Lc5idzM3EDl2lL4/voLUMlz0/PIladAy3ZHfnKyWCmR7MGMjGERs4illcDnZWomP+VJDOZ7kbG0MK2LPXz/vTImCd31ZcPwF1T1FxYRS8 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(366004)(376002)(39860400002)(396003)(136003)(346002)(66556008)(26005)(54906003)(66476007)(6666004)(110136005)(8676002)(186003)(316002)(83380400001)(426003)(7416002)(2616005)(9786002)(478600001)(36756003)(9746002)(86362001)(8936002)(5660300002)(2906002)(4326008)(66946007)(4216001)(921003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: /4D/DyWEB4mCgsnOmQ+OvNaKXO6hH2kk6VQcFgQFaUQ7zSjZVJkx/VDJXN6cx7s4i77WNo+/Zm426B2uw5zSRiCEpLRpi/WvxXFbwfcGe+vBkoNxSQKW4NKVHJnVzNjfaSkLbRvjrdDN0nsRxNU3KVBK1gMMYwSYa229aYPN3pVmp4ieqGJTgErFUh6XVqdvKncq/UpU1BfiO/o1Qyea3FPHCuXXenpV3xlAIjdolqHrUm4d/GIWnaBEbc5WJ26n3kcxIgn7/KqYsKRBR41HeKacZGTox1+5pYya+2FaP3M5hsfScPg9cabMKMlmiHo+U5zILyAs4yqYOlYsBVrqdYJ/8SeThyIXdgUjViKl7r1BaYOkGmBcB4IXz+SedYOibjTmchoZnjIiHCHQW8+tLPO6ooC8RUyPKXxP5zYo4d4JmVZ8WZjuZV7/Vtf//F/sY5cxBW17amCwSjikws4Y+t4uZyQDpBxBpPQBsk/NBAiS5TZGUevEGmYSKbtJodvgEojVpw7aEp6+LIBOh7prYomrBEpxtE6Ucoi/0PAcNxVNzJv55u+LCVarvBUURINckOJWFvBL+nPjv5OhweKpLRgEowkr6F2kavCVppZ92iSHmTBF0kMqfB2rdX017g3jpd4QGxASI8l3yJu297S+6A== X-MS-Exchange-CrossTenant-Network-Message-Id: c7c940c8-5fce-4e8f-cf13-08d85123be1c X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:03.2506 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: SuEp1PtTE+St8ku9nNJHR8yxbFpsyvpqt0za9lbvYeROjgD0SCBp9nVJBxgekNW6 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2806 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259342; bh=u3PpiuRl9//9NIPj3pc0GTKxf0hzmaQi40cC0OCPo/o=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:CC: Subject:Date:Message-ID:In-Reply-To:References: Content-Transfer-Encoding:Content-Type:X-ClientProxiedBy: MIME-Version:X-MS-Exchange-MessageSentRepresentingType: X-Originating-IP:X-MS-PublicTrafficType: X-MS-Office365-Filtering-Correlation-Id:X-MS-TrafficTypeDiagnostic: X-Microsoft-Antispam-PRVS:X-MS-Oob-TLC-OOBClassifiers: X-MS-Exchange-SenderADCheck:X-Microsoft-Antispam: X-Microsoft-Antispam-Message-Info:X-Forefront-Antispam-Report: X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=XP7svUz+kfBn8jbVoh3CuMYg9RjNvVkMHoePWJuFYP4cO3iHxKNrDX4bM+qwNSxXC PrPyGYYwMBT+0EgsHwpAydORLKmxe8qeMnzMR1LigUrmSmvkk6cnOjgq6b7BASh4wI LwBj3H4KBctdeDDJv6mkS8OT84/C3XnfoS/GaIODzfUtac2jZvJ+tb+OA+5Ogf4Ukt IBxDgzq76nKnfuxI7TE1mKJCnhvkvIOL9nxiCX1fab6kPWKdKVCvmEjyE9ZaJqvtCp xa1ydBQQp4Nm6ZGth8imZi4RqFfcieWFh7X6ESKkpekvevZKPlgSlG7+S1rj9mzkxk 1lceG7oUxGO5A== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org This helper does the same as rdma_for_each_block(), except it works on a umem. This simplifies most of the call sites. Signed-off-by: Jason Gunthorpe Acked-by: Miguel Ojeda Acked-by: Shiraz Saleem --- .clang-format | 1 + drivers/infiniband/hw/bnxt_re/ib_verbs.c | 2 +- drivers/infiniband/hw/efa/efa_verbs.c | 3 +-- drivers/infiniband/hw/hns/hns_roce_alloc.c | 3 +-- drivers/infiniband/hw/i40iw/i40iw_verbs.c | 3 +-- include/rdma/ib_umem.h | 20 ++++++++++++++++++++ 6 files changed, 25 insertions(+), 7 deletions(-) diff --git a/.clang-format b/.clang-format index a0a96088c74f49..311ef2c61a1bdf 100644 --- a/.clang-format +++ b/.clang-format @@ -415,6 +415,7 @@ ForEachMacros: - 'rbtree_postorder_for_each_entry_safe' - 'rdma_for_each_block' - 'rdma_for_each_port' + - 'rdma_umem_for_each_dma_block' - 'resource_list_for_each_entry' - 'resource_list_for_each_entry_safe' - 'rhl_for_each_entry_rcu' diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index 5ee272d27aaade..9e26e651730cb3 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -3783,7 +3783,7 @@ static int fill_umem_pbl_tbl(struct ib_umem *umem, u64 *pbl_tbl_orig, u64 page_size = BIT_ULL(page_shift); struct ib_block_iter biter; - rdma_for_each_block(umem->sg_head.sgl, &biter, umem->nmap, page_size) + rdma_umem_for_each_dma_block(umem, &biter, page_size) *pbl_tbl++ = rdma_block_iter_dma_address(&biter); return pbl_tbl - pbl_tbl_orig; diff --git a/drivers/infiniband/hw/efa/efa_verbs.c b/drivers/infiniband/hw/efa/efa_verbs.c index de9a22f0fcc218..d85c63a5021a70 100644 --- a/drivers/infiniband/hw/efa/efa_verbs.c +++ b/drivers/infiniband/hw/efa/efa_verbs.c @@ -1142,8 +1142,7 @@ static int umem_to_page_list(struct efa_dev *dev, ibdev_dbg(&dev->ibdev, "hp_cnt[%u], pages_in_hp[%u]\n", hp_cnt, pages_in_hp); - rdma_for_each_block(umem->sg_head.sgl, &biter, umem->nmap, - BIT(hp_shift)) + rdma_umem_for_each_dma_block(umem, &biter, BIT(hp_shift)) page_list[hp_idx++] = rdma_block_iter_dma_address(&biter); return 0; diff --git a/drivers/infiniband/hw/hns/hns_roce_alloc.c b/drivers/infiniband/hw/hns/hns_roce_alloc.c index a522cb2d29eabc..a6b23dec1adcf6 100644 --- a/drivers/infiniband/hw/hns/hns_roce_alloc.c +++ b/drivers/infiniband/hw/hns/hns_roce_alloc.c @@ -268,8 +268,7 @@ int hns_roce_get_umem_bufs(struct hns_roce_dev *hr_dev, dma_addr_t *bufs, } /* convert system page cnt to hw page cnt */ - rdma_for_each_block(umem->sg_head.sgl, &biter, umem->nmap, - 1 << page_shift) { + rdma_umem_for_each_dma_block(umem, &biter, 1 << page_shift) { addr = rdma_block_iter_dma_address(&biter); if (idx >= start) { bufs[total++] = addr; diff --git a/drivers/infiniband/hw/i40iw/i40iw_verbs.c b/drivers/infiniband/hw/i40iw/i40iw_verbs.c index b51339328a51ef..beb611b157bc8d 100644 --- a/drivers/infiniband/hw/i40iw/i40iw_verbs.c +++ b/drivers/infiniband/hw/i40iw/i40iw_verbs.c @@ -1320,8 +1320,7 @@ static void i40iw_copy_user_pgaddrs(struct i40iw_mr *iwmr, if (iwmr->type == IW_MEMREG_TYPE_QP) iwpbl->qp_mr.sq_page = sg_page(region->sg_head.sgl); - rdma_for_each_block(region->sg_head.sgl, &biter, region->nmap, - iwmr->page_size) { + rdma_umem_for_each_dma_block(region, &biter, iwmr->page_size) { *pbl = rdma_block_iter_dma_address(&biter); pbl = i40iw_next_pbl_addr(pbl, &pinfo, &idx); } diff --git a/include/rdma/ib_umem.h b/include/rdma/ib_umem.h index 07a764eb692eed..b880512ba95f16 100644 --- a/include/rdma/ib_umem.h +++ b/include/rdma/ib_umem.h @@ -40,6 +40,26 @@ static inline size_t ib_umem_num_pages(struct ib_umem *umem) PAGE_SHIFT; } +static inline void __rdma_umem_block_iter_start(struct ib_block_iter *biter, + struct ib_umem *umem, + unsigned long pgsz) +{ + __rdma_block_iter_start(biter, umem->sg_head.sgl, umem->nmap, pgsz); +} + +/** + * rdma_umem_for_each_dma_block - iterate over contiguous DMA blocks of the umem + * @umem: umem to iterate over + * @pgsz: Page size to split the list into + * + * pgsz must be <= PAGE_SIZE or computed by ib_umem_find_best_pgsz(). The + * returned DMA blocks will be aligned to pgsz and span the range: + * ALIGN_DOWN(umem->address, pgsz) to ALIGN(umem->address + umem->length, pgsz) + */ +#define rdma_umem_for_each_dma_block(umem, biter, pgsz) \ + for (__rdma_umem_block_iter_start(biter, umem, pgsz); \ + __rdma_block_iter_next(biter);) + #ifdef CONFIG_INFINIBAND_USER_MEM struct ib_umem *ib_umem_get(struct ib_device *device, unsigned long addr, From patchwork Fri Sep 4 22:41:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758609 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 122D6925 for ; Fri, 4 Sep 2020 22:42:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E3D1C20C56 for ; Fri, 4 Sep 2020 22:42:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="UHjqEZ4z" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727954AbgIDWmc (ORCPT ); Fri, 4 Sep 2020 18:42:32 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:55313 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726265AbgIDWmZ (ORCPT ); Fri, 4 Sep 2020 18:42:25 -0400 Received: from hkpgpgate102.nvidia.com (Not Verified[10.18.92.77]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:11 +0800 Received: from HKMAIL101.nvidia.com ([10.18.16.10]) by hkpgpgate102.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:11 -0700 X-PGP-Universal: processed; by hkpgpgate102.nvidia.com on Fri, 04 Sep 2020 15:42:11 -0700 Received: from HKMAIL102.nvidia.com (10.18.16.11) by HKMAIL101.nvidia.com (10.18.16.10) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:11 +0000 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.170) by HKMAIL102.nvidia.com (10.18.16.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:10 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=L9wuQFVbBQ1seJsMEhT98SJbFQvpKfs+nSf07cSeKoeV0nb7dSnBDFng4xMwJDZ6uZ3/tY5UDoIMlDeTSMhnLqWKERiYSqVKXci0JLXVkxxU+PRgT28sFAfTIDu4e4G485hd0Lay9krwvT+5PUyE+X73YoBq4g9Uz1m3cE5fQZ1omvYx9Py2eXIa/qoEQgyqVEEM6hKZ2AwS/p/Edm4XnJmiOmW55M0rpTvYvVpxhWxKPfog/FyaeEfG1mfKQ8zz2Co8bNp4xrcByEyTklqecW1Xm8/gKR7SqZvbTX0jn/o3pDJAWNxEdPVbqlng/t9jgIT/HE2lW8IpA5Hnt5QBnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xW0VYGSVThf84oKF/6nm80GT9yWZZ06eVYiLNvsdpfI=; b=kMHr2fR5Z5tDeYBk5eGV/Pe155g0CYvpfhJlJef3u33GsjfPa4kbI1VVsKQCgWKQdOOhE9IIpSnQBfC4q5MeFIjHgfwQDnkAyIyJG8g9DWH8c2iJ1Xd+RiuFbHnhY807EgkW9IDg3Hioi+7rfmNNtBiJf2WMoRpHKziAEzCE7vU4BodgRo0rz8GNSkIW5bvH1gQe4fxZgKHatT8OS5absGkZ/i9lgkU7I9+sybaBd7AQrCEq/edAF+C6+/Vf8ZzPAUwm8jDUN/JuhIFGWbbsEtCpLzIIr04Wl+xX49sbFWoCoWcwa5PjiQfauMsm+IftR2WGCQUI6Id8HrLN4Jru1g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: vmware.com; dkim=none (message not signed) header.d=none;vmware.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2856.namprd12.prod.outlook.com (2603:10b6:a03:136::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.16; Fri, 4 Sep 2020 22:42:07 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:07 +0000 From: Jason Gunthorpe To: Adit Ranadive , Potnuri Bharat Teja , Devesh Sharma , "Doug Ledford" , , "VMware PV-Drivers" , Selvin Xavier Subject: [PATCH v2 05/17] RDMA/umem: Replace for_each_sg_dma_page with rdma_umem_for_each_dma_block Date: Fri, 4 Sep 2020 19:41:46 -0300 Message-ID: <5-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR17CA0017.namprd17.prod.outlook.com (2603:10b6:208:15e::30) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by MN2PR17CA0017.namprd17.prod.outlook.com (2603:10b6:208:15e::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.15 via Frontend Transport; Fri, 4 Sep 2020 22:42:04 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP8-001vCF-UB; Fri, 04 Sep 2020 19:41:58 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8f9509c1-9b33-4931-59b6-08d85123bed3 X-MS-TrafficTypeDiagnostic: BYAPR12MB2856: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5516; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5NQ1AsjM+Hve3wK19rHGYuB8WJ3WPYPnjEkTfZwuncGTsbHOUhpg+efsPA1Osz2eJnpFO7/mTQCJ/Tw4zCMHZe5L2GzJSGotaq4vsa4+9tRxmrnw5NpoZY2uYkzTd8L362UxQ1F7FJ7VOqhBYyf+m4PLN2JP89OzBLNABpN3ayhpmOWbOI9ekuassyVLoqJ4Fp7bOfD9V/SSqkUfQ274OJ4bk219nbuUlRSj2Xr8oF6TnsmZMT5FZzQlSXZzVZt634fpffVMLCEyzPVm3RFPPLoZXeMSVY5oKDRiLyM1KKWnD6eGb8Hhdr5u/H7hLRlIwj4lN2rC4986R7RSAfmVjc61tY/wHBdiKJ4essC5wn6BjQ6XuY3ZlStbqWrHMYYT X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(376002)(136003)(366004)(396003)(346002)(36756003)(8936002)(8676002)(9746002)(86362001)(9786002)(5660300002)(2906002)(2616005)(66476007)(26005)(186003)(478600001)(83380400001)(6666004)(426003)(316002)(66556008)(66946007)(110136005)(4216001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 1UxOSB8zP8p9098A+OujOgF78JXNwSzVy/eC+5pikF1GZ2ioWlkf7LSMEZECoV5aqnIKHY4vRU6vUdJvtSX0FjdBTS2LXu5x+AMril/zmGYj9eix1FzV/vCCyBjhiStQGktgCuqEJSeAtCb1yAo8punWpngr5Eg0lyN5wFTjpthF3NLATszLSCu1mzq5A/H2cu/N7JOAI2a+bpeTJIbouAEtodGLmRNGmm16ZERQ8OxRyN9+LW0PVeC8wwVjEmz1TtkAHztl9NpYTpBGZ6M0fcCpIysUMtQ6YRHeWheAQE/vIn0wIW/S0MNxIToEbfEo/5DuUhzOTFWmZHXf+v9pretLyaHF+s159cBgcUmQN/BC5+8rzddv23S0tPqwGQ6g7DkZr7AHNrI2T6rYIQyYtpa68le+OWGBKvF6OE+Spd/X3LkwNTVUaiGWpxvGBE1lHdf3Oo4LoXi4Y6JNL7BCDEsPoPhgFX3PttWOCtDRWjBR2l7FFLUCz3ubaAuc2NyoIENBc/tcse3Bgls6HLHMJ8dtNwA4FwgUloqpFPzd917W+v+BF6eQCW35Mjk3UMbhwm6FScSksfAuea1u7Dl016KEwRZ6DJObXfaBHcjuNDDNkoX8TZid2+tGslrRTBJnJa0dCpiw03NOE57EKfoJsw== X-MS-Exchange-CrossTenant-Network-Message-Id: 8f9509c1-9b33-4931-59b6-08d85123bed3 X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:04.8407 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: BAJOSoto/ydveFf8+I+9K7DvhVtPdRgNYWJl4ymZhdy0l/aXtpRxEJqgOMUJIsp7 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2856 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259331; bh=bmUtxrzFMiAf2jFyxtS+c0Tdfq8lu2dqe86kmaH/5Pg=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:Subject: Date:Message-ID:In-Reply-To:References:Content-Transfer-Encoding: Content-Type:X-ClientProxiedBy:MIME-Version: X-MS-Exchange-MessageSentRepresentingType:X-Originating-IP: X-MS-PublicTrafficType:X-MS-Office365-Filtering-Correlation-Id: X-MS-TrafficTypeDiagnostic:X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers:X-MS-Exchange-SenderADCheck: X-Microsoft-Antispam:X-Microsoft-Antispam-Message-Info: X-Forefront-Antispam-Report:X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=UHjqEZ4zRN8OokRJTre0zZ9Z8Xxch5p1lb74fDUTd5C0WUmdMZWvgFZwARZTVArI+ 8iMNCmql4Q8fvQnrX/PJkunOvRBC8mmCa4MFklQqfs0VpzMTLHti9D8F7taW/ynaAD LSAWuIRtuRFC4OhFz4ZMskveSZfZJHtPUFWZc3SVYZPb1LoF38nE1Q4Z3XPP4L86HR tEg3WMuIRhSMDV+fC3c7PVVi5p5tcYZbyQ6rUXYTjVDVCPL8Aw/UDhrKMum+Q+OLiZ n1Bg+E6fdceWkt8C04xvuU5W8Rn8xugCtisFZV4CdNGSJg36MpgSyCJQMKRzGeizZM zX7r9aumhvDKQ== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Generally drivers should be using this core helper to split up the umem into DMA pages. These drivers are all probably wrong in some way to pass PAGE_SIZE in as the HW page size. Either the driver doesn't support other page sizes and it should use 4096, or the driver does support other page sizes and should use ib_umem_find_best_pgsz() to select the best HW pages size of the HW supported set. The only case it could be correct is if the HW has a global setting for PAGE_SIZE set at driver initialization time. Signed-off-by: Jason Gunthorpe --- drivers/infiniband/hw/cxgb4/mem.c | 6 +++--- drivers/infiniband/hw/mthca/mthca_provider.c | 6 +++--- drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 7 +++---- drivers/infiniband/hw/vmw_pvrdma/pvrdma_misc.c | 9 ++++----- 4 files changed, 13 insertions(+), 15 deletions(-) diff --git a/drivers/infiniband/hw/cxgb4/mem.c b/drivers/infiniband/hw/cxgb4/mem.c index 73936c3341b77c..82afdb1987eff6 100644 --- a/drivers/infiniband/hw/cxgb4/mem.c +++ b/drivers/infiniband/hw/cxgb4/mem.c @@ -510,7 +510,7 @@ struct ib_mr *c4iw_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, __be64 *pages; int shift, n, i; int err = -ENOMEM; - struct sg_dma_page_iter sg_iter; + struct ib_block_iter biter; struct c4iw_dev *rhp; struct c4iw_pd *php; struct c4iw_mr *mhp; @@ -561,8 +561,8 @@ struct ib_mr *c4iw_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, i = n = 0; - for_each_sg_dma_page(mhp->umem->sg_head.sgl, &sg_iter, mhp->umem->nmap, 0) { - pages[i++] = cpu_to_be64(sg_page_iter_dma_address(&sg_iter)); + rdma_umem_for_each_dma_block(mhp->umem, &biter, 1 << shift) { + pages[i++] = cpu_to_be64(rdma_block_iter_dma_address(&biter)); if (i == PAGE_SIZE / sizeof(*pages)) { err = write_pbl(&mhp->rhp->rdev, pages, mhp->attr.pbl_addr + (n << 3), i, diff --git a/drivers/infiniband/hw/mthca/mthca_provider.c b/drivers/infiniband/hw/mthca/mthca_provider.c index 9fa2f9164a47b6..317e67ad915fe8 100644 --- a/drivers/infiniband/hw/mthca/mthca_provider.c +++ b/drivers/infiniband/hw/mthca/mthca_provider.c @@ -846,7 +846,7 @@ static struct ib_mr *mthca_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, u64 virt, int acc, struct ib_udata *udata) { struct mthca_dev *dev = to_mdev(pd->device); - struct sg_dma_page_iter sg_iter; + struct ib_block_iter biter; struct mthca_ucontext *context = rdma_udata_to_drv_context( udata, struct mthca_ucontext, ibucontext); struct mthca_mr *mr; @@ -895,8 +895,8 @@ static struct ib_mr *mthca_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, write_mtt_size = min(mthca_write_mtt_size(dev), (int) (PAGE_SIZE / sizeof *pages)); - for_each_sg_dma_page(mr->umem->sg_head.sgl, &sg_iter, mr->umem->nmap, 0) { - pages[i++] = sg_page_iter_dma_address(&sg_iter); + rdma_umem_for_each_dma_block(mr->umem, &biter, PAGE_SIZE) { + pages[i++] = rdma_block_iter_dma_address(&biter); /* * Be friendly to write_mtt and pass it chunks diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c index c1751c9a0f625c..933b297de2ba86 100644 --- a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c +++ b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c @@ -814,9 +814,8 @@ static void build_user_pbes(struct ocrdma_dev *dev, struct ocrdma_mr *mr, u32 num_pbes) { struct ocrdma_pbe *pbe; - struct sg_dma_page_iter sg_iter; + struct ib_block_iter biter; struct ocrdma_pbl *pbl_tbl = mr->hwmr.pbl_table; - struct ib_umem *umem = mr->umem; int pbe_cnt, total_num_pbes = 0; u64 pg_addr; @@ -826,9 +825,9 @@ static void build_user_pbes(struct ocrdma_dev *dev, struct ocrdma_mr *mr, pbe = (struct ocrdma_pbe *)pbl_tbl->va; pbe_cnt = 0; - for_each_sg_dma_page (umem->sg_head.sgl, &sg_iter, umem->nmap, 0) { + rdma_umem_for_each_dma_block (mr->umem, &biter, PAGE_SIZE) { /* store the page address in pbe */ - pg_addr = sg_page_iter_dma_address(&sg_iter); + pg_addr = rdma_block_iter_dma_address(&biter); pbe->pa_lo = cpu_to_le32(pg_addr); pbe->pa_hi = cpu_to_le32(upper_32_bits(pg_addr)); pbe_cnt += 1; diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_misc.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_misc.c index 7944c58ded0e59..ba43ad07898c2b 100644 --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_misc.c +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_misc.c @@ -182,17 +182,16 @@ int pvrdma_page_dir_insert_dma(struct pvrdma_page_dir *pdir, u64 idx, int pvrdma_page_dir_insert_umem(struct pvrdma_page_dir *pdir, struct ib_umem *umem, u64 offset) { + struct ib_block_iter biter; u64 i = offset; int ret = 0; - struct sg_dma_page_iter sg_iter; if (offset >= pdir->npages) return -EINVAL; - for_each_sg_dma_page(umem->sg_head.sgl, &sg_iter, umem->nmap, 0) { - dma_addr_t addr = sg_page_iter_dma_address(&sg_iter); - - ret = pvrdma_page_dir_insert_dma(pdir, i, addr); + rdma_umem_for_each_dma_block (umem, &biter, PAGE_SIZE) { + ret = pvrdma_page_dir_insert_dma( + pdir, i, rdma_block_iter_dma_address(&biter)); if (ret) goto exit; From patchwork Fri Sep 4 22:41:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758597 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DC539112E for ; Fri, 4 Sep 2020 22:42:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ACEF120C56 for ; Fri, 4 Sep 2020 22:42:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="lFdU5gOj" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726277AbgIDWmT (ORCPT ); Fri, 4 Sep 2020 18:42:19 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:15223 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728160AbgIDWmR (ORCPT ); Fri, 4 Sep 2020 18:42:17 -0400 Received: from hkpgpgate101.nvidia.com (Not Verified[10.18.92.77]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:13 +0800 Received: from HKMAIL103.nvidia.com ([10.18.16.12]) by hkpgpgate101.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:13 -0700 X-PGP-Universal: processed; by hkpgpgate101.nvidia.com on Fri, 04 Sep 2020 15:42:13 -0700 Received: from HKMAIL102.nvidia.com (10.18.16.11) by HKMAIL103.nvidia.com (10.18.16.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:13 +0000 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.170) by HKMAIL102.nvidia.com (10.18.16.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:13 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IXUlwGkEJQ6JM0avda2nyvLn2mBE7KoI4sAy32X1qPk7LuuX2d+7P2vI77rznRJQfxnUsUkvTGYHC0e/dQb0TEhe1T06axw2RNg0fEUBK2UdkHsPcjcJUWPJlM07bOIa3N3Q3xHDN0z23M+b2vLWCqlK4V2tMb7mt+OKDk10GSPr2G/i+vrLRJsZ/o/Mb/K2PR++l0Dd1uKbaBDd7qvl2Qp/8DJvOQ6IKQoZitQZEjHOI7IPBH/QpJH6FL1RZXtBR425uxGZOSZpDpB7QAxwal2IT9cOjzvLjy1E3jrevhxIbMQaujqyeqfCk3W+E0QK5UeNyHoGH+t57ZDJVWukeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2rP441f5O30gALdl44CikwASZxWM6W2gj9AdCxp5ZlU=; b=Kdeg9tx7tKlAqaX1im65VhaWIoX/V4e+8qDOtlZB/aWlsoGi1qfgq00hyq50NkyIcXaic1a0CgqNfVYAypXaak5PF9cVzwjgMcBVCKHZyO2A3/yQRSh10yGfq+xBwqcBupBaaTZ4s4pFMrR0PP9BodkG7HbaA0IcwrsXcFMecZpx06cUgvVbAku06KYCAuBt7Na0nXJjpO3hdP33tIadHPuXAXas/nCQgU0jtiDfsxgWhvMr/vEHp61VYG5Oug90QcKU0CA2SoCuZuKpYQg303TnXPA19R0PYQ02oJ+7lkrdtAkXxweeeVYsBtZsqLhzVd/a/dbAH4irXaDtqaPecw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: vmware.com; dkim=none (message not signed) header.d=none;vmware.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2856.namprd12.prod.outlook.com (2603:10b6:a03:136::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.16; Fri, 4 Sep 2020 22:42:09 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:09 +0000 From: Jason Gunthorpe To: Adit Ranadive , Potnuri Bharat Teja , Doug Ledford , Leon Romanovsky , , VMware PV-Drivers Subject: [PATCH v2 06/17] RDMA/umem: Split ib_umem_num_pages() into ib_umem_num_dma_blocks() Date: Fri, 4 Sep 2020 19:41:47 -0300 Message-ID: <6-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR13CA0022.namprd13.prod.outlook.com (2603:10b6:208:160::35) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by MN2PR13CA0022.namprd13.prod.outlook.com (2603:10b6:208:160::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3370.9 via Frontend Transport; Fri, 4 Sep 2020 22:42:05 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP9-001vCJ-08; Fri, 04 Sep 2020 19:41:59 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e175b47f-9d81-422c-c435-08d85123bfdd X-MS-TrafficTypeDiagnostic: BYAPR12MB2856: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5797; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FhpxTEP23BwSvDFeizFWH064Kk+JxAHGniwSWrNI71Si8E4kYaMzzNLwT+217SV05PhDnFvqaKnYG5k0yRIPWpyc2ZnrMQFS/71OO/va9mHNmbdk2iXQXMvhx/exM74YYeXdGqOYgO9ZW0VCOW9ZpRsHXroLnrLJUJMmkb+/wCyNj+TebUbrdas62rOYZyFXDKfcLjGzOAY77XTETFlkRoIWk/OLfgvMgr3kszDZFko1CgcQTuQrKGW/qN/biRYe8FrA9CMYWJlcMM6EpctLOfeRRMr+zsalB+pNqVZE9he6QvJ1dnfe8SnBwIQgBEqWLMPeYkX02aku4YDyphjB0IsagStsJ0537hau93PDkiw6/I8cG/qLrhHRf64DdGO1 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(376002)(136003)(366004)(396003)(346002)(36756003)(8936002)(8676002)(9746002)(86362001)(9786002)(5660300002)(2906002)(2616005)(66476007)(26005)(186003)(478600001)(83380400001)(6666004)(426003)(316002)(66556008)(66946007)(110136005)(4216001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 2Qls2msMsUDyMvCU+e06hq/vL7ratCEb3dR8QUk7hsLeXJB0L27YGouxpW1VLQOPTcR/r19wPdZwNCVYocidG8rKEwS8U9HfPUIO055rjSIC3ahBjrgCQ9mpRNDyQaudBbj6BI1/9wQakDH6jT18MzIai++GVsM4X8rTbx/YfgMhsoOkAlV118KZVhymkjnIA8FuxUV28ePOldfM817qHwneTDGX/I02enxZwPtHfGQ7HLL/b8ZZnDBOyjm4NlxXbMponZJtbLN/UuTsC5aV+Vf0LKQ159jcoEwlLAqfBhXw8nPerCdT9e65mHoyCHXtepdmOZqRQVaRperdkXzwBx7oPRBrB+L0EP6SKXGTQ3CDohv54wCjqaDBY3kqGOQHYHnKJ3sQvvwi1YfgQu1kHaJXLApIuR5B4Urvbt5WWHwgx33FesV3G7F7Vb3YuBRU9osoblBQXgL6uucj4t8EOqLRBVrWpAiXRKb1ToMMTslyR9F4/91OzMwFlwNcf3IzJuSqPtOPua7eXjxOJD1+ZX8FH1ltK6oWngld3VEpon2Fke/5wicRGr3sf2AJ2XXZB+HPBJetYWDc1oUs1fkMzXt5ylpGdIAXqar4raFyWumHa9fy7coddEBamFaAayWoQ2JnFmbPIKXzxPKM6W+oEA== X-MS-Exchange-CrossTenant-Network-Message-Id: e175b47f-9d81-422c-c435-08d85123bfdd X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:06.2179 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: xB50WPqmkXRvUMiIJvF7lZ9yxuy0j4Cra7vhYTm4J0HRoSWf9lvP4WviBsKekwu7 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2856 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259334; bh=fEtbnP9lRhZurTGj6SuS/fPwWIlNYUcDxIyy5M5jptY=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:Subject: Date:Message-ID:In-Reply-To:References:Content-Transfer-Encoding: Content-Type:X-ClientProxiedBy:MIME-Version: X-MS-Exchange-MessageSentRepresentingType:X-Originating-IP: X-MS-PublicTrafficType:X-MS-Office365-Filtering-Correlation-Id: X-MS-TrafficTypeDiagnostic:X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers:X-MS-Exchange-SenderADCheck: X-Microsoft-Antispam:X-Microsoft-Antispam-Message-Info: X-Forefront-Antispam-Report:X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=lFdU5gOjpebkgxQSTdYH5rHgysKLnOr6B1s5vqP9HGSyUZ12vg7if7WQKlFckHYD9 HADbRImmbWQbVJMzitKdqYcKWvkscVEDoSLY664suGgs2crRADL6mcDNJT3lZ7lWZy FInlSAlknnskRuWxYMy7ryhHPKbwz0OiVeZdenqNOBPQMPjVuc0thvTbxaXOkD3A9U n+UIYqdMFHDkmbRPxWUvvmvjzhmdvgE03mqqeCpewCqVjWVdBvu8aCiMmt72BIOHBm oReaWeH396qg97Bzl2aGicmhG5NY4IT/AJgsqtPAzZ25SHgNIrdsUCH++85qBfbS+P 0Lxwzsh4ngs9A== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org ib_num_pages() should only be used by things working with the SGL in CPU pages directly. Drivers building DMA lists should use the new ib_num_dma_blocks() which returns the number of blocks rdma_umem_for_each_block() will return. To make this general for DMA drivers requires a different implementation. Computing DMA block count based on umem->address only works if the requested page size is < PAGE_SIZE and/or the IOVA == umem->address. Instead the number of DMA pages should be computed in the IOVA address space, not umem->address. Thus the IOVA has to be stored inside the umem so it can be used for these calculations. For now set it to umem->address by default and fix it up if ib_umem_find_best_pgsz() was called. This allows drivers to be converted to ib_umem_num_dma_blocks() safely. Signed-off-by: Jason Gunthorpe --- drivers/infiniband/core/umem.c | 7 ++++++- drivers/infiniband/hw/cxgb4/mem.c | 2 +- drivers/infiniband/hw/mlx5/mem.c | 4 ++-- drivers/infiniband/hw/mthca/mthca_provider.c | 2 +- drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c | 2 +- include/rdma/ib_umem.h | 15 ++++++++++++--- 6 files changed, 23 insertions(+), 9 deletions(-) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index fb7630e7aac3a7..b57dbb14de8378 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -161,7 +161,7 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, if (WARN_ON(!(pgsz_bitmap & GENMASK(PAGE_SHIFT, 0)))) return 0; - va = virt; + umem->iova = va = virt; /* The best result is the smallest page size that results in the minimum * number of required pages. Compute the largest page size that could * work based on VA address bits that don't change. @@ -240,6 +240,11 @@ struct ib_umem *ib_umem_get(struct ib_device *device, unsigned long addr, umem->ibdev = device; umem->length = size; umem->address = addr; + /* + * Drivers should call ib_umem_find_best_pgsz() to set the iova + * correctly. + */ + umem->iova = addr; umem->writable = ib_access_writable(access); umem->owning_mm = mm = current->mm; mmgrab(mm); diff --git a/drivers/infiniband/hw/cxgb4/mem.c b/drivers/infiniband/hw/cxgb4/mem.c index 82afdb1987eff6..22c8f5745047db 100644 --- a/drivers/infiniband/hw/cxgb4/mem.c +++ b/drivers/infiniband/hw/cxgb4/mem.c @@ -548,7 +548,7 @@ struct ib_mr *c4iw_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, shift = PAGE_SHIFT; - n = ib_umem_num_pages(mhp->umem); + n = ib_umem_num_dma_blocks(mhp->umem, 1 << shift); err = alloc_pbl(mhp, n); if (err) goto err_umem_release; diff --git a/drivers/infiniband/hw/mlx5/mem.c b/drivers/infiniband/hw/mlx5/mem.c index c19ec9fd8a63c3..13de3d2edd34e3 100644 --- a/drivers/infiniband/hw/mlx5/mem.c +++ b/drivers/infiniband/hw/mlx5/mem.c @@ -169,8 +169,8 @@ void mlx5_ib_populate_pas(struct mlx5_ib_dev *dev, struct ib_umem *umem, int page_shift, __be64 *pas, int access_flags) { return __mlx5_ib_populate_pas(dev, umem, page_shift, 0, - ib_umem_num_pages(umem), pas, - access_flags); + ib_umem_num_dma_blocks(umem, PAGE_SIZE), + pas, access_flags); } int mlx5_ib_get_buf_offset(u64 addr, int page_shift, u32 *offset) { diff --git a/drivers/infiniband/hw/mthca/mthca_provider.c b/drivers/infiniband/hw/mthca/mthca_provider.c index 317e67ad915fe8..b785fb9a2634ff 100644 --- a/drivers/infiniband/hw/mthca/mthca_provider.c +++ b/drivers/infiniband/hw/mthca/mthca_provider.c @@ -877,7 +877,7 @@ static struct ib_mr *mthca_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, goto err; } - n = ib_umem_num_pages(mr->umem); + n = ib_umem_num_dma_blocks(mr->umem, PAGE_SIZE); mr->mtt = mthca_alloc_mtt(dev, n); if (IS_ERR(mr->mtt)) { diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c index 91f0957e611543..e80848bfb3bdbf 100644 --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c @@ -133,7 +133,7 @@ struct ib_mr *pvrdma_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, return ERR_CAST(umem); } - npages = ib_umem_num_pages(umem); + npages = ib_umem_num_dma_blocks(umem, PAGE_SIZE); if (npages < 0 || npages > PVRDMA_PAGE_DIR_MAX_PAGES) { dev_warn(&dev->pdev->dev, "overflow %d pages in mem region\n", npages); diff --git a/include/rdma/ib_umem.h b/include/rdma/ib_umem.h index b880512ba95f16..0f1ab3d8f77dea 100644 --- a/include/rdma/ib_umem.h +++ b/include/rdma/ib_umem.h @@ -17,6 +17,7 @@ struct ib_umem_odp; struct ib_umem { struct ib_device *ibdev; struct mm_struct *owning_mm; + u64 iova; size_t length; unsigned long address; u32 writable : 1; @@ -33,11 +34,17 @@ static inline int ib_umem_offset(struct ib_umem *umem) return umem->address & ~PAGE_MASK; } +static inline size_t ib_umem_num_dma_blocks(struct ib_umem *umem, + unsigned long pgsz) +{ + return (ALIGN(umem->iova + umem->length, pgsz) - + ALIGN_DOWN(umem->iova, pgsz)) / + pgsz; +} + static inline size_t ib_umem_num_pages(struct ib_umem *umem) { - return (ALIGN(umem->address + umem->length, PAGE_SIZE) - - ALIGN_DOWN(umem->address, PAGE_SIZE)) >> - PAGE_SHIFT; + return ib_umem_num_dma_blocks(umem, PAGE_SIZE); } static inline void __rdma_umem_block_iter_start(struct ib_block_iter *biter, @@ -55,6 +62,8 @@ static inline void __rdma_umem_block_iter_start(struct ib_block_iter *biter, * pgsz must be <= PAGE_SIZE or computed by ib_umem_find_best_pgsz(). The * returned DMA blocks will be aligned to pgsz and span the range: * ALIGN_DOWN(umem->address, pgsz) to ALIGN(umem->address + umem->length, pgsz) + * + * Performs exactly ib_umem_num_dma_blocks() iterations. */ #define rdma_umem_for_each_dma_block(umem, biter, pgsz) \ for (__rdma_umem_block_iter_start(biter, umem, pgsz); \ From patchwork Fri Sep 4 22:41:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758617 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 414DF925 for ; Fri, 4 Sep 2020 22:42:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1FBC320C56 for ; Fri, 4 Sep 2020 22:42:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="GZO3Tr/3" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728160AbgIDWmf (ORCPT ); Fri, 4 Sep 2020 18:42:35 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:33066 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727850AbgIDWma (ORCPT ); Fri, 4 Sep 2020 18:42:30 -0400 Received: from hkpgpgate102.nvidia.com (Not Verified[10.18.92.100]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:13 +0800 Received: from HKMAIL102.nvidia.com ([10.18.16.11]) by hkpgpgate102.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:13 -0700 X-PGP-Universal: processed; by hkpgpgate102.nvidia.com on Fri, 04 Sep 2020 15:42:13 -0700 Received: from HKMAIL102.nvidia.com (10.18.16.11) by HKMAIL102.nvidia.com (10.18.16.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:09 +0000 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.170) by HKMAIL102.nvidia.com (10.18.16.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:09 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SNdKxZoNkI/lts2tf4XdNLdIGYxbW+YhQOdExo8ty1QS0XO4SYYerIfJ6UzAIcC98Zv/iU2xSD05AlM+SDe6XKODOXs7bvPgbq7J8oLn8iMnkrRzHmwENBq+vgd2cmxL6U0Z5jhY7a9OsPZ4WgYdl9XPmbzwN945RxyNOBpWl4CfTT+A1xTtZBrW/pM0VMgLpTUKsFrI2BZmSTrb0FvoDKwc7lnlgdqx9TDaoqv8840j7L/i1QHgyiQZZ3PbAuIwmLsytlfk3Fl7xJcqxIoICbhxZXXHvk/rvE3PNgNsjxSKn62El77SfYj4oLYkuZUcD3PiTgIbGK9ASoFP9PY1Zw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=giitQZMY89i+b77dK3w5npwF6FrFWKig3IPRepa88No=; b=ilftIGi3O6eI2F/EmhbQZCpy+XN0nuujxLY+HoU10aGjmH2DPtCqW5t5Mpv2ZPYNFPulkCjHy7fOMfk0tx4CU1L8IUaP+jieYaWYkpVSRjbL+1cpvyqWm21iDZ+1RHetr6KkOosmQoll7dk573CFb5IP/R2ObtVyAIBubSXtIx0m5CJFMZHGMwSmgDGC7KzxD1c4y445x49E6LSMvRSSeol1EoGAA1g9wbEBvl/YzOLZhcz0/V5OzDA1U5P9hpps31DFRmJMdnjF0tIqImcm7thRZ99MRcZio9mLaKdTNidTHqYH9Lie92SpOiK6wXdxGxmMkuMfZsqs2ecSvaBoxg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2856.namprd12.prod.outlook.com (2603:10b6:a03:136::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.16; Fri, 4 Sep 2020 22:42:06 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:06 +0000 From: Jason Gunthorpe To: CC: Doug Ledford , Firas JahJah , "Gal Pressman" , Shiraz Saleem , Yossi Leybovich Subject: [PATCH v2 07/17] RDMA/efa: Use ib_umem_num_dma_pages() Date: Fri, 4 Sep 2020 19:41:48 -0300 Message-ID: <7-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR02CA0017.namprd02.prod.outlook.com (2603:10b6:207:3c::30) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by BL0PR02CA0017.namprd02.prod.outlook.com (2603:10b6:207:3c::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.16 via Frontend Transport; Fri, 4 Sep 2020 22:42:03 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP9-001vCO-1j; Fri, 04 Sep 2020 19:41:59 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9798ef20-7bb9-4c49-e3fd-08d85123be9e X-MS-TrafficTypeDiagnostic: BYAPR12MB2856: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1303; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: eQ8ETT93Nel+s7/Tc5ETpDApGGhe/ECf/PVSBKPkx4wAFzvu0OdAQRIyJC1xOAczc+sC4VasShHKqijiTdMOt/NiQQswAna7yOTLHB0BU5BUCfBXgZ3AdmvdCkzhI8FuyPR6lmegbEb0DeHTcuze2/E2Mbc34/dQPCLW4uYWOyOSJlcRx7lYp0qEx9NiSZ1oLeid+3Zx2gS5nFxrq+rB5z+Ey/g5ovyXtpfRWZkPrVidTETW6fCJ0jLIn/Bjd1fm67iSNa1mgmq4zFtK6Vqz16ozy5+YpBykKIqYFCLyErqxiE4ziRm9T8TRTiq2o0ffqDEJ/YlJoGPqP/O4noTN1YpqoqhwoxQpWqfwOICQrcBJSHtnptBXWHw6LUEEK0ie X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(376002)(136003)(366004)(396003)(346002)(36756003)(4326008)(6916009)(8936002)(8676002)(9746002)(86362001)(9786002)(5660300002)(2906002)(2616005)(66476007)(26005)(186003)(478600001)(83380400001)(6666004)(426003)(316002)(66556008)(54906003)(66946007)(4216001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: pCvMXcjTLMLCKIW6ra2d9627ZSfJm7c/ccq/JSHXmRmxyrXVAwp9g6ZDOnrOMpljqb1YiwXoHMskA9OlJcTjskK0zOjM33GoBRmJh7jNEKDt97b6mQMCX+w8CYH3eslPYZIHdISZA0Dkf/Rfpy/rTG3+u77RDflzPfJ0VmzsyOptD5CTN3qJGp50aCqhnrHW8dC4hjSCpKDUcO96Sipy1A3lL+g2tJZnwYSPyzhGjFZLqfPnjn1PYLeyX17VSLM4lVTjkgSDaC35I9sbNYfIDm8aUmeoCQJgPcxf+f3Azaz7BRg/byBt+pXbVNhWak8M+ERl2mufhf1Qw/ijG3U9qxeqEQt8PJIxAWRzeGid7w4Mz5RrkObbp+e1N2GsupFraHeLQ3dUbMtiP2RKv7JczquK+0ljzqb00t472R/v62s60PxVmENN7QcSh59TERv+mtKg2GCNqBwH8UbPHhtCAnTLvUY23I9qqlA9sd7UCEJrXhFOwzk0jFVXufxqD6Od8FKIP/kLRM4xdViakq6yslp9F0A+Ntm/ZXBlcVkKGMuD9FSSv+sJX4xM7GyTYWM7l5rOL+MrwBiNpwwCNuMcTlLen6RxTBAm8mN7biLG4KWgLk+iEnvWgVe/8QZhMfoR+jZtVsig4n7vGBgC8bej1g== X-MS-Exchange-CrossTenant-Network-Message-Id: 9798ef20-7bb9-4c49-e3fd-08d85123be9e X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:03.9922 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: xWF0BLCHjedm8JZAMuP0TOi/JSlPAALIPv/K8NJKZ6dlp1FvUaDJYab2WLM0qMKY X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2856 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259333; bh=5sP45arY4Epqu8L54pHgZwqZurQIs/hxdCsem5yJ8so=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:CC: Subject:Date:Message-ID:In-Reply-To:References: Content-Transfer-Encoding:Content-Type:X-ClientProxiedBy: MIME-Version:X-MS-Exchange-MessageSentRepresentingType: X-Originating-IP:X-MS-PublicTrafficType: X-MS-Office365-Filtering-Correlation-Id:X-MS-TrafficTypeDiagnostic: X-Microsoft-Antispam-PRVS:X-MS-Oob-TLC-OOBClassifiers: X-MS-Exchange-SenderADCheck:X-Microsoft-Antispam: X-Microsoft-Antispam-Message-Info:X-Forefront-Antispam-Report: X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=GZO3Tr/3wf3v/qqxTpTAwjcfCb2SlF7c/wDV2zF3NuahMPulHBw5VZhsqhAT3SZoz uAl6v2DxRFjLGEtpNINSgHjMEaBnemNqlcj51o1PltUpJn5T6DzrvFvoybFZCaIKBk 2CnGZ0wpgFYFQfsQYTac4twvuulvjcefpddojn/LRoCRBdpB3oLHHD40EtMeAjlqEg 2sck6DHPzRFsvxUyKvtnEfnwOppc1LhjOJLcgaBY8m+X+wvxdQ1MlT4KjkLavcfJQh IcHaDuS3cq3mnKMjkSkFW0gkttI0jpKPnQrRNBo8p6qsUNr8c2JNP3kCbq2WOyd7Xq D2MFeQaNzWyVw== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org If ib_umem_find_best_pgsz() returns > PAGE_SIZE then the equation here is not correct. 'start' should be 'virt'. Change it to use the core code for page_num and the canonical calculation of page_shift. Fixes: 40ddb3f02083 ("RDMA/efa: Use API to get contiguous memory blocks aligned to device supported page size") Signed-off-by: Jason Gunthorpe Tested-by: Gal Pressman Acked-by: Gal Pressman --- drivers/infiniband/hw/efa/efa_verbs.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/efa/efa_verbs.c b/drivers/infiniband/hw/efa/efa_verbs.c index d85c63a5021a70..72da0faa7ebf97 100644 --- a/drivers/infiniband/hw/efa/efa_verbs.c +++ b/drivers/infiniband/hw/efa/efa_verbs.c @@ -4,6 +4,7 @@ */ #include +#include #include #include @@ -1538,9 +1539,8 @@ struct ib_mr *efa_reg_mr(struct ib_pd *ibpd, u64 start, u64 length, goto err_unmap; } - params.page_shift = __ffs(pg_sz); - params.page_num = DIV_ROUND_UP(length + (start & (pg_sz - 1)), - pg_sz); + params.page_shift = order_base_2(pg_sz); + params.page_num = ib_umem_num_dma_blocks(mr->umem, pg_sz); ibdev_dbg(&dev->ibdev, "start %#llx length %#llx params.page_shift %u params.page_num %u\n", From patchwork Fri Sep 4 22:41:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758603 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0AA17112E for ; Fri, 4 Sep 2020 22:42:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E0AB520C56 for ; Fri, 4 Sep 2020 22:42:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="pjZuX9Xa" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726948AbgIDWmY (ORCPT ); Fri, 4 Sep 2020 18:42:24 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:19849 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727827AbgIDWmU (ORCPT ); Fri, 4 Sep 2020 18:42:20 -0400 Received: from hkpgpgate102.nvidia.com (Not Verified[10.18.92.100]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:17 +0800 Received: from HKMAIL104.nvidia.com ([10.18.16.13]) by hkpgpgate102.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:17 -0700 X-PGP-Universal: processed; by hkpgpgate102.nvidia.com on Fri, 04 Sep 2020 15:42:17 -0700 Received: from HKMAIL102.nvidia.com (10.18.16.11) by HKMAIL104.nvidia.com (10.18.16.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:12 +0000 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.170) by HKMAIL102.nvidia.com (10.18.16.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=S202TeE6LxoArj/pcpfxISVu4RaOOJ0+nRj5fmdtqmBBk8zkaeNQY9F0tDQot+pxpQn7yH8jOMyZFvFvo11MNHZKdqpwYRYWiJfbo2v02tmYA11vx5QzYKXSmWvzdsGGjqLpLVh9Fb5gqcMhayMK04N0V9t8dV9B0hkmJm/9WneuYHHvOGArs0lw2rvxT2rCHtVwc8XPQhhc4u6Hyp66ArrLJPcO75nDeHxTWVQ120oFnAM156CDw5bUypx8dgSy190kSAfvBFzuLroMqqFlKoJMU4vPRf/qoWxUn7PsTnUb7hanM85zahAGP3SzN/kC/+RrkzIoGxLlAoYbyoWI/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TigaYwxJvEBZQqjvGv/rOIfMzzjp+V0cB+2nhDYtIQk=; b=N3KXVYsF8Rjkm3MPYJhad2fKY+fhc4SOk4IIe2FjutoK38oi/MeVQ0Du/L+Z6Zuz2d4oyofYNWAXsiH7vn5Ekq99ETuUXnZdiVFXaCz1z6iHHtckTdF74DQS5kf89yxi09px24k1Nqi5xHQdSr57wbbfLvcAaxOLI69sFjvij+mKtv8BkndG1SIlx1Worpx5LiDIlK5CQC3WJk0XUov1l8QDEBuZWnH4fQVZkPVpNQmA8hiItzyLUHDXYxwJAW2sWkdH36Lxx0enLnp9GbDa6+a0rNrbhLsfAfw8vFsiZ4mD/4siLqV2q2/vd1EMqqb2rVkuDf60GttBTVPtF18rOw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2856.namprd12.prod.outlook.com (2603:10b6:a03:136::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.16; Fri, 4 Sep 2020 22:42:08 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:08 +0000 From: Jason Gunthorpe To: Faisal Latif , CC: Doug Ledford , Henry Orosco , "Michael J. Ruhl" , Shiraz Saleem Subject: [PATCH v2 08/17] RDMA/i40iw: Use ib_umem_num_dma_pages() Date: Fri, 4 Sep 2020 19:41:49 -0300 Message-ID: <8-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR17CA0004.namprd17.prod.outlook.com (2603:10b6:208:15e::17) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by MN2PR17CA0004.namprd17.prod.outlook.com (2603:10b6:208:15e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.16 via Frontend Transport; Fri, 4 Sep 2020 22:42:05 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP9-001vCS-2t; Fri, 04 Sep 2020 19:41:59 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1baba601-dfe3-4ca9-51b4-08d85123bfb9 X-MS-TrafficTypeDiagnostic: BYAPR12MB2856: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6430; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vthwxRrvfAnsBtdJ1dX99ptezCkQDBOJyg0aEpNzv1MnnMQXOiEas/iAXcfrjugQ6zDKUbd60HdBplPoeWaj89pFxKcaEz8oMMueMqhjsgr7pbXVXQzu+ysOOuLbdtoPE3CODoC/YHMYIZ87wk8xuAIKAjlErGrSiS/v23AkM2TjydZR4d1aKm/48wT8Ccc6GuU6u2vN0AGKZYi2qFo2BWbpxEEdZjYIpeIT4fOd3H2lEM5aanmLuUO2qKTy2AbolpIGyhi+8T00pyPWvNNhEv+MNYOiKZrLq7OqJI4Pj/ZM5NkmQgpZjMtOPEW2PxvTjTOBSrsyMBR8etXL7aC/IGwvK2UHK2f6n4xlXlOEXq4Zf+gymQMNdJMrry0uVNhk X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(376002)(136003)(366004)(396003)(346002)(36756003)(4326008)(8936002)(8676002)(9746002)(86362001)(9786002)(5660300002)(2906002)(2616005)(66476007)(26005)(186003)(478600001)(83380400001)(6666004)(426003)(316002)(66556008)(54906003)(66946007)(4216001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 7KLV3OvF+opzEj+96beOOIjhmZdKh7v9i6zdFVklyRl2GjQvUNA1RpCnCiY+AgvHzIkLblbrlMDRiRCQE3PrSZeISPDC4eyMup0I34aqiHkgRVMXaL+K4TilJ8qXEQd/DHNyOMR07yFMYG4RR1ObYh0rNBqLwpzBjdmQhS54sCTa7nFx0GQXEFehokU29mVRMQh2HbKe7SjE9Mai0clY+srOARGFk1l1miGaaQT38Roq5K5aWsdfDgsinsagOOHCpdCqvpBS/0pltLiThoxmSp7LU5h0HtqVr/tk6XKMddyTJIoTtgY3V9ze0w6tEQ3Y8QeVwGX0K973DLEBfgTUBB1yZpnv6QlZsDs4pC5PBhYdSJTw/ClmgJDVG/Wjqtlz4FiEVwf8UeB9ktvL7n2pxnzabDtZpdaKZGeSKKES5Bv6Lc+1BbshYXCN6J4orf+yfxJS7tSJqYIwMjWw118H1AW9vOyLnVdp7R3fR7y1kF33K++J/QL9kusEhY8MwJmUNG1mnGGaIzSAlceAv8vkTjeHvAPtDpVz3SbXRFecF5em9CuSDVwtmCKqVtZA9VmRjom8GeubHxXTKPc2aEfvh7aGwEo5E2R7lHBgS4bnq89nzY84QxRBd+LfLWbfJpuA8vryx9Kmasj3WjxFM3QO0g== X-MS-Exchange-CrossTenant-Network-Message-Id: 1baba601-dfe3-4ca9-51b4-08d85123bfb9 X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:05.8391 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CK3qeO705X7lBmQ83vHSU3tPDJvd2YMRjdNoUrtaVLIFgecH0vOCZhTIJAGLBVAX X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2856 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259337; bh=Y9LfiPRq3vkqGDn/3lZekeuVnterUlXjF1IFDPrP/ME=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:CC: Subject:Date:Message-ID:In-Reply-To:References: Content-Transfer-Encoding:Content-Type:X-ClientProxiedBy: MIME-Version:X-MS-Exchange-MessageSentRepresentingType: X-Originating-IP:X-MS-PublicTrafficType: X-MS-Office365-Filtering-Correlation-Id:X-MS-TrafficTypeDiagnostic: X-Microsoft-Antispam-PRVS:X-MS-Oob-TLC-OOBClassifiers: X-MS-Exchange-SenderADCheck:X-Microsoft-Antispam: X-Microsoft-Antispam-Message-Info:X-Forefront-Antispam-Report: X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=pjZuX9XaAcpxx/+FIF+ve36dTL1hzrv0GVAKSUEOdWiLHitmlkc9BJVITka1X9ybw UVglhHTe6fl/dRY6IubyPUuLv53UP0qlq3vyDUAUT6REyI28s5BhTbeJOJYYNHKbhD QW+utIusgk5eAMhM7Ck+js0J098fr+FCF1i5L9drzKRFrFGMolt9jAc3qe2zoQxum3 xAkhh6eOM15sm5u26aIrYvtpqofJs4eQDCTAeBCNc0gVYUZTBGWpdkH684DZV8NvRE IYuGPmtpSFQcGZa9hqhdSVeeMv04znGJQik+/RoRm+A+gUl0I5SRldB0w9WagvZigP XYrrpKhL5+LnA== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org If ib_umem_find_best_pgsz() returns > PAGE_SIZE then the equation here is not correct. 'start' should be 'virt'. Change it to use the core code for page_num and the canonical calculation of page_shift. Fixes: eb52c0333f06 ("RDMA/i40iw: Use core helpers to get aligned DMA address within a supported page size") Signed-off-by: Jason Gunthorpe --- drivers/infiniband/hw/i40iw/i40iw_verbs.c | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/drivers/infiniband/hw/i40iw/i40iw_verbs.c b/drivers/infiniband/hw/i40iw/i40iw_verbs.c index beb611b157bc8d..ebfece162f98a4 100644 --- a/drivers/infiniband/hw/i40iw/i40iw_verbs.c +++ b/drivers/infiniband/hw/i40iw/i40iw_verbs.c @@ -1743,15 +1743,12 @@ static struct ib_mr *i40iw_reg_user_mr(struct ib_pd *pd, struct i40iw_mr *iwmr; struct ib_umem *region; struct i40iw_mem_reg_req req; - u64 pbl_depth = 0; u32 stag = 0; u16 access; - u64 region_length; bool use_pbles = false; unsigned long flags; int err = -ENOSYS; int ret; - int pg_shift; if (!udata) return ERR_PTR(-EOPNOTSUPP); @@ -1786,18 +1783,13 @@ static struct ib_mr *i40iw_reg_user_mr(struct ib_pd *pd, if (req.reg_type == IW_MEMREG_TYPE_MEM) iwmr->page_size = ib_umem_find_best_pgsz(region, SZ_4K | SZ_2M, virt); - - region_length = region->length + (start & (iwmr->page_size - 1)); - pg_shift = ffs(iwmr->page_size) - 1; - pbl_depth = region_length >> pg_shift; - pbl_depth += (region_length & (iwmr->page_size - 1)) ? 1 : 0; iwmr->length = region->length; iwpbl->user_base = virt; palloc = &iwpbl->pble_alloc; iwmr->type = req.reg_type; - iwmr->page_cnt = (u32)pbl_depth; + iwmr->page_cnt = ib_umem_num_dma_blocks(region, iwmr->page_size); switch (req.reg_type) { case IW_MEMREG_TYPE_QP: From patchwork Fri Sep 4 22:41:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758595 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 79DCE925 for ; Fri, 4 Sep 2020 22:42:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 58AFD20C56 for ; Fri, 4 Sep 2020 22:42:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="qJKph0Ns" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727877AbgIDWmS (ORCPT ); Fri, 4 Sep 2020 18:42:18 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:12920 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728098AbgIDWmN (ORCPT ); Fri, 4 Sep 2020 18:42:13 -0400 Received: from hkpgpgate101.nvidia.com (Not Verified[10.18.92.100]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:08 +0800 Received: from HKMAIL102.nvidia.com ([10.18.16.11]) by hkpgpgate101.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:08 -0700 X-PGP-Universal: processed; by hkpgpgate101.nvidia.com on Fri, 04 Sep 2020 15:42:08 -0700 Received: from HKMAIL104.nvidia.com (10.18.16.13) by HKMAIL102.nvidia.com (10.18.16.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:05 +0000 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.107) by HKMAIL104.nvidia.com (10.18.16.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:05 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GoGStxB7eKuekByvdgDr2d1IjEPvC8TjbGuDUQGVxltrdu/54yvbST9nboDmvXrO19rlVL6jpzbSO6NwOAi53c9/eKMk3BYUqKbutf09ULSRTbnxSvg8hyqj9WXlHUTawLVibwa/Mlavze6q/+69OL0fPfmA3IKdVRJtYB/9PKtwwVdXeAGwpGWPdsBb8XD0F+Gayjpa9b8VHgZR4g2007Sy9BrO6wrjm9I3Kt48SDssAyWwBHZj7/hum+xtgJI0xEKEr1uy85oXmXavR1ZcTX1ZtBSvWavN2RMyDYicNIJ6OtA62FbGrkI1K5vT7u6RLBdb/ATL4NvIAY1L93IOhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7icTMrAHnoQ9jUwAeZ1FEGugWpgUKoGaxrIvSjRLkVY=; b=ZjWBZfcQYiv9o8k7bUP7YLyus4odzAUwSZbnhwmd9pUPkr90p7zipB/dVpl+q9wpw/xUDvmkGcR+ge7RbkoD7ndR2uC3WYVOBgTiLpFSDqEDeFqT0sG8fkcwdx+KpLfqBjMjDKTg6FjTkNy5LA1FvGGktrHFiS9iREs5GCfD0iGaUUFSL5uN50Q2gMgxgRe3N0g1Ei32+6Hwj25JRAeQiMTS6t+gsjvB8BtkZeVtVPjEUNm//B3KnC5BQfIk3iWX5CR19hI2pxjBAufmNJKwuTPaYDHoPLpaGk3/ludK0epi8Petzag7XkTaCjADWKJjUP2sA9xuxd/c3mxFQb+LPA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: marvell.com; dkim=none (message not signed) header.d=none;marvell.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2806.namprd12.prod.outlook.com (2603:10b6:a03:70::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3326.22; Fri, 4 Sep 2020 22:42:03 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:03 +0000 From: Jason Gunthorpe To: Ariel Elior , Doug Ledford , , Michal Kalderon CC: Michal Kalderon Subject: [PATCH v2 09/17] RDMA/qedr: Use rdma_umem_for_each_dma_block() instead of open-coding Date: Fri, 4 Sep 2020 19:41:50 -0300 Message-ID: <9-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR13CA0030.namprd13.prod.outlook.com (2603:10b6:208:160::43) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by MN2PR13CA0030.namprd13.prod.outlook.com (2603:10b6:208:160::43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3370.8 via Frontend Transport; Fri, 4 Sep 2020 22:42:01 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP9-001vCW-3q; Fri, 04 Sep 2020 19:41:59 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b2ad40b1-a540-4560-c275-08d85123bd9f X-MS-TrafficTypeDiagnostic: BYAPR12MB2806: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2399; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: x41rWn3kH15qSytdJ64cYO23KcKRYcfYQIs2B/dZXn2HyOppeaaDe8VHpcURGXp3x8CXf9dgI7tySWep8Z/g9zcmlrxYYFAzCR1L5mW4KwAy2f6FAn9Ocqtvtg6unc6VhxxI5FvydvmCLpWo0tS6gaBYmfuUA8zyZb1qnRx3ppN4oEFoJkn7pTGv9az5SvCY26ZeMfhpFtg4rZdxAhyUuoET6x69RcW09p0F9fxSoxEdUTOY2BnLuKjReBjHNxcCeFlGsCLI0nIwiek4IgqMFi+7Q5vxfQHO7N2fGi3vblN2SFyi3Zt7SE07O44TyCLq//wjv3ljp5P0MD6Q2Ae1MZipMogVzar5hXY9chY4YSAZAiT8i14XCoPi6wxxdyE6 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(366004)(376002)(39860400002)(396003)(136003)(346002)(66556008)(26005)(66476007)(6666004)(110136005)(8676002)(186003)(316002)(83380400001)(426003)(2616005)(9786002)(478600001)(36756003)(9746002)(86362001)(8936002)(5660300002)(2906002)(4326008)(66946007)(4216001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: zvB4+sB+jmiMic8NHA6RRNF4YpYFRGuYLNzSE25IaxeUxHXW9+ukJK8y0L+Ji5en0Ft95EXSvshey8KlQNeaLlyZHlBKO4JbjjCZpENcc/i5k5fqBTGN66kE++Ds6aqBJcWL3ZRRJNfLprgH32Rf4nadCdZQ4bUIAbri4AhfcCmrrayZ5HOQEFLsq/iJFLuDkFePiHh2bMzLx7s/Gdi3xvSqAq/k7tSWnipZJ5tP1EJ4UZZz3ioBTFQpltRlT/vR0UuWHjoLai6yij1T0G47H38R7qSa+w1pml0nowWnDZInyCIsYtc95IDkU1FI+PJCe16hAPSHfPlowkf5J6IYIau3uv8tw5pz2n69A06WkGkxOy3IFAQacltKvsXwxnjakwRZlIzJ+h4p7KCz71hoo5cY77KirGQyMfaxS91cXt0fMv5PHxk1b99HAF8UKcvaUO7/tluJyKX9FYn3eMgLbdS/XBhb7UIzU+s20bPmVvZiZopM3QJJfucf+X/M2W4dfa+GB3/NJrGcucpjdseWEE/kcDjcHq03TILfG1vpEGn7P7JQparDHV08UijUHC0LtA4NkAWNpf1mxf6wJYvBFlpAXqHUgsgvWDwFQTJB2oM8TTjsTSKft4WBN1NwOj+yYdmkc5cD41iZ6JOt6Q2xkA== X-MS-Exchange-CrossTenant-Network-Message-Id: b2ad40b1-a540-4560-c275-08d85123bd9f X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:02.2632 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qsAeaKGeQlMDK5GkP0oFs5DZwvaHtUqLgP6gDNsSzHSNZgAskb4WDAT6jxS5zxln X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2806 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259328; bh=iIh8nXFWRGrA1GmIFAq4hJ7Dz/6/m7dhzG6+bSl+AXo=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:CC: Subject:Date:Message-ID:In-Reply-To:References: Content-Transfer-Encoding:Content-Type:X-ClientProxiedBy: MIME-Version:X-MS-Exchange-MessageSentRepresentingType: X-Originating-IP:X-MS-PublicTrafficType: X-MS-Office365-Filtering-Correlation-Id:X-MS-TrafficTypeDiagnostic: X-Microsoft-Antispam-PRVS:X-MS-Oob-TLC-OOBClassifiers: X-MS-Exchange-SenderADCheck:X-Microsoft-Antispam: X-Microsoft-Antispam-Message-Info:X-Forefront-Antispam-Report: X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=qJKph0Ns8yQieudGvDl4ytYM9xBMe/lURHVnRmdtM6H9+fi6kfLQ0V2h5YWBxjna/ 4WhlDn5+OfztbHdyeAPvotz9Jmb528AMQvkdqCBfBBIwga9vLDdqbA3C2kIXGBzu+A 1vfbyhL973VEisohJqfYEnLEbFveHy+bL2C+3Wt2EyekMxe0GAfg+j52fbRpg/4zf5 ldN039Y+bCluLfm0HSMIlwmyheHRxMELxIIK0tSdN60mlV/5G7Br8x/l46KsuK4ne9 zUf9Wzcx4Xt/RX23a8RdaXdVSJ1uezlKZu6R2UNL9unyQ70PrTp/X+Or2pDorxD+Em eHPLiOKQxbWRg== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org This loop is splitting the DMA SGL into pg_shift sized pages, use the core code for this directly. Signed-off-by: Jason Gunthorpe Acked-by: Michal Kalderon --- drivers/infiniband/hw/qedr/verbs.c | 41 ++++++++++++------------------ 1 file changed, 16 insertions(+), 25 deletions(-) diff --git a/drivers/infiniband/hw/qedr/verbs.c b/drivers/infiniband/hw/qedr/verbs.c index b49bef94637e50..cbb49168d9f7ed 100644 --- a/drivers/infiniband/hw/qedr/verbs.c +++ b/drivers/infiniband/hw/qedr/verbs.c @@ -600,11 +600,9 @@ static void qedr_populate_pbls(struct qedr_dev *dev, struct ib_umem *umem, struct qedr_pbl_info *pbl_info, u32 pg_shift) { int pbe_cnt, total_num_pbes = 0; - u32 fw_pg_cnt, fw_pg_per_umem_pg; struct qedr_pbl *pbl_tbl; - struct sg_dma_page_iter sg_iter; + struct ib_block_iter biter; struct regpair *pbe; - u64 pg_addr; if (!pbl_info->num_pbes) return; @@ -625,32 +623,25 @@ static void qedr_populate_pbls(struct qedr_dev *dev, struct ib_umem *umem, pbe_cnt = 0; - fw_pg_per_umem_pg = BIT(PAGE_SHIFT - pg_shift); + rdma_umem_for_each_dma_block (umem, &biter, BIT(pg_shift)) { + u64 pg_addr = rdma_block_iter_dma_address(&biter); - for_each_sg_dma_page (umem->sg_head.sgl, &sg_iter, umem->nmap, 0) { - pg_addr = sg_page_iter_dma_address(&sg_iter); - for (fw_pg_cnt = 0; fw_pg_cnt < fw_pg_per_umem_pg;) { - pbe->lo = cpu_to_le32(pg_addr); - pbe->hi = cpu_to_le32(upper_32_bits(pg_addr)); + pbe->lo = cpu_to_le32(pg_addr); + pbe->hi = cpu_to_le32(upper_32_bits(pg_addr)); - pg_addr += BIT(pg_shift); - pbe_cnt++; - total_num_pbes++; - pbe++; + pbe_cnt++; + total_num_pbes++; + pbe++; - if (total_num_pbes == pbl_info->num_pbes) - return; + if (total_num_pbes == pbl_info->num_pbes) + return; - /* If the given pbl is full storing the pbes, - * move to next pbl. - */ - if (pbe_cnt == (pbl_info->pbl_size / sizeof(u64))) { - pbl_tbl++; - pbe = (struct regpair *)pbl_tbl->va; - pbe_cnt = 0; - } - - fw_pg_cnt++; + /* If the given pbl is full storing the pbes, move to next pbl. + */ + if (pbe_cnt == (pbl_info->pbl_size / sizeof(u64))) { + pbl_tbl++; + pbe = (struct regpair *)pbl_tbl->va; + pbe_cnt = 0; } } } From patchwork Fri Sep 4 22:41:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758607 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B9FCB925 for ; Fri, 4 Sep 2020 22:42:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9B5E420C56 for ; Fri, 4 Sep 2020 22:42:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="hn+eWeIz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727921AbgIDWmb (ORCPT ); Fri, 4 Sep 2020 18:42:31 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:20502 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728116AbgIDWmV (ORCPT ); Fri, 4 Sep 2020 18:42:21 -0400 Received: from hkpgpgate101.nvidia.com (Not Verified[10.18.92.100]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:10 +0800 Received: from HKMAIL103.nvidia.com ([10.18.16.12]) by hkpgpgate101.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:10 -0700 X-PGP-Universal: processed; by hkpgpgate101.nvidia.com on Fri, 04 Sep 2020 15:42:10 -0700 Received: from HKMAIL104.nvidia.com (10.18.16.13) by HKMAIL103.nvidia.com (10.18.16.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:05 +0000 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.107) by HKMAIL104.nvidia.com (10.18.16.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:05 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FvXssbY9skEOPUgYJgcrASHjEiNucEo5MVpqKn4Dztq3cERBdIDI6lIL1gdT4oP2TE7xGRJS9wM/4svdL5nhsxtguYTM8cIEF264S+JdlljzRy6tW3M9EJZfMoLo32XMqFEHf3KvG+ub/6Lne7aqqKJzNzPqskcKrb4nyIgPvPUvHpnaMMU5/CaReWwx4reZj/Y77mcai2ZqMJhmsb9dSD0QZKqAosKoh5ezFUMD89PmKZNttvvVyBA1i+SYz2t+Olni8z/q8JTelnlIl9Fsln34h7e6Zt1wF59i+lYC7e3YwyGQ/75/AIID+e4crSQdZNLd4rsFI6QirbXM1mm8Ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7c04WC3o2eLVhtwACSJGPMYcNRjdtWY65CCqcFkl6Os=; b=JVOodibClnfPOIRUZc/cBfBOOaUWjRdhtOeqQGDObgKrn2wL7dL6JqQiGksw73mflwUjMdI/sYG/ISp5mXmN2vd4LSHGAAREPp7vhfTJSnjE5tiIzn3d6WawnufTEDPNTkt/9TortDz349bfrakC5ewFYzUPWA7Kzy1DBqzBDyOM9nOTs4Z5bGTYGd2A4uBFKJ8sS/yCMhNhOqiTRgjVht/EKhu0vfNkQUp41cgeSD3mQUI59o9QBZdhrkChYRpdwQzsHZeHluiDLJDtJL5cNXG63k7RzgIfWUfSk6pfg5VaKxrtemt61e+9e/y78CDqi/08gYp9qQN37YDgpiMDAQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: marvell.com; dkim=none (message not signed) header.d=none;marvell.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2806.namprd12.prod.outlook.com (2603:10b6:a03:70::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3326.22; Fri, 4 Sep 2020 22:42:03 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:03 +0000 From: Jason Gunthorpe To: Ariel Elior , Doug Ledford , , Michal Kalderon CC: Michal Kalderon Subject: [PATCH v2 10/17] RDMA/qedr: Use ib_umem_num_dma_blocks() instead of ib_umem_page_count() Date: Fri, 4 Sep 2020 19:41:51 -0300 Message-ID: <10-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR13CA0029.namprd13.prod.outlook.com (2603:10b6:208:160::42) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by MN2PR13CA0029.namprd13.prod.outlook.com (2603:10b6:208:160::42) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3370.8 via Frontend Transport; Fri, 4 Sep 2020 22:42:01 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP9-001vCZ-58; Fri, 04 Sep 2020 19:41:59 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b5e45a9f-5d0a-457a-c275-08d85123bd9f X-MS-TrafficTypeDiagnostic: BYAPR12MB2806: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2733; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: aRKy1WJNRHxfVJz4qDVEVqJda9er8WBfTsilwmdciAMB7joRvu2d4g465eiT4b7wXZ5meJ+9m0vlWz0JImQ8+GDF/jb/+mU2C+gTj7Hy2iHuIoJPAmc3I3b6wJtxFY2VvySZ9SofmcjXvO7JygPhn79xAkA6XrpUGGHLDX+IEOrfjKXVN21Y6S2ho84Wkl+/In44dqBHSeMQDaHq83GkKjZWwkwxrSEVF2NCs9K4eyy29HZruLGETGiy57LrN8xrGjRcufh6lri7N9zb9md51btTo4XT1d+xaavvrEY5zHV2Ma7xpkG7Xa414wezI0enunavAClQ8nMVUW7eCJcqWA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(366004)(376002)(39860400002)(396003)(136003)(346002)(66556008)(26005)(66476007)(6666004)(110136005)(8676002)(186003)(316002)(83380400001)(426003)(2616005)(9786002)(478600001)(36756003)(9746002)(86362001)(8936002)(5660300002)(2906002)(4326008)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: qwBtRRxKwJ9ZokEuMb/2SW2u8vEATP8HD/bVeZg0qMjviBWosduWFUtKJgjKAhfDmNcmCeSl960q1S7muSrDpks6wif0FxuGmq4nVfSmR+16f+GPpjPuYv6KNo44GbheesGsdou97rlr12H3Y4fCgtikV86Ftc+syYf4Z7SsJYKQ41O0QwgcjKXYhezzN96ainYGslQlpj2kBP+YChLAYn0Kn/PbkPAm3BRlp+LNOdUqI4EEvLkge7kN/CeAnZbU6dwG9huKGShklE8BJOBpw8FZ9RfjIJ065C3AZyLmVTE5JGiTZJKmd/uGzkoUU9G+HjLQV9V4qpUHeppewY4v6s5WS1qfcz5S7l7YDmP7upmFPdYvOt5fOZYEdoSNfIUt7ufENAa7gkiaUEVGvgKLVNRe1WAAyK81RN+2L3xUWdKN25+rG7gQ2Qo7/dgVrYjwINfdCyklWXYPbp+tWuqCOO4PV+OpwH9sO34Ml7iUKtwjbDKqlcv6TQYu9X8q6nmpyvlSTBQAfOtRcVkvhmkQ8QJhPYM1O1ArIwdc0I6ByhOQunVh99sgL9rRQ60dEfubrAkst/EuzHB7IT4Lvf1qzkwK3p1vlaFIyAF5RjG0L2lwyLnl5+XgTJ/kDW9Py2QBCInIOxXgnmeR83laitpHKg== X-MS-Exchange-CrossTenant-Network-Message-Id: b5e45a9f-5d0a-457a-c275-08d85123bd9f X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:02.3302 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ExGEVYpxwatgs21twYKhcbGZQ2sAWIVmFHTp7bsU6mUGF6HrK7zNe0EwPNaAoPe4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2806 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259330; bh=3bhMv1rv/dc+Mxiw8T/sjYSqIdaqZoUhcMwOB2y8rGk=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:CC: Subject:Date:Message-ID:In-Reply-To:References: Content-Transfer-Encoding:Content-Type:X-ClientProxiedBy: MIME-Version:X-MS-Exchange-MessageSentRepresentingType: X-Originating-IP:X-MS-PublicTrafficType: X-MS-Office365-Filtering-Correlation-Id:X-MS-TrafficTypeDiagnostic: X-Microsoft-Antispam-PRVS:X-MS-Oob-TLC-OOBClassifiers: X-MS-Exchange-SenderADCheck:X-Microsoft-Antispam: X-Microsoft-Antispam-Message-Info:X-Forefront-Antispam-Report: X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=hn+eWeIzNlS40lIUWaa+HynC+0YKyOKdSZaCFnL2mDrBT5oX6rPhOY3Q9vx2lKhXM k/XV4UEgOmWf9BnZerHVu3tgJYzvTnL8Cmt6u5G6/T61GLE69WLPMhkzkwoJlnFehr 9dUpfCq4u0ZyS8ArtQsUoTmMUttzQyTcICREhV+hRrhBOwzN9BuZLSaRX4h14Bke3k 72kbr6ljxGLkdzc5f0C51pJM5PmNBBz2H1XaA0CUvMKmQ0tq+N8ml9SN7LMqvTb9Db 1oS4zAxpdqBJFueQmF62o6IPa3W61NkvwpjFqPPtIelLvOBRMbyCcQbsU0842UKRSd Nu/jCe0OAi0aw== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The length of the list populated by qedr_populate_pbls() should be calculated using ib_umem_num_dma_blocks() with the same size/shift passed to qedr_populate_pbls(). Signed-off-by: Jason Gunthorpe Acked-by: Michal Kalderon --- drivers/infiniband/hw/qedr/verbs.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/hw/qedr/verbs.c b/drivers/infiniband/hw/qedr/verbs.c index cbb49168d9f7ed..278b48443aedba 100644 --- a/drivers/infiniband/hw/qedr/verbs.c +++ b/drivers/infiniband/hw/qedr/verbs.c @@ -783,9 +783,7 @@ static inline int qedr_init_user_queue(struct ib_udata *udata, return PTR_ERR(q->umem); } - fw_pages = ib_umem_page_count(q->umem) << - (PAGE_SHIFT - FW_PAGE_SHIFT); - + fw_pages = ib_umem_num_dma_blocks(q->umem, 1 << FW_PAGE_SHIFT); rc = qedr_prepare_pbl_tbl(dev, &q->pbl_info, fw_pages, 0); if (rc) goto err0; @@ -2852,7 +2850,8 @@ struct ib_mr *qedr_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 len, goto err0; } - rc = init_mr_info(dev, &mr->info, ib_umem_page_count(mr->umem), 1); + rc = init_mr_info(dev, &mr->info, + ib_umem_num_dma_blocks(mr->umem, PAGE_SIZE), 1); if (rc) goto err1; From patchwork Fri Sep 4 22:41:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758613 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 39779925 for ; Fri, 4 Sep 2020 22:42:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 16B0F20C56 for ; Fri, 4 Sep 2020 22:42:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="g7fb/RIf" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726441AbgIDWme (ORCPT ); Fri, 4 Sep 2020 18:42:34 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:23103 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726833AbgIDWm1 (ORCPT ); Fri, 4 Sep 2020 18:42:27 -0400 Received: from hkpgpgate102.nvidia.com (Not Verified[10.18.92.77]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:23 +0800 Received: from HKMAIL102.nvidia.com ([10.18.16.11]) by hkpgpgate102.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:23 -0700 X-PGP-Universal: processed; by hkpgpgate102.nvidia.com on Fri, 04 Sep 2020 15:42:23 -0700 Received: from HKMAIL102.nvidia.com (10.18.16.11) by HKMAIL102.nvidia.com (10.18.16.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:15 +0000 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.170) by HKMAIL102.nvidia.com (10.18.16.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:14 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CjYMl7DpDRYJN0nXs8JFeTXQ2065AyPll/GeB57JJIaLTJmLiW8DtBtPFQhB5/FQL2eJqVqhV4Ip3ZZ/eRbZCyiyeSYyzn/4HmZ3lw6IXxauP8Z4hQeyAyhEVRMd3H4Ipqyh37+t9Ep3dA5agCfyMeI5vw3Yx6GHLuedahem92R6DQeJXGcEDowuGcsztyuHgHZstrmmB+fnxJOLDwYkpp/01XU+/PqXAEkEaX6UY/1gbwwBLFDrAV7D57G5vdfOwwfN0jhQY0ZN0cE3ftXWeGh9lW2+a8dNPFx8BHVBHdzMkjM2m48drNrll6Yqff1Q80N9P6ZZbW8u0WelB3TNnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rv2/T8b/Q+FOdU8DYPpArF1TdL4VQTBfaa4HaMD/E7U=; b=kpye2K+O3a+ghC1pfKmY2bHWUlTrxAD2Q8/kVXm5YTYRACkEqkh8l5bxPxEWsTzZV9RqGqswimb3/uzKcjvD9CL1xZyYWkQro4k1/lIDl58aBuWHpYWjjLo4/vkNjYaw+dY2l6/0lum4XdPxMncy6bYT7XL4AuZji+aI1RQApWHNIur6GEWuMhMiv+eYUCSAQ3BJBVB8Iv+R7xlCicbUFCxcKDm5Pq0zBz4C4Gam7OseXbKkP3bAovznx1HLX8xLqN+qH7FH9tJaox8QSX0oXCA7EEZUPGtMc27NKXYd6zJFVyrewydYg8026HBQ9Qmo3KhEpMZWMTzG84lqSIMOKQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: broadcom.com; dkim=none (message not signed) header.d=none;broadcom.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2856.namprd12.prod.outlook.com (2603:10b6:a03:136::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.16; Fri, 4 Sep 2020 22:42:10 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:09 +0000 From: Jason Gunthorpe To: Devesh Sharma , Doug Ledford , , Naresh Kumar PBS , Somnath Kotur , Sriharsha Basavapatna CC: Selvin Xavier , Shiraz Saleem Subject: [PATCH v2 11/17] RDMA/bnxt: Do not use ib_umem_page_count() or ib_umem_num_pages() Date: Fri, 4 Sep 2020 19:41:52 -0300 Message-ID: <11-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR13CA0012.namprd13.prod.outlook.com (2603:10b6:208:160::25) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by MN2PR13CA0012.namprd13.prod.outlook.com (2603:10b6:208:160::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3370.8 via Frontend Transport; Fri, 4 Sep 2020 22:42:06 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP9-001vCe-64; Fri, 04 Sep 2020 19:41:59 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b7823992-87a9-433a-8561-08d85123c036 X-MS-TrafficTypeDiagnostic: BYAPR12MB2856: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2150; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: jRWx9N/qtovUXz54OC3tcZSUd+1c2WL8ihgkjPeSTPNSegufwzJEP/j1pt/aoeXO3CwUv/LD86Wfb8Pd89b5mGCScbjqaC8EIGYSVntH2lfC7p7R4M99ELXAluu1q1OxYpa4Gyw1ZBhLTwTF2zI8+xDkJEjZQwFNeIKEHDFpFlvU0twcLbEb0ErihkhirU3FyOw5MFzepaTRNSgo0WT7BIRbhcZ3eNbusjp5uSYBT5z9e7soBFX8XpAbiOxE4UWHUYGFyHxO1lxD0v8OAKpJSZzQ5lXNpzWGEennYOVTyCQQrAAaM0ohCbSG0nw7cHLNo2b+pfzhAmBZg+nYL7utcQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(376002)(136003)(366004)(396003)(346002)(36756003)(4326008)(8936002)(8676002)(9746002)(86362001)(9786002)(5660300002)(2906002)(2616005)(66476007)(26005)(186003)(478600001)(83380400001)(6666004)(426003)(316002)(66556008)(54906003)(66946007)(110136005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: LRWuhlcQxR+lecwpHH/fv98oR2jwt85cXsjv0FnWxCd4xF6uEaZuk1LpyRfPorrClwWQEyVsAKNsC1jS25UYj+suwwseCFFTqgOHDf3PifyHPz6OQFktLngFy6YHfwD1CJl6/Rd3rB161K8oqFum2Zu5uq8c5steUPztPo6/X4eh2dZ9ST07TPQkwYYWDOr1pwTEKLvSwTHDbip4JXGcqk+y37osdU5VTx3vF0sYhjIvWuKE6Cobg70CW/4TG3g7tXEah4Jgv5GnnMTKhsH79WJ5eN8eTPJgvUs1UwFlOStmUAFpobFiPjYF1o+3430CgpqrRsAFUdVpActQ2gs1TAwQHoCi2bkVd7pdiOaCBtUK9Us/IRRV0+CQIGlCZixVo3/3Zr/4b/shNDBesaJ3Etip6QGRMrJ/7Rdd4tshaJvvDYw5jf503JYmHojUXVI5etNyIGWx5h1MIHSinVGoCi6VTHUjMEffKHVhErynH85sQRWwAyTD5o1FSMDnRXut32o2ciZhuOGudQk5SIOnh+fvyNqYmT6kq6cYMPkQWTpGFN/5tWVqPgmwTj9Jc5w/zYCqrSKYKsgN8KZTlGIv5xPxi4ZhWS8VBGm4OaamtYwOfs8PfNDhjc5UcphDXT4oSrbNEICtreZkySEKcrJVzA== X-MS-Exchange-CrossTenant-Network-Message-Id: b7823992-87a9-433a-8561-08d85123c036 X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:06.7446 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 0haVGWGh6gypGS1126MMhc1qOXiJdxBNRKUqZtbtz69m14FWo08VCEdSyZ6MUrr5 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2856 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259343; bh=lSjACUtbbN9stT2cijJ27tRmYJzWXivJvOx4HBOMlYA=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:CC: Subject:Date:Message-ID:In-Reply-To:References: Content-Transfer-Encoding:Content-Type:X-ClientProxiedBy: MIME-Version:X-MS-Exchange-MessageSentRepresentingType: X-Originating-IP:X-MS-PublicTrafficType: X-MS-Office365-Filtering-Correlation-Id:X-MS-TrafficTypeDiagnostic: X-Microsoft-Antispam-PRVS:X-MS-Oob-TLC-OOBClassifiers: X-MS-Exchange-SenderADCheck:X-Microsoft-Antispam: X-Microsoft-Antispam-Message-Info:X-Forefront-Antispam-Report: X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=g7fb/RIf2NuSq5TW3vX9S9awg9tuUC8mSWTFbEltN70xVg7QsW1NuPrw+pgPIjqOW wVpafenIBQ2q8ItQp1mqLrSth3zSQ1qRyjanrULfQpmkN4PFIJB88fvBzZKFfBEj6E ytMi3ixvwVYzM6DvU3CB2N2RtKsJbHRFVEgyM+1xnzb9h/naqMNqSyLtkGu6dRyPzU E77HLhxL2zb1BXqbq17DP5P6Tli/LwWu7arD3tRrx3FbknOG/l4AytGqWi86Yu+zAD D745C2k+r7Bx5hrLyhSFVTLpJYvdGImxj37XJuI7JER0yAH+TadWW41UJfpScFllNs 3a0KRwK852bLw== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org ib_umem_page_count() returns the number of 4k entries required for a DMA map, but bnxt_re already computes a variable page size. The correct API to determine the size of the page table array is ib_umem_num_dma_blocks(). Fix the overallocation of the page array in fill_umem_pbl_tbl() when working with larger page sizes by using the right function. Lightly re-organize this function to make it clearer. Replace the other calls to ib_umem_num_pages(). Fixes: d85582517e91 ("RDMA/bnxt_re: Use core helpers to get aligned DMA address") Signed-off-by: Jason Gunthorpe Acked-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 70 ++++++++---------------- 1 file changed, 24 insertions(+), 46 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index 9e26e651730cb3..9dbf9ab5a4c8db 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -939,7 +939,7 @@ static int bnxt_re_init_user_qp(struct bnxt_re_dev *rdev, struct bnxt_re_pd *pd, qp->sumem = umem; qplib_qp->sq.sg_info.sghead = umem->sg_head.sgl; - qplib_qp->sq.sg_info.npages = ib_umem_num_pages(umem); + qplib_qp->sq.sg_info.npages = ib_umem_num_dma_blocks(umem, PAGE_SIZE); qplib_qp->sq.sg_info.nmap = umem->nmap; qplib_qp->sq.sg_info.pgsize = PAGE_SIZE; qplib_qp->sq.sg_info.pgshft = PAGE_SHIFT; @@ -954,7 +954,8 @@ static int bnxt_re_init_user_qp(struct bnxt_re_dev *rdev, struct bnxt_re_pd *pd, goto rqfail; qp->rumem = umem; qplib_qp->rq.sg_info.sghead = umem->sg_head.sgl; - qplib_qp->rq.sg_info.npages = ib_umem_num_pages(umem); + qplib_qp->rq.sg_info.npages = + ib_umem_num_dma_blocks(umem, PAGE_SIZE); qplib_qp->rq.sg_info.nmap = umem->nmap; qplib_qp->rq.sg_info.pgsize = PAGE_SIZE; qplib_qp->rq.sg_info.pgshft = PAGE_SHIFT; @@ -1609,7 +1610,7 @@ static int bnxt_re_init_user_srq(struct bnxt_re_dev *rdev, srq->umem = umem; qplib_srq->sg_info.sghead = umem->sg_head.sgl; - qplib_srq->sg_info.npages = ib_umem_num_pages(umem); + qplib_srq->sg_info.npages = ib_umem_num_dma_blocks(umem, PAGE_SIZE); qplib_srq->sg_info.nmap = umem->nmap; qplib_srq->sg_info.pgsize = PAGE_SIZE; qplib_srq->sg_info.pgshft = PAGE_SHIFT; @@ -2861,7 +2862,8 @@ int bnxt_re_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, goto fail; } cq->qplib_cq.sg_info.sghead = cq->umem->sg_head.sgl; - cq->qplib_cq.sg_info.npages = ib_umem_num_pages(cq->umem); + cq->qplib_cq.sg_info.npages = + ib_umem_num_dma_blocks(cq->umem, PAGE_SIZE); cq->qplib_cq.sg_info.nmap = cq->umem->nmap; cq->qplib_cq.dpi = &uctx->dpi; } else { @@ -3759,23 +3761,6 @@ int bnxt_re_dealloc_mw(struct ib_mw *ib_mw) return rc; } -static int bnxt_re_page_size_ok(int page_shift) -{ - switch (page_shift) { - case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_4K: - case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_8K: - case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_64K: - case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_2M: - case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_256K: - case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_1M: - case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_4M: - case CMDQ_REGISTER_MR_LOG2_PBL_PG_SIZE_PG_1G: - return 1; - default: - return 0; - } -} - static int fill_umem_pbl_tbl(struct ib_umem *umem, u64 *pbl_tbl_orig, int page_shift) { @@ -3799,7 +3784,8 @@ struct ib_mr *bnxt_re_reg_user_mr(struct ib_pd *ib_pd, u64 start, u64 length, struct bnxt_re_mr *mr; struct ib_umem *umem; u64 *pbl_tbl = NULL; - int umem_pgs, page_shift, rc; + unsigned long page_size; + int umem_pgs, rc; if (length > BNXT_RE_MAX_MR_SIZE) { ibdev_err(&rdev->ibdev, "MR Size: %lld > Max supported:%lld\n", @@ -3833,42 +3819,34 @@ struct ib_mr *bnxt_re_reg_user_mr(struct ib_pd *ib_pd, u64 start, u64 length, mr->ib_umem = umem; mr->qplib_mr.va = virt_addr; - umem_pgs = ib_umem_page_count(umem); - if (!umem_pgs) { - ibdev_err(&rdev->ibdev, "umem is invalid!"); - rc = -EINVAL; - goto free_umem; - } - mr->qplib_mr.total_size = length; - - pbl_tbl = kcalloc(umem_pgs, sizeof(u64 *), GFP_KERNEL); - if (!pbl_tbl) { - rc = -ENOMEM; - goto free_umem; - } - - page_shift = __ffs(ib_umem_find_best_pgsz(umem, - BNXT_RE_PAGE_SIZE_4K | BNXT_RE_PAGE_SIZE_2M, - virt_addr)); - - if (!bnxt_re_page_size_ok(page_shift)) { + page_size = ib_umem_find_best_pgsz( + umem, BNXT_RE_PAGE_SIZE_4K | BNXT_RE_PAGE_SIZE_2M, virt_addr); + if (!page_size) { ibdev_err(&rdev->ibdev, "umem page size unsupported!"); rc = -EFAULT; - goto fail; + goto free_umem; } + mr->qplib_mr.total_size = length; - if (page_shift == BNXT_RE_PAGE_SHIFT_4K && + if (page_size == BNXT_RE_PAGE_SIZE_4K && length > BNXT_RE_MAX_MR_SIZE_LOW) { ibdev_err(&rdev->ibdev, "Requested MR Sz:%llu Max sup:%llu", length, (u64)BNXT_RE_MAX_MR_SIZE_LOW); rc = -EINVAL; - goto fail; + goto free_umem; + } + + umem_pgs = ib_umem_num_dma_blocks(umem, page_size); + pbl_tbl = kcalloc(umem_pgs, sizeof(u64 *), GFP_KERNEL); + if (!pbl_tbl) { + rc = -ENOMEM; + goto free_umem; } /* Map umem buf ptrs to the PBL */ - umem_pgs = fill_umem_pbl_tbl(umem, pbl_tbl, page_shift); + umem_pgs = fill_umem_pbl_tbl(umem, pbl_tbl, order_base_2(page_size)); rc = bnxt_qplib_reg_mr(&rdev->qplib_res, &mr->qplib_mr, pbl_tbl, - umem_pgs, false, 1 << page_shift); + umem_pgs, false, page_size); if (rc) { ibdev_err(&rdev->ibdev, "Failed to register user MR"); goto fail; From patchwork Fri Sep 4 22:41:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758605 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D29F4925 for ; Fri, 4 Sep 2020 22:42:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B199E20C56 for ; Fri, 4 Sep 2020 22:42:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="FRwo/9J1" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727827AbgIDWm1 (ORCPT ); Fri, 4 Sep 2020 18:42:27 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:44606 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728084AbgIDWmV (ORCPT ); Fri, 4 Sep 2020 18:42:21 -0400 Received: from hkpgpgate102.nvidia.com (Not Verified[10.18.92.100]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:17 +0800 Received: from HKMAIL104.nvidia.com ([10.18.16.13]) by hkpgpgate102.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:17 -0700 X-PGP-Universal: processed; by hkpgpgate102.nvidia.com on Fri, 04 Sep 2020 15:42:17 -0700 Received: from HKMAIL102.nvidia.com (10.18.16.11) by HKMAIL104.nvidia.com (10.18.16.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:14 +0000 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.170) by HKMAIL102.nvidia.com (10.18.16.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:14 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EwYT8s9Xe4D04B5Uul8zm0hLs4VQVEMP9Y22DGGScoYoJ9rtcYvAPG0m14Lw1Vvj53DTNDgI07XUCCJ8cP1o9to6qVXYVQGBo+xs6Uv9xdzhtljUPv1037aZelfg/66BpmKU8ES0Wci59Ujk1wf0zcpPdLjL5/UxW5ZujJCFVhNnfUuMBU9a4mHuzmfBtsnPwUwmQEc6EpTFO1BjY3YAXdFYM6XvbMy86sbGOMVow/wwj8K9GDsyLqfW+79b6KWeBV/d/VJeQ3U6F7bpoHQMsn0sx8w1Nw90gT5ptFSNLV+moMnt4KE1pnDCryisnIhMkPl/tBHiEBiASQWHHtJAGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9DaW4fMlu+pQfCvGTkMLYrO8NO+aT6iia6vs1aHUbXw=; b=j+Tmmr/fA83P/GAXRCWfPHy8gGl+pGv9LuNrv4SAKrcyJQsiTUm2xPstPyD0cSQWXheldl2PzmXVkh5qrSBn7Mb9VDXdjNNGm0haW5caZTK0vJ20AggknqADmVSZFe8KnKMKz4d3u9i7W30o1Dx66JgWcZtLfgiumoB29Iqspcieux+S5dkQzDvnc/S4KujwF9nEvWawZz1bTeHPA/5iFFzIm8J5OudGbC2gtUNBeZhndN0BQV9pcg61+M8teL0SvRZrf3bYctkDv73SyFn7OOe2PSqrGRVORGx7l+pl1DE50Xjyl+GH71OVwO980UhgWkviRdERnbCR/p0rzjCrRg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2856.namprd12.prod.outlook.com (2603:10b6:a03:136::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.16; Fri, 4 Sep 2020 22:42:09 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:09 +0000 From: Jason Gunthorpe To: Doug Ledford , "Wei Hu(Xavier)" , , Weihang Li , Lijun Ou Subject: [PATCH v2 12/17] RDMA/hns: Use ib_umem_num_dma_blocks() instead of opencoding Date: Fri, 4 Sep 2020 19:41:53 -0300 Message-ID: <12-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR17CA0016.namprd17.prod.outlook.com (2603:10b6:208:15e::29) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by MN2PR17CA0016.namprd17.prod.outlook.com (2603:10b6:208:15e::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.15 via Frontend Transport; Fri, 4 Sep 2020 22:42:05 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP9-001vCi-77; Fri, 04 Sep 2020 19:41:59 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 836eb680-1b9c-459f-1d21-08d85123bffc X-MS-TrafficTypeDiagnostic: BYAPR12MB2856: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3631; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 28x57vPs0DdudC+lO4HbCFeef4WEaRCYtpqfpJafK4kqLAOarcSwBX0jPMgZT5Sed+m313UZ+7+Bm982SNSIFUsnd4DwZCjfN/UCG2IFBr446vx/4N+opOuj/YF9fSxhI1i5JIvFZQp3UrJAtsk01KvjJYTmkpisjLkQ4pG4DUxKXrV4HuFj52oLIBYKEJKrdReNjaaAXZsgNXTMQnJ0G4011LQd9PhxOxlzCBZu3jI6MWZzMOXJF5wxPiPsUTuVYS7o96o1QOCRKqDljVcyA0qSqCULWUX9MI136GVQZT5SpNv66wC6d4EdcgRTtvKTihu2WFULSBen60YdzKmBTQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(376002)(136003)(366004)(396003)(346002)(36756003)(8936002)(8676002)(9746002)(86362001)(9786002)(5660300002)(2906002)(2616005)(66476007)(26005)(186003)(478600001)(83380400001)(6666004)(426003)(316002)(66556008)(66946007)(110136005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: Wpxyirs0uXHLXlZuxh3FjRR6LjxMtsPVuTlO9GMCETSHJX6lm6QW/49/L8fUuU2/SdP/Zj5Dm7y0PaBi7FX/VT7Cyron/rh/4i/EupU4/rQ8SpBe17XdQtl20EVv52Aq3+nqclvZ0ajTAeSaCtB7WzRgV2C+xZcDUM1wbR+NjwC60eCLgIPhUVssYP+9v8+aJKRMRyZ72I9QzqV0b5jpHRUCtDc80L5ZBMoeGh7fmooiKtknntDeb7HM4LQHTTjXXHJFh7ipZLPLj1+ABLWXZv3JauPLwQwgMgRmzw/nkWj8hXUiEekGwUhx0XMIQQr4cOrugsYX4ObIsGYJXFx9TUsK1aZGyznpNAfrwasj4Xz3aWf7d7+7WqsTstI7nnOfD700z9PGmZVp/pOXWwXqiolNn63eS05CrgCWzr6qI89q+eu2F9JqGFQNl8/2GCTOoK0+keGrmBhDdeKTZynlyav3OuiuEXNzEThpVsmTV8tsHvkapUDaRgcySWzqeMQhSorIN5Vb4bCCeMiCs8FdOFK7RsDCX8vNzx7sRAkJlWN3K45Ioa/DyGrYChul15kZ8a4G/ZtT94g//Cqrd6p3Pp1g5Gg7egg2rOdi6fJfs7aHn8ElaNMUp6rB4e0A+ikYkAWv3Qvp0LFIXUHvG5quwg== X-MS-Exchange-CrossTenant-Network-Message-Id: 836eb680-1b9c-459f-1d21-08d85123bffc X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:06.5107 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MRZWdOKvCEZe4ZOHMxOlCtJCDsXYHF+DQ8BqNYyX5+KVHOrRWrMEvpBj+UaPMR3v X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2856 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259337; bh=Y0rbSpvE2utNEz97wF2kX2sNXyfdDN0RCk6d2kyvxEY=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:Subject: Date:Message-ID:In-Reply-To:References:Content-Transfer-Encoding: Content-Type:X-ClientProxiedBy:MIME-Version: X-MS-Exchange-MessageSentRepresentingType:X-Originating-IP: X-MS-PublicTrafficType:X-MS-Office365-Filtering-Correlation-Id: X-MS-TrafficTypeDiagnostic:X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers:X-MS-Exchange-SenderADCheck: X-Microsoft-Antispam:X-Microsoft-Antispam-Message-Info: X-Forefront-Antispam-Report:X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=FRwo/9J1ykSaZMChL97NVgyCDneE8JYLmb/OIglLkGgqE0mCSIY1E2jBqjjtpY4ii LhkpZY4xlAF9TeL56SP94hhLyvDt5JavZvACIArLeDs+8YR8Y0XLnyDzsit9WFBLgM u5NlDIVxHFwP5xU2x8dsi7uAm2hQpJsQc6rreKAp2XoMe2rP7ESi3RWy8QhnOd0bXl GeCBbjUOJ6fR/bc6oRmQeMg17RJ79qcsMIg32ZDlPioXLDaooKKnSvIcMqVb279ZW+ L8GI9GZGsaSVn/5jilldz0GGJvtvcEC44+GsodfyX0dZbdLpri7IJa6plkQ1hrPWwE Wxyg6A53yOetg== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org mtr_umem_page_count() does the same thing, replace it with the core code. Also, ib_umem_find_best_pgsz() should always be called to check that the umem meets the page_size requirement. If there is a limited set of page_sizes that work it the pgsz_bitmap should be set to that set. 0 is a failure and the umem cannot be used. Lightly tidy the control flow to implement this flow properly. Signed-off-by: Jason Gunthorpe Acked-by: Weihang Li --- drivers/infiniband/hw/hns/hns_roce_mr.c | 49 ++++++++++--------------- 1 file changed, 19 insertions(+), 30 deletions(-) diff --git a/drivers/infiniband/hw/hns/hns_roce_mr.c b/drivers/infiniband/hw/hns/hns_roce_mr.c index e5df3884b41dda..16699f6bb03a51 100644 --- a/drivers/infiniband/hw/hns/hns_roce_mr.c +++ b/drivers/infiniband/hw/hns/hns_roce_mr.c @@ -707,19 +707,6 @@ static inline size_t mtr_bufs_size(struct hns_roce_buf_attr *attr) return size; } -static inline int mtr_umem_page_count(struct ib_umem *umem, - unsigned int page_shift) -{ - int count = ib_umem_page_count(umem); - - if (page_shift >= PAGE_SHIFT) - count >>= page_shift - PAGE_SHIFT; - else - count <<= PAGE_SHIFT - page_shift; - - return count; -} - static inline size_t mtr_kmem_direct_size(bool is_direct, size_t alloc_size, unsigned int page_shift) { @@ -767,12 +754,10 @@ static int mtr_alloc_bufs(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr, struct ib_udata *udata, unsigned long user_addr) { struct ib_device *ibdev = &hr_dev->ib_dev; - unsigned int max_pg_shift = buf_attr->page_shift; - unsigned int best_pg_shift = 0; + unsigned int best_pg_shift; int all_pg_count = 0; size_t direct_size; size_t total_size; - unsigned long tmp; int ret = 0; total_size = mtr_bufs_size(buf_attr); @@ -782,6 +767,9 @@ static int mtr_alloc_bufs(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr, } if (udata) { + unsigned long pgsz_bitmap; + unsigned long page_size; + mtr->kmem = NULL; mtr->umem = ib_umem_get(ibdev, user_addr, total_size, buf_attr->user_access); @@ -790,15 +778,17 @@ static int mtr_alloc_bufs(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr, PTR_ERR(mtr->umem)); return -ENOMEM; } - if (buf_attr->fixed_page) { - best_pg_shift = max_pg_shift; - } else { - tmp = GENMASK(max_pg_shift, 0); - ret = ib_umem_find_best_pgsz(mtr->umem, tmp, user_addr); - best_pg_shift = (ret <= PAGE_SIZE) ? - PAGE_SHIFT : ilog2(ret); - } - all_pg_count = mtr_umem_page_count(mtr->umem, best_pg_shift); + if (buf_attr->fixed_page) + pgsz_bitmap = 1 << buf_attr->page_shift; + else + pgsz_bitmap = GENMASK(buf_attr->page_shift, PAGE_SHIFT); + + page_size = ib_umem_find_best_pgsz(mtr->umem, pgsz_bitmap, + user_addr); + if (!page_size) + return -EINVAL; + best_pg_shift = order_base_2(page_size); + all_pg_count = ib_umem_num_dma_blocks(mtr->umem, page_size); ret = 0; } else { mtr->umem = NULL; @@ -808,16 +798,15 @@ static int mtr_alloc_bufs(struct hns_roce_dev *hr_dev, struct hns_roce_mtr *mtr, return -ENOMEM; } direct_size = mtr_kmem_direct_size(is_direct, total_size, - max_pg_shift); + buf_attr->page_shift); ret = hns_roce_buf_alloc(hr_dev, total_size, direct_size, - mtr->kmem, max_pg_shift); + mtr->kmem, buf_attr->page_shift); if (ret) { ibdev_err(ibdev, "Failed to alloc kmem, ret %d\n", ret); goto err_alloc_mem; - } else { - best_pg_shift = max_pg_shift; - all_pg_count = mtr->kmem->npages; } + best_pg_shift = buf_attr->page_shift; + all_pg_count = mtr->kmem->npages; } /* must bigger than minimum hardware page shift */ From patchwork Fri Sep 4 22:41:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758601 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F2CD1112E for ; Fri, 4 Sep 2020 22:42:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CFD1620FC3 for ; Fri, 4 Sep 2020 22:42:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="oMQM2I6I" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726360AbgIDWmX (ORCPT ); Fri, 4 Sep 2020 18:42:23 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:57018 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727851AbgIDWmT (ORCPT ); Fri, 4 Sep 2020 18:42:19 -0400 Received: from hkpgpgate101.nvidia.com (Not Verified[10.18.92.100]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:10 +0800 Received: from HKMAIL103.nvidia.com ([10.18.16.12]) by hkpgpgate101.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:10 -0700 X-PGP-Universal: processed; by hkpgpgate101.nvidia.com on Fri, 04 Sep 2020 15:42:10 -0700 Received: from HKMAIL102.nvidia.com (10.18.16.11) by HKMAIL103.nvidia.com (10.18.16.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:08 +0000 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.170) by HKMAIL102.nvidia.com (10.18.16.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=f0mHfdhmYFtXBR5ArfWpTEWnnohA67E9gc/R8WHHONn/D1nR3ayiNOsXpzwx+/NiStVJVOD5ic+zQPNM4fj+c4PHrxZ5/cjF1bMyMXM8YDpvm9X/3Z/rTRo0Wzjpnda8m+f6PUtwCLQ+welxFKm2eaSV9oFsDnHEKclmMinZRrJMqq/KEUuPB5hwMAsPvHCunktROMXwhBZZRNqi+Cxjmm4xZpX1SfJRP7iattidhxrsyjRSuihOIV3uI8XW1uQtrCm9gaN2YxKKVqDvX13cbNoGShRwxi9vXLNVo4W7sj929qNNPIq8o90sPpgIwPR2bR3cUP2QIvoW8rusoFk45g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CtP+9dzmVEQIg2BjmMqTEmQzK/UU3apz+HyQkmWI2p0=; b=LRICUdIHXwn3ZGA35Nr/UCgNFnAeTnr0KvyMkm1HalzUbhMDUosXhQfnbHPx9Q1040Fhq8Et/0warj2WlPV9sQ/1X1DDEnGkoefCKCb5l+Zmmlt5KaJa4lfzu8rH3rfjsoN4bs9VoDFQCYrLf8Kxhg1VODW2Usv9GITnbPd7NVpzBVIXsfoA/JMVncUo/fEqEd8mtNM3vFPMLrIU+invTyVgS12XmZj/4OUJZfTIw46Dl7PELTDWrss5BhqSZquhjrCueItNIOQuo6kB+ixW70sqoKJ8md3yAaoCbhfv6LDeyh3K69ETm15TjONJQeVjjHYAvLSi3ftAP/YJXaw97w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: broadcom.com; dkim=none (message not signed) header.d=none;broadcom.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2856.namprd12.prod.outlook.com (2603:10b6:a03:136::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.16; Fri, 4 Sep 2020 22:42:06 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:06 +0000 From: Jason Gunthorpe To: Devesh Sharma , Doug Ledford , , Selvin Xavier Subject: [PATCH v2 13/17] RDMA/ocrdma: Use ib_umem_num_dma_blocks() instead of ib_umem_page_count() Date: Fri, 4 Sep 2020 19:41:54 -0300 Message-ID: <13-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR02CA0021.namprd02.prod.outlook.com (2603:10b6:207:3c::34) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by BL0PR02CA0021.namprd02.prod.outlook.com (2603:10b6:207:3c::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.15 via Frontend Transport; Fri, 4 Sep 2020 22:42:02 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP9-001vCl-82; Fri, 04 Sep 2020 19:41:59 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 35895dcd-d249-447f-f1a1-08d85123be1e X-MS-TrafficTypeDiagnostic: BYAPR12MB2856: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4941; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ISj60HXOBSswx/6lnilOlR0apOSBCp1UzQv4U9nkUb454yJVRLzhkmR2LZE9jN2gsLo2UO8P29KYwk1JzWVMJwlLnP3eIY/QKCGoZHW074ZECu4Yewx7KOiVA1M+Ym1fiRdlzrD883zzuxq96t0jH6YCjQdyCPoBa4CTcededKAYLhtVkxRRgEI2LQ5LaO1U/VCVC9vEn4ICp1/x6KPimsAfRqJu72OpzRqd7TY15zpvFLnWddmEjJFMz1H1603YfeNwXmRLTLJCq6u7k6Dhkqm32arIzPKhIj/geRMCPUepCpIx6W5HmDtF3BzHtf2lsXcaOO88dcbm2M3YshdMTw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(376002)(136003)(366004)(396003)(346002)(36756003)(8936002)(8676002)(9746002)(86362001)(9786002)(5660300002)(2906002)(2616005)(66476007)(26005)(186003)(478600001)(83380400001)(6666004)(426003)(316002)(66556008)(66946007)(110136005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 4N8kUfnsfWcEXydYPxXDlbcxKpC51OOqe84qyKhoKaVVD8pTNiqilnJ2kJ6YP1oYAac5sUWuWM7OzNSoFBhrl97w602ZKD3qcujUP6ZPq/HWBZFaT48r3f1pkswj+BcGfVOSnz6Y85lpeF8jTSBwpcI6oPIRWsZNFpOmYkCotA5jggX3vJS+LGiIGv9qxzDdx+erl9NIYPxzdHARbp/0KGcGhKsnPAxyIDE5nrkdUDdUoC6WrC0LPhYX/EV5Xta8jNRryOxiFwn+/c6WQF94MPeqnPxG7mCOQglWJiRXS5bA9ceUEN+U+WMHw0U/WS2Pdl337XPdu97zDbez5e1agQKyYQo4nE9DOckU/VpVJbqjE2k74RUO9R7rRPTewIVo39VKPfENXpICnNgnRsJJ6dihbAUWXuwbnGwgwxKRZBiL4N2T0ca2tVzP4D0E6o3Zi1rwivY0E4BLfQoIKXDdLn+5mKa61PH+sHdaOCJP0s8Z6qVqGpEYUwRJXqkyky8KeYmd2vQocZsBVxgdfuC4RUkXhF/xIN+D9segRe8cUhs6S+RVtnJok66v39GYosykjtQxBv27iXUce8NhzJSP79pY96+7O/ztyeNJ80RcrG/a9XGiNgiJSv51nJaJy0WYSVmIDU8R0TqMVsWWGUMshQ== X-MS-Exchange-CrossTenant-Network-Message-Id: 35895dcd-d249-447f-f1a1-08d85123be1e X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:03.2696 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wBe5++c2s7iQn7Icqa3DKTcjxvek58fffVAMH8StjAJdW76eDlx6hDfGPaJKjIFk X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2856 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259330; bh=2yDF311buzQpmn8URkhHzX1v09sc+kqfHKxyvPUL3Cc=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:Subject: Date:Message-ID:In-Reply-To:References:Content-Transfer-Encoding: Content-Type:X-ClientProxiedBy:MIME-Version: X-MS-Exchange-MessageSentRepresentingType:X-Originating-IP: X-MS-PublicTrafficType:X-MS-Office365-Filtering-Correlation-Id: X-MS-TrafficTypeDiagnostic:X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers:X-MS-Exchange-SenderADCheck: X-Microsoft-Antispam:X-Microsoft-Antispam-Message-Info: X-Forefront-Antispam-Report:X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=oMQM2I6IYy1SLyP1oPyz4PdLcN2lqdLGhRcCjwqlRTJ8fqiv14CQK4lHVw0ppFnST BD624tCQDSnJ2aiyRadFXEBP9MiZ3z1MJCElgYkUQDsuiUdCu6jwRRbU08OLEeFcNQ WBjwD41oBWohJ1PurZ5E1dxB4EuAalWWsru14HehmMvhFg/IiUFpXe0mOELO5HvzQR fhif41EQZ0j2G0tfNk8TyMOuZaDATuFlGnvtQMUa78LDYgPN7RCSqfWgv9UNIzzgal y3Qc7Qx074dr6EvBCEHZgPAZfFxdSpQ+TyGHKeOM4uG6/JD1lHWiuciopmD3SnB9bY 4mTZXYnLsnDDA== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org This driver always uses a DMA array made up of PAGE_SIZE elements, so just use ib_umem_num_dma_blocks(). Since rdma_for_each_dma_block() always iterates exactly ib_umem_num_dma_blocks() there is no need for the early exit check in build_user_pbes(), delete it. Signed-off-by: Jason Gunthorpe --- drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c index 933b297de2ba86..1fb8da6d613674 100644 --- a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c +++ b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c @@ -810,13 +810,12 @@ static int ocrdma_build_pbl_tbl(struct ocrdma_dev *dev, struct ocrdma_hw_mr *mr) return status; } -static void build_user_pbes(struct ocrdma_dev *dev, struct ocrdma_mr *mr, - u32 num_pbes) +static void build_user_pbes(struct ocrdma_dev *dev, struct ocrdma_mr *mr) { struct ocrdma_pbe *pbe; struct ib_block_iter biter; struct ocrdma_pbl *pbl_tbl = mr->hwmr.pbl_table; - int pbe_cnt, total_num_pbes = 0; + int pbe_cnt; u64 pg_addr; if (!mr->hwmr.num_pbes) @@ -831,13 +830,8 @@ static void build_user_pbes(struct ocrdma_dev *dev, struct ocrdma_mr *mr, pbe->pa_lo = cpu_to_le32(pg_addr); pbe->pa_hi = cpu_to_le32(upper_32_bits(pg_addr)); pbe_cnt += 1; - total_num_pbes += 1; pbe++; - /* if done building pbes, issue the mbx cmd. */ - if (total_num_pbes == num_pbes) - return; - /* if the given pbl is full storing the pbes, * move to next pbl. */ @@ -856,7 +850,6 @@ struct ib_mr *ocrdma_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 len, struct ocrdma_dev *dev = get_ocrdma_dev(ibpd->device); struct ocrdma_mr *mr; struct ocrdma_pd *pd; - u32 num_pbes; pd = get_ocrdma_pd(ibpd); @@ -871,8 +864,8 @@ struct ib_mr *ocrdma_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 len, status = -EFAULT; goto umem_err; } - num_pbes = ib_umem_page_count(mr->umem); - status = ocrdma_get_pbl_info(dev, mr, num_pbes); + status = ocrdma_get_pbl_info( + dev, mr, ib_umem_num_dma_blocks(mr->umem, PAGE_SIZE)); if (status) goto umem_err; @@ -888,7 +881,7 @@ struct ib_mr *ocrdma_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 len, status = ocrdma_build_pbl_tbl(dev, &mr->hwmr); if (status) goto umem_err; - build_user_pbes(dev, mr, num_pbes); + build_user_pbes(dev, mr); status = ocrdma_reg_mr(dev, &mr->hwmr, pd->id, acc); if (status) goto mbx_err; From patchwork Fri Sep 4 22:41:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758593 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 477CF13B1 for ; Fri, 4 Sep 2020 22:42:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 26ABA20FC3 for ; Fri, 4 Sep 2020 22:42:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="cRhECy2G" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726406AbgIDWmR (ORCPT ); Fri, 4 Sep 2020 18:42:17 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:30453 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728140AbgIDWmL (ORCPT ); Fri, 4 Sep 2020 18:42:11 -0400 Received: from hkpgpgate102.nvidia.com (Not Verified[10.18.92.77]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:07 +0800 Received: from HKMAIL103.nvidia.com ([10.18.16.12]) by hkpgpgate102.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:07 -0700 X-PGP-Universal: processed; by hkpgpgate102.nvidia.com on Fri, 04 Sep 2020 15:42:07 -0700 Received: from HKMAIL104.nvidia.com (10.18.16.13) by HKMAIL103.nvidia.com (10.18.16.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:07 +0000 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.107) by HKMAIL104.nvidia.com (10.18.16.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:07 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DFL0LoR3kfn73DQGrC5NheUK8khfJN+g+j96yvVptJ7rhWpFKyvu8OTrhzxTQfe0q6pTDnpjXONFt4173zfhnQIWHpg7khJ4kDmkdxx9+B06mdwUJxBoJ4a5cnsYFa27eDyQFvEgnujcQtAc7mPFEMlxqUKBAW88eX9hKgS9g/Gg99Q93hTRlatI0BqyRYfUgxRZ+HxSGURTm6EASaCE0B5td5G84wkZrKzZakEWsnXZ65L48vo+t4XThUqQIOt2WpGVQrJvQljRQXTQa9V2ak1KfqkOaexu2p0vYd5MQwNQbb55xgTo2XMAp3kJU37UFMCsYNVWrW3acqM7WY++Kg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+Dn++izHGouDs5Cs8xCtu2rE36czqn7AL0Zwc4kUV6Q=; b=D6ARxjc3Fm4CtkqlBEXmSBk3fw8YM+kkxo9TBuN0Wt2JLBx2qZ0/ur50MZTiA23EocipCg+ObY9d+LsbsWP2SdbzjQL6jaZufy9T1bNr5ZdBw45hYmtNx+K4XKeZp15MpumJH9G2KHt4sHRiYdBDuG9EjRaqPSnk6UZuJ6/zZ8J84l3VBUAYAEH+VEDkwe/Tnt63sOXI+MOXK+HknZXeFa49yICWZTKRCUCHBfrn3FxcXTg27UTcdZae9QVm7bvDj+jwbvTO7bGflfGiYH7FJ6ofnN0lT4krtzloxm18Au5cOFzQgvaQnQn0g0oZH4Btg3dW758IBsogQ2t4BI0T4Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: vmware.com; dkim=none (message not signed) header.d=none;vmware.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2806.namprd12.prod.outlook.com (2603:10b6:a03:70::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3326.22; Fri, 4 Sep 2020 22:42:05 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:04 +0000 From: Jason Gunthorpe To: Adit Ranadive , Doug Ledford , , VMware PV-Drivers Subject: [PATCH v2 14/17] RDMA/pvrdma: Use ib_umem_num_dma_blocks() instead of ib_umem_page_count() Date: Fri, 4 Sep 2020 19:41:55 -0300 Message-ID: <14-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: MN2PR20CA0043.namprd20.prod.outlook.com (2603:10b6:208:235::12) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by MN2PR20CA0043.namprd20.prod.outlook.com (2603:10b6:208:235::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.15 via Frontend Transport; Fri, 4 Sep 2020 22:42:02 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP9-001vCp-8r; Fri, 04 Sep 2020 19:41:59 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5fe2ca13-ae04-4e4e-9d32-08d85123bdfe X-MS-TrafficTypeDiagnostic: BYAPR12MB2806: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1303; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: wELI5z5ai1Z+Dn42gbt8DujEayKT4bt1upaOFaNiDOoL7MJ6sXQa08PzacErlEBvZ8LyyAto6TlMi8Hzwq1EzFpU/9XlPvyOd9oX1TeBTKHMqpllBntxdsS/LjykK6s2FaJqxU/KfZg1cZYVR0j/YLmaGe7ZiRwuD+jxvgvHrXGmRHQQ+z+9KethZyFNUmriaY2y3+RqCnChvDkidtYy8dGtoHw/+UAzf0SZ4+HpKeQxUZQwYZuazT3sGKGZq3mEOnZN6+PKCogem3iZ84ysCTFNF3p8wWDMvjSB7y8+k1ARoBXuQLp2m4pnmIw/pWw7Du3Jhfc8Y95HkFPv011yrg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(366004)(376002)(39860400002)(396003)(136003)(346002)(66556008)(26005)(66476007)(6666004)(110136005)(8676002)(186003)(316002)(83380400001)(426003)(2616005)(9786002)(478600001)(36756003)(9746002)(86362001)(8936002)(5660300002)(2906002)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: MBxUqWNFD/WinrG16vacABqSlLR0djgUrxJCcFxIvq58BoKnJy+UDyCh4ZElRuzFdvqdaslEkeD/hNLTVRq1ip6gj+E9MuZ2nGNsW8IIslOFYFP+EjfjWhL3BXzOrwUQWwYzfqSVqNWOLmQpJE8UnnlzpvNJzwXS8Z8Ntz2BAjhGVaGtfmm9lQfTHIMGsuY3y0BI+mPmUFoX4Sv9rFruT9fjWqjJVt0LWIgSnWCFhh/IjkXgIUktCcp+0ISZ5jilv2PKll7DzgX/yh3byWPzxRFp3yny98Veh9MG0hBHnaHG//FByCR7l0Bm937Ex0rxtIFom94RY4TDbP48J40/8L6axe3bJEezlLd6cV5AVtKuLKTvuTlEr9BpeoTOebfkVEOZZyphGowX1Z+sS+pzYGORxZTlN+mChV2CDOJs0vTe0PjPxtxF9hslquWbF49syURhZi06Y4HvOF41VeQf7q1uQ6i2UKkCzE8nEOpdNPwwBdIoFkzOvCwGmWHOQtzlVX9CWY1TSrHUTMVccIOHfyvrl2uLU7f1TcBlbBym2OsoW/Ggp1J5tBzsLjWeXv1VEcJnxwOL5eHqEE+JTfZX778ZpLI429v721hZs4p1BR59ArdUDy9571kGS3o03kgeFdzpXWb/YHUryJE+xo24BA== X-MS-Exchange-CrossTenant-Network-Message-Id: 5fe2ca13-ae04-4e4e-9d32-08d85123bdfe X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:02.9608 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: h7dys09Vol4786RABa4Xz3h8L2fbsyXw/2RqFA25T+/+abbnYhp8WC4fEDsR1WAi X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2806 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259327; bh=JZvu/USViSfOQ9PQd/NNfFOW1LUbomcaYRCUiZt1Dm4=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:Subject: Date:Message-ID:In-Reply-To:References:Content-Transfer-Encoding: Content-Type:X-ClientProxiedBy:MIME-Version: X-MS-Exchange-MessageSentRepresentingType:X-Originating-IP: X-MS-PublicTrafficType:X-MS-Office365-Filtering-Correlation-Id: X-MS-TrafficTypeDiagnostic:X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers:X-MS-Exchange-SenderADCheck: X-Microsoft-Antispam:X-Microsoft-Antispam-Message-Info: X-Forefront-Antispam-Report:X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=cRhECy2GF7E3nOWOCgFu5shxEQvLafNCqFHXF7yZI21rizR5vJUA1tbkd7/LqIkL3 PkeMVfnd0gMwb0i+XxQKh1e/0D5YYVvRf3fHTq8z3rrzcI6onGR6WauW/L78tacOxr /VquA3YcxXGpxnPdL1sDjkBmJJzqaxDCOuBXPtlN+PFW3Mp4Q1ki1qoI5BxnXECktJ UAO7uv0gfk3vPiiTW0fxRqYvk1HlswtBpis2rg7O2TL0uT8yMli8gUg6PwEibbj1/3 0VbXilx8/Q8fFUdlZ/OKFBKUsI9cORFC8T7VnyosIYLcCajfUKl59HnstS+d3JE7vx nizRBn7SQ7WVQ== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org This driver always uses PAGE_SIZE. Signed-off-by: Jason Gunthorpe --- drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 2 +- drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c | 6 ++++-- drivers/infiniband/hw/vmw_pvrdma/pvrdma_srq.c | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c index 01cd122a8b692f..ad7dfababf1fe1 100644 --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c @@ -142,7 +142,7 @@ int pvrdma_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, goto err_cq; } - npages = ib_umem_page_count(cq->umem); + npages = ib_umem_num_dma_blocks(cq->umem, PAGE_SIZE); } else { /* One extra page for shared ring state */ npages = 1 + (entries * sizeof(struct pvrdma_cqe) + diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c index 9a8f2a9507be07..8a385acf6f0c42 100644 --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c @@ -298,9 +298,11 @@ struct ib_qp *pvrdma_create_qp(struct ib_pd *pd, goto err_qp; } - qp->npages_send = ib_umem_page_count(qp->sumem); + qp->npages_send = + ib_umem_num_dma_blocks(qp->sumem, PAGE_SIZE); if (!is_srq) - qp->npages_recv = ib_umem_page_count(qp->rumem); + qp->npages_recv = ib_umem_num_dma_blocks( + qp->rumem, PAGE_SIZE); else qp->npages_recv = 0; qp->npages = qp->npages_send + qp->npages_recv; diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_srq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_srq.c index f60a8e81bddddb..6fd843cff21e70 100644 --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_srq.c +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_srq.c @@ -152,7 +152,7 @@ int pvrdma_create_srq(struct ib_srq *ibsrq, struct ib_srq_init_attr *init_attr, goto err_srq; } - srq->npages = ib_umem_page_count(srq->umem); + srq->npages = ib_umem_num_dma_blocks(srq->umem, PAGE_SIZE); if (srq->npages < 0 || srq->npages > PVRDMA_PAGE_DIR_MAX_PAGES) { dev_warn(&dev->pdev->dev, From patchwork Fri Sep 4 22:41:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758611 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A8C9F13B1 for ; Fri, 4 Sep 2020 22:42:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 84B6E20C56 for ; Fri, 4 Sep 2020 22:42:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="Vxzi2AtJ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728061AbgIDWmd (ORCPT ); Fri, 4 Sep 2020 18:42:33 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:50823 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726441AbgIDWmZ (ORCPT ); Fri, 4 Sep 2020 18:42:25 -0400 Received: from hkpgpgate102.nvidia.com (Not Verified[10.18.92.9]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:12 +0800 Received: from HKMAIL103.nvidia.com ([10.18.16.12]) by hkpgpgate102.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:12 -0700 X-PGP-Universal: processed; by hkpgpgate102.nvidia.com on Fri, 04 Sep 2020 15:42:12 -0700 Received: from HKMAIL102.nvidia.com (10.18.16.11) by HKMAIL103.nvidia.com (10.18.16.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:11 +0000 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.170) by HKMAIL102.nvidia.com (10.18.16.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:11 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fm+DoXtB5UqxdPZONo1QRXRt3Hb+0CjYq/v4NuL59NtxoYDli/DBBbDKnO8lRYcH9Gi5djCxtiLldhAxBZvby1K266ijSyXfxF+ug8WoJaWYupsoFBcNQ1EnrpWaqP3JSYmiwndMDYi/HGhJ2to1a8W2lPkcNFXieqKvaz79eyNBAvX+WHt1OWm3kX8xhn7vFirRQ/hnYnMk48NlBmbQ8MpZvD2Wxk3qc4v7t3Sqbl7EsiQqMBH6GVH4l6ZNAyYJ8fpNQAsaRtWn/q2UuCg34WWc7iueiC3bUF0yOw+Z7mRHJQ/7fY8LmnUAetmstjhwuzFSGhbvOJvH4rhc3eT10Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Z4fDVGjnMAWcxyoiTZR7kgvr7HA7uBbiWcbXtujeQEk=; b=LvMrkM/RAhfoeF+5CBrSxd6kdSN+q5wv/oIS2JhI4cZK8OZRJULpyG0DE2Gvi51eifrbpvGBgu9HYLhCEOL8nPVZO70bKUrR7Nu12BUD2f64D0ljG1r417qoVFTJyFtESf6Rtkt5g7X5GS47xokvGSUC53zBdGvESPpQ3PJDu2fTCXuB7hvkU85Z905S0y6s/i9Wr368vwH5uxvjOsDyzilxVGWdjst1H8i82ntS0msjwQB26HTo92Aui/ZdA4JFe5S9U0tcKeu860jPY/eFVw37KbvciYmduafwf1Wfa/IVKMKm63VT71sxqkX91aaoUNyMc9ogM1bTT+Nja93GSg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2856.namprd12.prod.outlook.com (2603:10b6:a03:136::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.16; Fri, 4 Sep 2020 22:42:08 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:08 +0000 From: Jason Gunthorpe To: Doug Ledford , , Yishai Hadas Subject: [PATCH v2 15/17] RDMA/mlx4: Use ib_umem_num_dma_blocks() Date: Fri, 4 Sep 2020 19:41:56 -0300 Message-ID: <15-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR02CA0014.namprd02.prod.outlook.com (2603:10b6:207:3c::27) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by BL0PR02CA0014.namprd02.prod.outlook.com (2603:10b6:207:3c::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3326.20 via Frontend Transport; Fri, 4 Sep 2020 22:42:04 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP9-001vCt-9a; Fri, 04 Sep 2020 19:41:59 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d0f6e2e6-a080-40d1-ca42-08d85123bf12 X-MS-TrafficTypeDiagnostic: BYAPR12MB2856: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:785; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: nFG6X/fwZzxzn+UZtqI9I7i8+k1FpEJy4iMHHzMd4cZt4isnqJP9Bh/v8cdO+wOwlUqHFssLqySWj2/9btoacNDc4CL3z7NnnCEG5ha8oF2UqVtRemLKphGtD/xRWYDW3RgJKrOSfjOdunl2P150WcGTMKCnotA4Sy4TwGk+f+oaccVLbvHeWW4tn/uFpN2v0+E+06hiEbuwffRlrJmnR0KP3BAxP+G7uz4H3hC/TnykK5ztNKxbssuWy5QIp717U8TekYWVSjRvYYjaXywOGzjVJ0HGx/hg2Oc+gn7KBhrnIfmfIMrVj+LbiLXmEO6E2askZDbBNZNuHvHCHHLAiQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(376002)(136003)(366004)(396003)(346002)(36756003)(6636002)(8936002)(8676002)(9746002)(86362001)(9786002)(5660300002)(2906002)(2616005)(66476007)(26005)(186003)(478600001)(83380400001)(6666004)(426003)(316002)(66556008)(66946007)(110136005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: Qon1vU6WnTgFjIn7sb0ytkrQmdLwH8HH6mmptkhquImsGa/MQZfdg5ikZMIg1atekUWj8zALIf26aQDHzcUTPgBAEP4ixrcTqeDdPCRFcZfoZx/5ydE+wcFg1pBw9pLQEaDqOgx2dqfe4rQg0K8a2+7IlW+5Tpb8koagGnfaiFrS60ofObGsWSWyVPD8lbVTptrKCyy1z9leKOcshqelmQUC3Z2M//NfjY2LqcUK+Q9RzhGwDVtyL7Ik+Q+/aQBqJ/EpfwaI9DnTOgSWGh8AoHES97HjuJqJLI2vwCoxpDvUCWwY0dP5gGTLCuc3mrWEWLU4dgxsn8q0Sxmv8V92aipLYmWyzDZDqgm3ulXStZxM8k/s7nKmGUbGn5Qx/maGBrpCZre4FvZ3slRXNKqIY8QQaWLKIGqpoBeIEohXOG6xexflWxsCYlkFFQu380rbc0tGhF3Da4WtXaXmfABSh/rDMVQfAcR3GggLrAlotcLHyx+9llup+Dn8Acpvs4HXr0w+GulxGbMJPiRxhwcqH7ZWv6Yq21/ZlQAYbCFGsfYoQ4Rw7lqKtqU15ELbLxae5ugFXYZpYqWr/Eun6QVGhssWK1kodTB5tvn6T+NWY2vcWvYWb6tDulS1NSJmOj2BaTqwgwfJm1BAu2zgYG1bHA== X-MS-Exchange-CrossTenant-Network-Message-Id: d0f6e2e6-a080-40d1-ca42-08d85123bf12 X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:05.1266 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8gZnNGZhf3QxpsrUkE9cqzsVx/SOT/Q9c+KVOnTqs+pfxMpjrtxm+nAT5SYxSVEk X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2856 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259332; bh=VF+xCA0rps0FjffVsg48abVfPfJqo+u8RZxi1lPjvrg=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:Subject: Date:Message-ID:In-Reply-To:References:Content-Transfer-Encoding: Content-Type:X-ClientProxiedBy:MIME-Version: X-MS-Exchange-MessageSentRepresentingType:X-Originating-IP: X-MS-PublicTrafficType:X-MS-Office365-Filtering-Correlation-Id: X-MS-TrafficTypeDiagnostic:X-MS-Exchange-Transport-Forked: X-Microsoft-Antispam-PRVS:X-MS-Oob-TLC-OOBClassifiers: X-MS-Exchange-SenderADCheck:X-Microsoft-Antispam: X-Microsoft-Antispam-Message-Info:X-Forefront-Antispam-Report: X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=Vxzi2AtJkBACaqO4g02QX/PzPd1zGLfgO0vEaZzWsT2Hu7iI1qvyei9mAuDh9KUxF HdvCqoyAH+cFsiB9FEdWxrzOhMxwjqQiaCpIQeX+5avKtSFhaUoeKNFSpHpyAKWXks owP3JyVPEWxB9yRyv1xUy6f9CjhTajq6Kq/xaABaKSVwV1EZ7xsN8DqgDedNu9XEF+ yd2brvrJgQKsg5h89WWeMmTCySo1H6QzKTraKXYKqYLg1Mle8ji8TLGxtOi/YDA9Ra k0f2KECOI4f3BGfZ6so0KrB/TaBIO9iqDaPfhoEfmVoELxepUD0Xgy/RhtvMkPHfDj PcM5WEJ/03hnQ== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org For the calls linked to mlx4_ib_umem_calc_optimal_mtt_size() use ib_umem_num_dma_blocks() inside the function, it is just some weird static default. All other places are just using it with PAGE_SIZE, switch to ib_umem_num_dma_blocks(). As this is the last call site, remove ib_umem_num_count(). Signed-off-by: Jason Gunthorpe --- drivers/infiniband/core/umem.c | 12 ------------ drivers/infiniband/hw/mlx4/cq.c | 1 - drivers/infiniband/hw/mlx4/mr.c | 5 +++-- drivers/infiniband/hw/mlx4/qp.c | 2 -- drivers/infiniband/hw/mlx4/srq.c | 5 +++-- include/rdma/ib_umem.h | 2 -- 6 files changed, 6 insertions(+), 21 deletions(-) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index b57dbb14de8378..c1ab6a4f2bc386 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -350,18 +350,6 @@ void ib_umem_release(struct ib_umem *umem) } EXPORT_SYMBOL(ib_umem_release); -int ib_umem_page_count(struct ib_umem *umem) -{ - int i, n = 0; - struct scatterlist *sg; - - for_each_sg(umem->sg_head.sgl, sg, umem->nmap, i) - n += sg_dma_len(sg) >> PAGE_SHIFT; - - return n; -} -EXPORT_SYMBOL(ib_umem_page_count); - /* * Copy from the given ib_umem's pages to the given buffer. * diff --git a/drivers/infiniband/hw/mlx4/cq.c b/drivers/infiniband/hw/mlx4/cq.c index 8a3436994f8097..f62afc13d34885 100644 --- a/drivers/infiniband/hw/mlx4/cq.c +++ b/drivers/infiniband/hw/mlx4/cq.c @@ -149,7 +149,6 @@ static int mlx4_ib_get_cq_umem(struct mlx4_ib_dev *dev, struct ib_udata *udata, if (IS_ERR(*umem)) return PTR_ERR(*umem); - n = ib_umem_page_count(*umem); shift = mlx4_ib_umem_calc_optimal_mtt_size(*umem, 0, &n); err = mlx4_mtt_init(dev->dev, n, shift, &buf->mtt); diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c index 1d5ef0de12c950..bfb779b5eeb3d2 100644 --- a/drivers/infiniband/hw/mlx4/mr.c +++ b/drivers/infiniband/hw/mlx4/mr.c @@ -271,6 +271,8 @@ int mlx4_ib_umem_calc_optimal_mtt_size(struct ib_umem *umem, u64 start_va, u64 total_len = 0; int i; + *num_of_mtts = ib_umem_num_dma_blocks(umem, PAGE_SIZE); + for_each_sg(umem->sg_head.sgl, sg, umem->nmap, i) { /* * Initialization - save the first chunk start as the @@ -421,7 +423,6 @@ struct ib_mr *mlx4_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, goto err_free; } - n = ib_umem_page_count(mr->umem); shift = mlx4_ib_umem_calc_optimal_mtt_size(mr->umem, start, &n); err = mlx4_mr_alloc(dev->dev, to_mpd(pd)->pdn, virt_addr, length, @@ -511,7 +512,7 @@ int mlx4_ib_rereg_user_mr(struct ib_mr *mr, int flags, mmr->umem = NULL; goto release_mpt_entry; } - n = ib_umem_page_count(mmr->umem); + n = ib_umem_num_dma_blocks(mmr->umem, PAGE_SIZE); shift = PAGE_SHIFT; err = mlx4_mr_rereg_mem_write(dev->dev, &mmr->mmr, diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c index 2975f350b9fd10..31839f95d44af9 100644 --- a/drivers/infiniband/hw/mlx4/qp.c +++ b/drivers/infiniband/hw/mlx4/qp.c @@ -922,7 +922,6 @@ static int create_rq(struct ib_pd *pd, struct ib_qp_init_attr *init_attr, goto err; } - n = ib_umem_page_count(qp->umem); shift = mlx4_ib_umem_calc_optimal_mtt_size(qp->umem, 0, &n); err = mlx4_mtt_init(dev->dev, n, shift, &qp->mtt); @@ -1117,7 +1116,6 @@ static int create_qp_common(struct ib_pd *pd, struct ib_qp_init_attr *init_attr, goto err; } - n = ib_umem_page_count(qp->umem); shift = mlx4_ib_umem_calc_optimal_mtt_size(qp->umem, 0, &n); err = mlx4_mtt_init(dev->dev, n, shift, &qp->mtt); diff --git a/drivers/infiniband/hw/mlx4/srq.c b/drivers/infiniband/hw/mlx4/srq.c index 8f9d5035142d33..108b2d0118d064 100644 --- a/drivers/infiniband/hw/mlx4/srq.c +++ b/drivers/infiniband/hw/mlx4/srq.c @@ -115,8 +115,9 @@ int mlx4_ib_create_srq(struct ib_srq *ib_srq, if (IS_ERR(srq->umem)) return PTR_ERR(srq->umem); - err = mlx4_mtt_init(dev->dev, ib_umem_page_count(srq->umem), - PAGE_SHIFT, &srq->mtt); + err = mlx4_mtt_init( + dev->dev, ib_umem_num_dma_blocks(srq->umem, PAGE_SIZE), + PAGE_SHIFT, &srq->mtt); if (err) goto err_buf; diff --git a/include/rdma/ib_umem.h b/include/rdma/ib_umem.h index 0f1ab3d8f77dea..fa556da3337c86 100644 --- a/include/rdma/ib_umem.h +++ b/include/rdma/ib_umem.h @@ -74,7 +74,6 @@ static inline void __rdma_umem_block_iter_start(struct ib_block_iter *biter, struct ib_umem *ib_umem_get(struct ib_device *device, unsigned long addr, size_t size, int access); void ib_umem_release(struct ib_umem *umem); -int ib_umem_page_count(struct ib_umem *umem); int ib_umem_copy_from(void *dst, struct ib_umem *umem, size_t offset, size_t length); unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, @@ -92,7 +91,6 @@ static inline struct ib_umem *ib_umem_get(struct ib_device *device, return ERR_PTR(-EINVAL); } static inline void ib_umem_release(struct ib_umem *umem) { } -static inline int ib_umem_page_count(struct ib_umem *umem) { return 0; } static inline int ib_umem_copy_from(void *dst, struct ib_umem *umem, size_t offset, size_t length) { return -EINVAL; From patchwork Fri Sep 4 22:41:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758619 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EA960925 for ; Fri, 4 Sep 2020 22:42:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C0F0F20C56 for ; Fri, 4 Sep 2020 22:42:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="cD9hcz3T" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727850AbgIDWmj (ORCPT ); Fri, 4 Sep 2020 18:42:39 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:1977 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727986AbgIDWmf (ORCPT ); Fri, 4 Sep 2020 18:42:35 -0400 Received: from hkpgpgate102.nvidia.com (Not Verified[10.18.92.9]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:15 +0800 Received: from HKMAIL103.nvidia.com ([10.18.16.12]) by hkpgpgate102.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:15 -0700 X-PGP-Universal: processed; by hkpgpgate102.nvidia.com on Fri, 04 Sep 2020 15:42:15 -0700 Received: from HKMAIL102.nvidia.com (10.18.16.11) by HKMAIL103.nvidia.com (10.18.16.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:10 +0000 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.170) by HKMAIL102.nvidia.com (10.18.16.11) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:10 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kUGBtfuj3YZ9DyvpbTSbRgSzywcazdd/jfSEjnzo78hnoR/OVuKUfk6snbrI2R2SDANvHE8oVNWq6P2wJdbE0WPJxs3KZqeeREu42WPuqGxdUvNwqz8lbBeA+UmO2NTgwDRdYCgM/hspI4XXft8YQMqBAkbIoDS/yp0cqk4ABWOTDUqpQz84jEngkRh/4CkXUWjM7Fb59Hm9yhGXNBC/2eIdrqk70/ndZCCB22AROT8Kwpjdm3MkXgd8Bo9L2bpVRLLqwsWizozhGpnlBiopvqGR+05HnFSd5XQyGiJScUqUfnQbZhQUr2I69SWCwTYh9BfhmzZo0ShFQ5JJpq7pJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=78YmlDAOAM5hY3E0l2V+NhEdcQKv9sGUiu6TNcL9+zI=; b=bed2qLsjDAajN9ERKPmNtIV+VOtLNEyC4cKa3oP41bW7ZnvD+eiDOWHpcLC7H0wfVgWvaKZFyqjcFVjYyj8z4KXWZb9L04JZ1WTB6VHkD+gOwjRA7CMWmihe2q46Nc/F0+ydhdNu5ZLJyW+om+W1FWVGgdqv+ACdH2/6WT5UkBCdWF88NjKEH4kdd1tbozOd7lyLEX8l93MSyoV0/V534siupjNTIb5Bq0TlOFsh1N4j537MeQTrDUQ5G45YG9SDjn6Zyyl6EZVUtlYb6wQR9rRpNkDAaRewD7fyzXxXh+xow30xivozLjBNQ96C7uHwPJseatudvQQjjX28M3HDVQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: marvell.com; dkim=none (message not signed) header.d=none;marvell.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2856.namprd12.prod.outlook.com (2603:10b6:a03:136::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.16; Fri, 4 Sep 2020 22:42:07 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:07 +0000 From: Jason Gunthorpe To: Ariel Elior , "David S. Miller" , Doug Ledford , , "Jakub Kicinski" , , Michal Kalderon , Subject: [PATCH v2 16/17] RDMA/qedr: Remove fbo and zbva from the MR Date: Fri, 4 Sep 2020 19:41:57 -0300 Message-ID: <16-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR02CA0025.namprd02.prod.outlook.com (2603:10b6:207:3c::38) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by BL0PR02CA0025.namprd02.prod.outlook.com (2603:10b6:207:3c::38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.15 via Frontend Transport; Fri, 4 Sep 2020 22:42:04 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP9-001vCy-Ae; Fri, 04 Sep 2020 19:41:59 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 28088da4-d1be-43be-59b6-08d85123bed3 X-MS-TrafficTypeDiagnostic: BYAPR12MB2856: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:826; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: jc06xDVFUww6IhPSEU3nZjsnnQseJqmTHeDl1DaWKLSVGJfsQqmYeQM30BCWVILlaVPukoDbp1r/HvFWnRERacupXnxZ/n1PFoInO9YYOaLqDsqNMjv7XaOOkcWFpgrSlUVwoYHlQVKib6PB4NFBXMSurQTSCPQ6bBdUdS7+vozPDfrNWxnjs2/RrqtncgkGHb1Y0hgH5cA0LT2rjpaVZf+tDfaxxoWEm8plzyyzEj+1GEY3Kk4iS/FN7JQFqnfzIRXRCvbXl/WG0mhG9afTtsiZxn+RwX2m9An7Hm5ip/7lG7rEX4a7Lnkq2UWjZ9WuOGCRq0ywSfggxqBpubKAXg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(376002)(136003)(366004)(396003)(346002)(36756003)(8936002)(8676002)(9746002)(86362001)(9786002)(5660300002)(2906002)(2616005)(66476007)(26005)(186003)(478600001)(83380400001)(426003)(316002)(66556008)(66946007)(110136005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: lSoc13Kg7cJE9Are48SD8flm2DTPdznSOvG08g8x0wVGGjmhCopj+r1QPXZWDKAex/pA6DNBkCqr+wF0QwCngvgdjorVa4OyPRBIwpZGXMpKU6G50+lkkVtuwh6dbhnrCp5Ru3MdHAlJM4JA1kZpyxjxWB8XIKRMcxhYyC4ZB2ipFhZi70uTN/qdQ8165cjejf9nO6Jlr2hiYoUpChAb7JGJhPMmenHUMfV6BP4KQDyZ2FUhg91DJ8q6wkDQHmxcMLR2g3igz7r4aldI8sF/nLLXWPhwxN3+Kg8K2vMCYbTutd2BXDopplDJTmHiuLpOr33VEmgRP90kwrHHM9Cuqwnx2+9c6b3dYZhy3ZqeSJlCDSsxxRs+N7yuQHF4mEq7oZsRLx3/O00Kd044kmUx5MKzJA3CdVUlEt9o4TyXb8GbGr2jt9Hp48VL+gStfDYUOOz/EPDB4U4DBID3MwOL+uU+LFSetqRUnwN+VGD7niZ12z/2yir591S/HeRIP4m+DqxB6gBeshgFtvvkrl47AhY9n0AELKp3shoCElM5SYRvVqevS9Zw6Bd2n38SRvFDGK/VZWxIQaFBep79WDR1t46RRGxL5b6k7ZL8CirgIGaJI0B9Wjg64wUjAuiyKjaogbeghhy9wNIarnLSOSj90A== X-MS-Exchange-CrossTenant-Network-Message-Id: 28088da4-d1be-43be-59b6-08d85123bed3 X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:04.4400 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +G3ua4eC8iYqrTD7siJnDS1QWNdrbF6D8SRM42Oi5zduqJ5A4QV1SYim99CyT2vy X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2856 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259335; bh=UOra5+wekr5q0qfF4XyYZ8V/hit7Txzfrc8C63+hk0g=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:Subject: Date:Message-ID:In-Reply-To:References:Content-Transfer-Encoding: Content-Type:X-ClientProxiedBy:MIME-Version: X-MS-Exchange-MessageSentRepresentingType:X-Originating-IP: X-MS-PublicTrafficType:X-MS-Office365-Filtering-Correlation-Id: X-MS-TrafficTypeDiagnostic:X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers:X-MS-Exchange-SenderADCheck: X-Microsoft-Antispam:X-Microsoft-Antispam-Message-Info: X-Forefront-Antispam-Report:X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=cD9hcz3TI7B7QmhqnuubxyMpOXJxcvhF3OpknPFnFCi+KArfeul67DrjuEPJQaEl9 lApckQdQoiGgOqIFelSpbeLL8iBezdMLK/AEabGey0a4GZM5f32fT/nN1048gTAdX4 I3L07OCDUD7N3DKKIcWafTig4d4w9NRMOQC1QVK/c3vMjSHB2ICMgqwD7PeBUpkuqP FlDyiqZj312CRRZ/Imz7AwkYoWvh6opkozS1Wn0S9zn/o9CmRFOW5/qVu24FPwUOyf ++1MgXz5SQ3WS5F6YjK0t4/3Is0tTg6aXQZohVMpH7Vyd9Z/PCGLOfHtmOc43lS8u5 3iH/jX8f7DCmg== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org zbva is always false, so fbo is never read. A 'zero-based-virtual-address' is simply IOVA == 0, and the driver already supports this. Signed-off-by: Jason Gunthorpe Acked-by: Michal Kalderon  --- drivers/infiniband/hw/qedr/verbs.c | 4 ---- drivers/net/ethernet/qlogic/qed/qed_rdma.c | 12 ++---------- include/linux/qed/qed_rdma_if.h | 2 -- 3 files changed, 2 insertions(+), 16 deletions(-) diff --git a/drivers/infiniband/hw/qedr/verbs.c b/drivers/infiniband/hw/qedr/verbs.c index 278b48443aedba..cca69b4ed354ea 100644 --- a/drivers/infiniband/hw/qedr/verbs.c +++ b/drivers/infiniband/hw/qedr/verbs.c @@ -2878,10 +2878,8 @@ struct ib_mr *qedr_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 len, mr->hw_mr.pbl_two_level = mr->info.pbl_info.two_layered; mr->hw_mr.pbl_page_size_log = ilog2(mr->info.pbl_info.pbl_size); mr->hw_mr.page_size_log = PAGE_SHIFT; - mr->hw_mr.fbo = ib_umem_offset(mr->umem); mr->hw_mr.length = len; mr->hw_mr.vaddr = usr_addr; - mr->hw_mr.zbva = false; mr->hw_mr.phy_mr = false; mr->hw_mr.dma_mr = false; @@ -2974,10 +2972,8 @@ static struct qedr_mr *__qedr_alloc_mr(struct ib_pd *ibpd, mr->hw_mr.pbl_ptr = 0; mr->hw_mr.pbl_two_level = mr->info.pbl_info.two_layered; mr->hw_mr.pbl_page_size_log = ilog2(mr->info.pbl_info.pbl_size); - mr->hw_mr.fbo = 0; mr->hw_mr.length = 0; mr->hw_mr.vaddr = 0; - mr->hw_mr.zbva = false; mr->hw_mr.phy_mr = true; mr->hw_mr.dma_mr = false; diff --git a/drivers/net/ethernet/qlogic/qed/qed_rdma.c b/drivers/net/ethernet/qlogic/qed/qed_rdma.c index a4bcde522cdf9d..baa4c36608ea91 100644 --- a/drivers/net/ethernet/qlogic/qed/qed_rdma.c +++ b/drivers/net/ethernet/qlogic/qed/qed_rdma.c @@ -1520,7 +1520,7 @@ qed_rdma_register_tid(void *rdma_cxt, params->pbl_two_level); SET_FIELD(flags, RDMA_REGISTER_TID_RAMROD_DATA_ZERO_BASED, - params->zbva); + false); SET_FIELD(flags, RDMA_REGISTER_TID_RAMROD_DATA_PHY_MR, params->phy_mr); @@ -1582,15 +1582,7 @@ qed_rdma_register_tid(void *rdma_cxt, p_ramrod->pd = cpu_to_le16(params->pd); p_ramrod->length_hi = (u8)(params->length >> 32); p_ramrod->length_lo = DMA_LO_LE(params->length); - if (params->zbva) { - /* Lower 32 bits of the registered MR address. - * In case of zero based MR, will hold FBO - */ - p_ramrod->va.hi = 0; - p_ramrod->va.lo = cpu_to_le32(params->fbo); - } else { - DMA_REGPAIR_LE(p_ramrod->va, params->vaddr); - } + DMA_REGPAIR_LE(p_ramrod->va, params->vaddr); DMA_REGPAIR_LE(p_ramrod->pbl_base, params->pbl_ptr); /* DIF */ diff --git a/include/linux/qed/qed_rdma_if.h b/include/linux/qed/qed_rdma_if.h index f464d85e88a410..aeb242cefebfa8 100644 --- a/include/linux/qed/qed_rdma_if.h +++ b/include/linux/qed/qed_rdma_if.h @@ -242,10 +242,8 @@ struct qed_rdma_register_tid_in_params { bool pbl_two_level; u8 pbl_page_size_log; u8 page_size_log; - u32 fbo; u64 length; u64 vaddr; - bool zbva; bool phy_mr; bool dma_mr; From patchwork Fri Sep 4 22:41:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11758621 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8AFBD112E for ; Fri, 4 Sep 2020 22:42:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6D57A20C56 for ; Fri, 4 Sep 2020 22:42:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="X5N7Ofbd" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728197AbgIDWmk (ORCPT ); Fri, 4 Sep 2020 18:42:40 -0400 Received: from nat-hk.nvidia.com ([203.18.50.4]:26704 "EHLO nat-hk.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726265AbgIDWmh (ORCPT ); Fri, 4 Sep 2020 18:42:37 -0400 Received: from hkpgpgate101.nvidia.com (Not Verified[10.18.92.9]) by nat-hk.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sat, 05 Sep 2020 06:42:22 +0800 Received: from HKMAIL104.nvidia.com ([10.18.16.13]) by hkpgpgate101.nvidia.com (PGP Universal service); Fri, 04 Sep 2020 15:42:22 -0700 X-PGP-Universal: processed; by hkpgpgate101.nvidia.com on Fri, 04 Sep 2020 15:42:22 -0700 Received: from HKMAIL103.nvidia.com (10.18.16.12) by HKMAIL104.nvidia.com (10.18.16.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 4 Sep 2020 22:42:17 +0000 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.168) by HKMAIL103.nvidia.com (10.18.16.12) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 4 Sep 2020 22:42:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iHDCPqJEC9MZgXtTIJRMcE9znxGahr4WqD1+mmgAI/DyDg27YWVUJwICUWBB18Et0mdqHAaBKAoPpuDQ3TxcBxvrYyrkdCCpQeOcYmRY2noFRECuJHmmwB0NmsA1r7jhQWfYWlSXLOZnWF7pLqgctpcOy9URuMipsqHyDXGJ8zWhnafb/ICoYusOyWjYdDFhXYNcbSj4d9ywVbxFX1+nsxBQsfslaMjl5idvN+Wwht+UBbJRyNp2xc0IPq0V3VuvQUuurnqX8x5LHPempVXrOSCqfwhvzEJuuD66eUtsf+qNBu0+v1GKgVXm324CwPExgC42NEPlXHgeBj+EtFUcuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D00Qzxhzhdr5BPLAZrw/Ap5G4iWy4TiReUTYKXLhA/Y=; b=fLMm9s4t3SyWR1PMK6nS/Zw58NuOYMstugUk5VNGANZG3ihO099OLqyBOO4MAHO2m/vWUgrktObhdOxh5DBP8+EWZeKK5z/sUpJHryKp31/ft/PvQifJsRUBJ433HFmQ3H0JPIG7953/DX/OKBJUD+k3beD4XzuIPjs1pzBjoVJRy0Odw5CZlvG5E0Mit+vguCw8Y2h+Fd52JZKiIFjhg1cNCluBpPiZblT0fJ0UXubBbccqsb7mfA/Pgr6nMicXcVGIghk8F1vPXg4x+5DqH+xH6CM/AxfcJzGjFMOgByBmRRn0zfuLKKbehYadP4QCODpsQsxtlj9Kn93IfTsGSw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Authentication-Results: broadcom.com; dkim=none (message not signed) header.d=none;broadcom.com; dmarc=none action=none header.from=nvidia.com; Received: from BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) by BYAPR12MB2856.namprd12.prod.outlook.com (2603:10b6:a03:136::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.16; Fri, 4 Sep 2020 22:42:10 +0000 Received: from BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9]) by BY5PR12MB3827.namprd12.prod.outlook.com ([fe80::445b:111e:a091:66f9%6]) with mapi id 15.20.3326.025; Fri, 4 Sep 2020 22:42:10 +0000 From: Jason Gunthorpe To: Devesh Sharma , Doug Ledford , , Selvin Xavier Subject: [PATCH v2 17/17] RDMA/ocrdma: Remove fbo from MR Date: Fri, 4 Sep 2020 19:41:58 -0300 Message-ID: <17-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> In-Reply-To: <0-v2-270386b7e60b+28f4-umem_1_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR02CA0022.namprd02.prod.outlook.com (2603:10b6:207:3c::35) To BY5PR12MB3827.namprd12.prod.outlook.com (2603:10b6:a03:1ab::16) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mlx.ziepe.ca (156.34.48.30) by BL0PR02CA0022.namprd02.prod.outlook.com (2603:10b6:207:3c::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3348.15 via Frontend Transport; Fri, 4 Sep 2020 22:42:06 +0000 Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kEKP9-001vD1-Bp; Fri, 04 Sep 2020 19:41:59 -0300 X-Originating-IP: [156.34.48.30] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: df0056ca-8376-4d57-47ec-08d85123c040 X-MS-TrafficTypeDiagnostic: BYAPR12MB2856: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2399; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ppcPRkj5iA/InNhv2FsM/AOKoqDJlsNEg6V9ne9GBSUAOVscIXjCkaJ5uVVbNMk5TpA3FWG70nwFt7uM6ojdjPlMPJPw4zpYfAWHmCtF4//y54RN/QkfPPlu71vQdzbl6cFTKZe4NNICy+pVjwchh/WJIXGirJcnIe7Lnfk86YnoAfxnqHlHil+NZ5Z9G/ix0JSmxZtVZ/2Xsy/Ho+Sx6UNk2Nr4tOzN88PiuvlhBbgP7z0bN9WqMgUKYZjq9LH8mYNCsN/35V3FVBDe+Q40bbaSjUjBNrSuTMjlqtmZ2Q/e1xf4RMxxgmAfwu0xF7S2R92jIAsheISTaXrcJt4xKg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB3827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(376002)(136003)(366004)(396003)(346002)(36756003)(8936002)(8676002)(9746002)(86362001)(9786002)(5660300002)(2906002)(2616005)(66476007)(26005)(186003)(478600001)(83380400001)(426003)(316002)(66556008)(66946007)(110136005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: Ntdgy/7KAj5wQrVvbdV1WhDajh3r8Ow0MQI1rQqqnvxrNmz3QLkRtc6aya/pm9iIi9QgBTBqEDMffvrvETo6Ls5BBLhGUBJQsBb3xDQUfuuHCau0HmqzPLVIQkj00SYYe01gJlVMYHFNCKgKLBJ/HUNs34m6Jm5HajBFo/VSh7y8NDRvorHEqWtRMh9Z5pWSnCij0JyncVUClEfwob/LCkFrkHgMVw2VNifyHcGyieC7oVYwt7j6mJrluCYFvrU3hm7X/M56485N39pJpTLU/I3TiLGRksaO23LH7/p2SgtrU9BColU0eEaeh8kj5r996JufRUcZ0uSKy33m+S7vX9gsp1FvStYdzex5iBJRzE8+ZybxzUyCkAwBAz4qhS1W6wWlBKR7ACMKI57uN9v0oOsMvQk63GuWMn6eLGt/fs/Zo0dJMXUL9pGv//nUDjqhQQROzPgPfA6tIZJJZeUMdHXRUsHz7lbxzEf59riTGO/2FYqBhPZNofj0a0toGwIiCheBqNuEBkY4ydyhXRG17AFB8O1UOzeJpGJSvrlbZJCIXkqoJcscyv12XRiGUH5XRbHFa4S52JAJcZ/bCrOXzpfsPIzm3gAF38ZdI9H1HlzFpDxUO3B9ycqqWmzdyrrvCZ/WjMJrCsfECTQ0r7w+Ew== X-MS-Exchange-CrossTenant-Network-Message-Id: df0056ca-8376-4d57-47ec-08d85123c040 X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2020 22:42:06.7906 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cqGgfj7UwowaC7O0SN8VgG7AgF3K5tL+eq3uzbbWyJSYrd/JWqdTBfmbAhohW8lD X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2856 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1599259342; bh=yWthnRm084FwnQf6BP4XrXIENEl6mNz2DAIec3GiXd0=; h=X-PGP-Universal:ARC-Seal:ARC-Message-Signature: ARC-Authentication-Results:Authentication-Results:From:To:Subject: Date:Message-ID:In-Reply-To:References:Content-Transfer-Encoding: Content-Type:X-ClientProxiedBy:MIME-Version: X-MS-Exchange-MessageSentRepresentingType:X-Originating-IP: X-MS-PublicTrafficType:X-MS-Office365-Filtering-Correlation-Id: X-MS-TrafficTypeDiagnostic:X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers:X-MS-Exchange-SenderADCheck: X-Microsoft-Antispam:X-Microsoft-Antispam-Message-Info: X-Forefront-Antispam-Report:X-MS-Exchange-AntiSpam-MessageData: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-OriginalArrivalTime: X-MS-Exchange-CrossTenant-FromEntityHeader: X-MS-Exchange-CrossTenant-Id:X-MS-Exchange-CrossTenant-MailboxType: X-MS-Exchange-CrossTenant-UserPrincipalName: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=X5N7OfbdgSzGJww4f7mWhOe4VtM6P8ThZozVTBNBd+o9Oxz3scknLd2hR8s0WjZ7o 5TbgQo1Ye8HywWtezdtHQQI8A9TFsLjGPmWYWTAYJTFIcPka7WLXjpEjIj9/jkLUlH apQXanwKa7N0M3nGi6vPfeRHiqREG7Uuxk7OFPwLbYM+whSYrEdJsLH7YEtA6/UAY8 D+b46z8LkEaPd8oiY/80Aww3vITikThQxqYA0gsKC5fFkpUhGLUsedivXZX7BcBAhY hjPbix2h97004LnqwBc3p7MEDQEWZ5tDUZm/csC7ddkLgl1IwWxydX2FXasFGRyspr hjX/zi6FCHJDg== Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org This is always the same value as IOVA masked by the page size, just use that clearer calculation directly. It is unclear of ocrdma hardware can actually support a true fbo, if so it could use a different algorithm to compute the best page size. Signed-off-by: Jason Gunthorpe --- drivers/infiniband/hw/ocrdma/ocrdma.h | 1 - drivers/infiniband/hw/ocrdma/ocrdma_hw.c | 5 +++-- drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 1 - 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/hw/ocrdma/ocrdma.h b/drivers/infiniband/hw/ocrdma/ocrdma.h index fcfe0e82197a24..5eb61c1100900d 100644 --- a/drivers/infiniband/hw/ocrdma/ocrdma.h +++ b/drivers/infiniband/hw/ocrdma/ocrdma.h @@ -185,7 +185,6 @@ struct ocrdma_hw_mr { u32 num_pbes; u32 pbl_size; u32 pbe_size; - u64 fbo; u64 va; }; diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c index e07bf0b2209a4c..18ed658f8dba10 100644 --- a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c +++ b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c @@ -1962,6 +1962,7 @@ static int ocrdma_mbx_reg_mr(struct ocrdma_dev *dev, struct ocrdma_hw_mr *hwmr, int i; struct ocrdma_reg_nsmr *cmd; struct ocrdma_reg_nsmr_rsp *rsp; + u64 fbo = hwmr->va & (hwmr->pbe_size - 1); cmd = ocrdma_init_emb_mqe(OCRDMA_CMD_REGISTER_NSMR, sizeof(*cmd)); if (!cmd) @@ -1987,8 +1988,8 @@ static int ocrdma_mbx_reg_mr(struct ocrdma_dev *dev, struct ocrdma_hw_mr *hwmr, OCRDMA_REG_NSMR_HPAGE_SIZE_SHIFT; cmd->totlen_low = hwmr->len; cmd->totlen_high = upper_32_bits(hwmr->len); - cmd->fbo_low = (u32) (hwmr->fbo & 0xffffffff); - cmd->fbo_high = (u32) upper_32_bits(hwmr->fbo); + cmd->fbo_low = (u32) (fbo & 0xffffffff); + cmd->fbo_high = (u32) upper_32_bits(fbo); cmd->va_loaddr = (u32) hwmr->va; cmd->va_hiaddr = (u32) upper_32_bits(hwmr->va); diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c index 1fb8da6d613674..3b98a3b3e2272d 100644 --- a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c +++ b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c @@ -870,7 +870,6 @@ struct ib_mr *ocrdma_reg_user_mr(struct ib_pd *ibpd, u64 start, u64 len, goto umem_err; mr->hwmr.pbe_size = PAGE_SIZE; - mr->hwmr.fbo = ib_umem_offset(mr->umem); mr->hwmr.va = usr_addr; mr->hwmr.len = len; mr->hwmr.remote_wr = (acc & IB_ACCESS_REMOTE_WRITE) ? 1 : 0;