From patchwork Wed Nov 6 13:49:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13864484 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24962D44D54 for ; Wed, 6 Nov 2024 11:50:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B28A18D0003; Wed, 6 Nov 2024 06:50:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AAF808D0001; Wed, 6 Nov 2024 06:50:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 927488D0003; Wed, 6 Nov 2024 06:50:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7131B8D0001 for ; Wed, 6 Nov 2024 06:50:50 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 2DD78161066 for ; Wed, 6 Nov 2024 11:50:50 +0000 (UTC) X-FDA: 82755502620.15.F9D029D Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf12.hostedemail.com (Postfix) with ESMTP id DADF84001B for ; Wed, 6 Nov 2024 11:50:32 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=EOk5Tzsm; spf=pass (imf12.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730893711; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9wohjECrItiTe3wGt73yRaTZkJ8cJH9lXeliHlLBqbI=; b=KqQ3p0qu4+CxC57mR0Qwep7E8QSlmjPynJqRWDmoNG2W2XFz7SfICk9wBOjQ6jRD1WugGE M5YHCL3Vd9nN9apWz18iCbNQX6jAEDfc7XsnO2UhYT8wnbQ2YIobOjlV3ueA/+hsqJdCXw f7NCRhEm4fc358rQiEi+zW+MijxuLr4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730893711; a=rsa-sha256; cv=none; b=Nf3c38c7/0sNirHpdvoFHXTqMKkDK0zppokSARMm4N8qHxPEZBzTn7QR1416XBiyDpZ6R4 ty0QLvX+99i7xtUlGIgzL0l5jWS7nsjsio5Ho4rvsui5zj2d3DYg7ihb//mEVvdqn7DYWE Jqn2SmBYtesU0mC/La/fZL7N43ea0Zg= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=EOk5Tzsm; spf=pass (imf12.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id F199A5C20FE; Wed, 6 Nov 2024 11:50:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A2D6EC4CECD; Wed, 6 Nov 2024 11:50:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730893847; bh=faZKR+GLvGy07GeNnto8v510/6+S1UMSrHjOqNZtBkg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EOk5TzsmzB3j83bvLGxj0ufqueYA0lsVRGTcymCHL1Yd7/MMCVyLZukaIlmwwKo4Z 0BkOp4PJNmruUsKpQYJkBmuU54A0Bzd5OxvpU3SLCT+VQsEYRg+5VdThT6JS3/WRoD 8LyryAOqBtUqrVX4OWL2ws1llPN/d7KiW9+UuTthAgpFsyNmMShr+KL+BRjmNv+bW3 xV3mNfREvaubJLLHfj3W0BFCohdhYZXywDWYWQyI810gSi2JmbwZ0Nta/ks04r6lJW EZ7CI+dwbz4fAvZgt/reVu/R5U/ONIiVf2VbxOY9iUqgFRYHsoyh0i43vIpdOnmd9y adYrKH6YKrniQ== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Leon Romanovsky , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Randy Dunlap Subject: [PATCH v2 14/17] RDMA/umem: Separate implicit ODP initialization from explicit ODP Date: Wed, 6 Nov 2024 15:49:42 +0200 Message-ID: <487c6208a36a3ccda6e0cea4716cbf50d5e4276d.1730892663.git.leon@kernel.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: DADF84001B X-Stat-Signature: d3yois1ojf4516eecizq1guznj8ftnjg X-HE-Tag: 1730893832-808722 X-HE-Meta: U2FsdGVkX1+BYGXvAJiaB8JLFWsoY0Zck7Cpn5AEhcA+JrJ/Ahq91s65Qjn0tr5YdrbxJ9vHaww/UNQXufY5+VpVNnKugimPQlrzP6FJDj1MJzJiUQabIgdEFOJuXWZm1SjdYl/6yR7g2Mm9BXYLNuDxwSUJEc3T63s7zieLJCNW80KdQrETs88lQNJjwqlQIcK3RuFrDBDDfLcNHgZx06t9AXzWxj05jyN2YmkLYEdT8ggwN5+/403p7iYMVQmpLJnbks7uR/xCYLct1BemOjeWhtE+13+dWxQ5zA07ly4y5ZKHLquaj+LvIyy6hdDuhSXCPykL+hDGtSsty5d0P1XCQrCguKvYYOSx810oA3UdK/P9a7ewXgW9mBNMJR7f3OXbx8jlBUp/hF25slkxMhcwHdxEVRJ8+i37wVMKliwSIu8WuCgwFE9qyPyZ+HYPUWDYYLTxhz25AzYB7hoBRdglWUPtCe7e3+rf+JEzoDIAPeG9gkrqqvORqVnj+2Kke3ixhIMaQ2uQhLI9USga+a30rmx+m8VS4aS0rxdTjsSLvUejhDc2HsJrE2WzzqIjSO/MuDxhYrYyimulZ2kizcFo5hcwYV5PDS9BHzRniU3kkFUFCDYaE3W1NZeLjHxlCKaQHWmQZs1hZt7e2qZUzxSxZ0mtL/G5vlmVjwoXBdQUu7KtLeysUFL6jFbNhjjc6hJR4HpvrEVPq9BbX6FNcfPPTEZQP3MYvXO75aJ9p5v5hVM7RznXlJuQIlHzLB4LXSUrrv88d+a3TQAe0+PwyeR8tYtzvGDHKlOjULhCu1hDeHKnvMNjCGdMmBSSw4pjcsCYLgpChyCS2cFBnYWYxY7O9RmYr4kTFr7kwv3yL8rkowdZY7yy8haDWIreiXSXve9vP/sIPcLmV5CAj6UmpwTEESeKgvXZQSaSA5uW7xebX7VJEgDDgJ5178qUsiLJjlhzVTKM8MeyeBdUP5l Rh9WQHtW 5ZSrllgpZ0TUFbN81KYppq+sTeJjsV1m8ZtQwD03JBP68uHsM3Wbz/NOwB+lrvzaSeDcXqjmiWsm+cfBpVx/HhErMbiujtZPi+V4D4RijQ+/93dSKj95x5ahYkAyOR9+iyUWZVAUI9r1hNwbKtmRnfimF1GQ686rTbHbWy5IekJ40tEO5z9Y583+ULEiQObN9T2Mgm2Xq1gort3ckNBDgVOBtorWKqVCl5/ZJr8gymUpmnBI9tUFNdVWtv+RAwAkiv/ZjysIX3uCceD/qC9Z5+HJFpyJMzDtD/3iS+L2edNW5DE60ta1q81NaK7ETJ3CGKBBTUaarXaAy7hjDqGEMV9IHEsBXboFg+jy7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky Create separate functions for the implicit ODP initialization which is different from the explicit ODP initialization. Signed-off-by: Leon Romanovsky --- drivers/infiniband/core/umem_odp.c | 91 +++++++++++++++--------------- 1 file changed, 46 insertions(+), 45 deletions(-) diff --git a/drivers/infiniband/core/umem_odp.c b/drivers/infiniband/core/umem_odp.c index 30cd8f353476..51d518989914 100644 --- a/drivers/infiniband/core/umem_odp.c +++ b/drivers/infiniband/core/umem_odp.c @@ -48,41 +48,44 @@ #include "uverbs.h" -static inline int ib_init_umem_odp(struct ib_umem_odp *umem_odp, - const struct mmu_interval_notifier_ops *ops) +static void ib_init_umem_implicit_odp(struct ib_umem_odp *umem_odp) +{ + umem_odp->is_implicit_odp = 1; + umem_odp->umem.is_odp = 1; + mutex_init(&umem_odp->umem_mutex); +} + +static int ib_init_umem_odp(struct ib_umem_odp *umem_odp, + const struct mmu_interval_notifier_ops *ops) { struct ib_device *dev = umem_odp->umem.ibdev; + size_t page_size = 1UL << umem_odp->page_shift; + unsigned long start; + unsigned long end; int ret; umem_odp->umem.is_odp = 1; mutex_init(&umem_odp->umem_mutex); - if (!umem_odp->is_implicit_odp) { - size_t page_size = 1UL << umem_odp->page_shift; - unsigned long start; - unsigned long end; - - start = ALIGN_DOWN(umem_odp->umem.address, page_size); - if (check_add_overflow(umem_odp->umem.address, - (unsigned long)umem_odp->umem.length, - &end)) - return -EOVERFLOW; - end = ALIGN(end, page_size); - if (unlikely(end < page_size)) - return -EOVERFLOW; - - ret = hmm_dma_map_alloc(dev->dma_device, &umem_odp->map, - (end - start) >> PAGE_SHIFT, - 1 << umem_odp->page_shift); - if (ret) - return ret; - - ret = mmu_interval_notifier_insert(&umem_odp->notifier, - umem_odp->umem.owning_mm, - start, end - start, ops); - if (ret) - goto out_free_map; - } + start = ALIGN_DOWN(umem_odp->umem.address, page_size); + if (check_add_overflow(umem_odp->umem.address, + (unsigned long)umem_odp->umem.length, &end)) + return -EOVERFLOW; + end = ALIGN(end, page_size); + if (unlikely(end < page_size)) + return -EOVERFLOW; + + ret = hmm_dma_map_alloc(dev->dma_device, &umem_odp->map, + (end - start) >> PAGE_SHIFT, + 1 << umem_odp->page_shift); + if (ret) + return ret; + + ret = mmu_interval_notifier_insert(&umem_odp->notifier, + umem_odp->umem.owning_mm, start, + end - start, ops); + if (ret) + goto out_free_map; return 0; @@ -106,7 +109,6 @@ struct ib_umem_odp *ib_umem_odp_alloc_implicit(struct ib_device *device, { struct ib_umem *umem; struct ib_umem_odp *umem_odp; - int ret; if (access & IB_ACCESS_HUGETLB) return ERR_PTR(-EINVAL); @@ -118,16 +120,10 @@ struct ib_umem_odp *ib_umem_odp_alloc_implicit(struct ib_device *device, umem->ibdev = device; umem->writable = ib_access_writable(access); umem->owning_mm = current->mm; - umem_odp->is_implicit_odp = 1; umem_odp->page_shift = PAGE_SHIFT; umem_odp->tgid = get_task_pid(current->group_leader, PIDTYPE_PID); - ret = ib_init_umem_odp(umem_odp, NULL); - if (ret) { - put_pid(umem_odp->tgid); - kfree(umem_odp); - return ERR_PTR(ret); - } + ib_init_umem_implicit_odp(umem_odp); return umem_odp; } EXPORT_SYMBOL(ib_umem_odp_alloc_implicit); @@ -248,7 +244,7 @@ struct ib_umem_odp *ib_umem_odp_get(struct ib_device *device, } EXPORT_SYMBOL(ib_umem_odp_get); -void ib_umem_odp_release(struct ib_umem_odp *umem_odp) +static void ib_umem_odp_free(struct ib_umem_odp *umem_odp) { struct ib_device *dev = umem_odp->umem.ibdev; @@ -258,14 +254,19 @@ void ib_umem_odp_release(struct ib_umem_odp *umem_odp) * It is the driver's responsibility to ensure, before calling us, * that the hardware will not attempt to access the MR any more. */ - if (!umem_odp->is_implicit_odp) { - mutex_lock(&umem_odp->umem_mutex); - ib_umem_odp_unmap_dma_pages(umem_odp, ib_umem_start(umem_odp), - ib_umem_end(umem_odp)); - mutex_unlock(&umem_odp->umem_mutex); - mmu_interval_notifier_remove(&umem_odp->notifier); - hmm_dma_map_free(dev->dma_device, &umem_odp->map); - } + mutex_lock(&umem_odp->umem_mutex); + ib_umem_odp_unmap_dma_pages(umem_odp, ib_umem_start(umem_odp), + ib_umem_end(umem_odp)); + mutex_unlock(&umem_odp->umem_mutex); + mmu_interval_notifier_remove(&umem_odp->notifier); + hmm_dma_map_free(dev->dma_device, &umem_odp->map); +} + +void ib_umem_odp_release(struct ib_umem_odp *umem_odp) +{ + if (!umem_odp->is_implicit_odp) + ib_umem_odp_free(umem_odp); + put_pid(umem_odp->tgid); kfree(umem_odp); }