From patchwork Tue Mar 5 11:18:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13582158 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8823C54798 for ; Tue, 5 Mar 2024 11:19:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F9176B00D3; Tue, 5 Mar 2024 06:19:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 779FA6B00D4; Tue, 5 Mar 2024 06:19:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CA2F6B00D6; Tue, 5 Mar 2024 06:19:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 48AF86B00D3 for ; Tue, 5 Mar 2024 06:19:36 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 24E4840C21 for ; Tue, 5 Mar 2024 11:19:36 +0000 (UTC) X-FDA: 81862739952.04.AA20C37 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf20.hostedemail.com (Postfix) with ESMTP id 095061C000E for ; Tue, 5 Mar 2024 11:19:33 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=gsVdVJp7; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf20.hostedemail.com: domain of leon@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709637574; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hDi5xvojWRIJEBMJj35ZWcDoY+2xWx5fHfKioYJwfDU=; b=OZjLmdPRBLOsxWFV8aGeKJ6EsOy4uj3/tPshoSzE8/F5eUqLhNWozy9sfH63qgdNZiVS2q UnANgBUzvb73q+7mG7Tgz9bWzVxjxbB5T0IC7lnBtXI4OTHm6q/qn97Y/g2IBHkFiN2K0X 0y4yqOt/f+jYfykqdk81/HltsLGCqlA= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=gsVdVJp7; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf20.hostedemail.com: domain of leon@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709637574; a=rsa-sha256; cv=none; b=OhouTh0A1s+ZTOQV0HiJNwCHHeZyQumiTLs98o1v+7Y4M0ZfhWu/WfBLaKvvtJPkiGDk1H ZYgSEA1Wv+QAqUlUylGbbNN/QjxqORai8XFh87RFpuMtTu/FU71PNNxhOS30wRY9WRqdta 6D0C9f52Io/fDBnxSJ+CxlZupoppc6A= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 5F8DFCE1924; Tue, 5 Mar 2024 11:19:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 69F55C433C7; Tue, 5 Mar 2024 11:19:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1709637570; bh=VVD1wxwJfNeGBkq+0kcqb+dcMkI1qAL6P7iY3YcEByk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gsVdVJp7zskmKa3d9GUOqTMHCQEwl4g4RFMzHw0F9Akw/QEvnxId23V4Xv7cqKot0 xRs5/G0U/Z7Cxg2a3sBK2FA0j5+Y6mELzKaeJH7JVfezh3qUANE+OQYzEdn7hfuV+X XJeje9A6U7pXEhmMNYneImOs3HGYtHRYiBpsw7oSs84WxpilNIp3CHJ3HJbi3eE6GJ FPHsYluwmY7jPEyxc/0NB2kONZAnoIeR3xQct8EiJ04Yki++9CJxnbmLNjNM0MUFM0 IYebXXRQLLepDcJStMjfViy39+U4doJWkp04ZNjWAC9PKT4QKxlaX4NwISPQ9mIAhA xPpJYEQf9FUtA== From: Leon Romanovsky To: Christoph Hellwig , Robin Murphy , Marek Szyprowski , Joerg Roedel , Will Deacon , Jason Gunthorpe , Chaitanya Kulkarni Cc: Leon Romanovsky , Jonathan Corbet , Jens Axboe , Keith Busch , Sagi Grimberg , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?utf-8?b?SsOpcsO0bWUgR2xp?= =?utf-8?b?c3Nl?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, kvm@vger.kernel.org, linux-mm@kvack.org, Bart Van Assche , Damien Le Moal , Amir Goldstein , "josef@toxicpanda.com" , "Martin K. Petersen" , "daniel@iogearbox.net" , Dan Williams , "jack@suse.com" , Zhu Yanjun Subject: [RFC RESEND 09/16] RDMA/core: Separate DMA mapping to caching IOVA and page linkage Date: Tue, 5 Mar 2024 13:18:40 +0200 Message-ID: <22f9bd2e33ca2ec2b3d3bbd4cbac55122991e02f.1709635535.git.leon@kernel.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 095061C000E X-Stat-Signature: xn9ge9a4p5t341s1idsdtu4ru5dzptwp X-Rspam-User: X-HE-Tag: 1709637573-223527 X-HE-Meta: U2FsdGVkX1/SHC8eAIflUkoCy7JDgmgzB1wezB6kBh9YH3T9FCxia1A2kzY200LspSFV9vWWyTO1CiXphXhqlGIjZriGgFawmynWUok5omrxLzRGim5xhDgSZmZId7uV+kmPn7bF2lviwBEwTrySCIB//p/SLYbPcDpeCukP4mCWTxV+ExxKKcI3JcFaetro2y0szBvRUAPNY67InjBJhdcg8hqaAt3eCtHk1jGOzjmAix+tlvfeJ8KXTXbO3+7xs1c40SVkcgssTy6kjbc3o4i8WcSKncgCyOskxNVGzge6f86sFn8Kp4siC9fXcSxE26U6x4RX6s3RKtkQZj7kqMdlLePoTJmGLbv7NtO25HQSjQDnelHS36pS9r5urhNxRd/UxUtL/dOZi1+tLBw9Pj5IT6X4RZ7JmahV1RTpRUFvJJkFfWuI6Fh9mwf2g7T0PkMXBZrJMnloHHun2m+JfZtIAJ1Jrjm3PX2/ShWXLhPtFk1K1cDvLLBmwPamLGIG1Ma3QElyKbjWuzf3qgKhTwAd6ro7BQGu6+HpJedacJE+TJjti9RtoG1WTdnbSR3XUgupffn3tEzyRcJqCrCZfpAAuyeCFF8bPcJXnvTudfdtmf6zhTweDmdpNH1cAQukHyEwXxCyTv5d31geRD3WADBKV+f1D+j89oUVnEbT6OKGjM5whGPBfvFR4IMj5wmansoOpIWMZsLSzm31fAnmTLuzUpmoQcruxPeFutpZZ8vxOaemSLNP3uo5isz1I9xOEbvHm077kVc7NHxowBWunqEksqM8UohJ49uEtqIjMfDfV/mhDBKX1bnuaUaB4xvyHG5f/7JsBU1QtxOwA3gI1IX9JYZEwvbKAJ/kz0nF0EcZ2Dd4UTchp/JSazNrjXWI7WfFuv5nSOkxdk2I3qrHUig8kg8kPkZ9QzvUEKNnuhkprQ9XOaxjkbnVedn02ue9X4Kj+VKTV4oBnAC3jBI r3Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky Reuse newly added DMA API to cache IOVA and only link/unlink pages in fast path. Signed-off-by: Leon Romanovsky --- drivers/infiniband/core/umem_odp.c | 57 ++---------------------------- drivers/infiniband/hw/mlx5/odp.c | 22 +++++++++++- include/rdma/ib_umem_odp.h | 8 +---- include/rdma/ib_verbs.h | 36 +++++++++++++++++++ 4 files changed, 61 insertions(+), 62 deletions(-) diff --git a/drivers/infiniband/core/umem_odp.c b/drivers/infiniband/core/umem_odp.c index 3619fb78f786..1301009a6b78 100644 --- a/drivers/infiniband/core/umem_odp.c +++ b/drivers/infiniband/core/umem_odp.c @@ -81,20 +81,13 @@ static inline int ib_init_umem_odp(struct ib_umem_odp *umem_odp, if (!umem_odp->pfn_list) return -ENOMEM; - umem_odp->dma_list = kvcalloc( - ndmas, sizeof(*umem_odp->dma_list), GFP_KERNEL); - if (!umem_odp->dma_list) { - ret = -ENOMEM; - goto out_pfn_list; - } umem_odp->iova.dev = dev->dma_device; umem_odp->iova.size = end - start; umem_odp->iova.dir = DMA_BIDIRECTIONAL; ret = ib_dma_alloc_iova(dev, &umem_odp->iova); if (ret) - goto out_dma_list; - + goto out_pfn_list; ret = mmu_interval_notifier_insert(&umem_odp->notifier, umem_odp->umem.owning_mm, @@ -107,8 +100,6 @@ static inline int ib_init_umem_odp(struct ib_umem_odp *umem_odp, out_free_iova: ib_dma_free_iova(dev, &umem_odp->iova); -out_dma_list: - kvfree(umem_odp->dma_list); out_pfn_list: kvfree(umem_odp->pfn_list); return ret; @@ -288,7 +279,6 @@ void ib_umem_odp_release(struct ib_umem_odp *umem_odp) mutex_unlock(&umem_odp->umem_mutex); mmu_interval_notifier_remove(&umem_odp->notifier); ib_dma_free_iova(dev, &umem_odp->iova); - kvfree(umem_odp->dma_list); kvfree(umem_odp->pfn_list); } put_pid(umem_odp->tgid); @@ -296,40 +286,10 @@ void ib_umem_odp_release(struct ib_umem_odp *umem_odp) } EXPORT_SYMBOL(ib_umem_odp_release); -/* - * Map for DMA and insert a single page into the on-demand paging page tables. - * - * @umem: the umem to insert the page to. - * @dma_index: index in the umem to add the dma to. - * @page: the page struct to map and add. - * @access_mask: access permissions needed for this page. - * - * The function returns -EFAULT if the DMA mapping operation fails. - * - */ -static int ib_umem_odp_map_dma_single_page( - struct ib_umem_odp *umem_odp, - unsigned int dma_index, - struct page *page) -{ - struct ib_device *dev = umem_odp->umem.ibdev; - dma_addr_t *dma_addr = &umem_odp->dma_list[dma_index]; - - *dma_addr = ib_dma_map_page(dev, page, 0, 1 << umem_odp->page_shift, - DMA_BIDIRECTIONAL); - if (ib_dma_mapping_error(dev, *dma_addr)) { - *dma_addr = 0; - return -EFAULT; - } - umem_odp->npages++; - return 0; -} - /** * ib_umem_odp_map_dma_and_lock - DMA map userspace memory in an ODP MR and lock it. * * Maps the range passed in the argument to DMA addresses. - * The DMA addresses of the mapped pages is updated in umem_odp->dma_list. * Upon success the ODP MR will be locked to let caller complete its device * page table update. * @@ -437,15 +397,6 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt, __func__, hmm_order, page_shift); break; } - - ret = ib_umem_odp_map_dma_single_page( - umem_odp, dma_index, hmm_pfn_to_page(range.hmm_pfns[pfn_index])); - if (ret < 0) { - ibdev_dbg(umem_odp->umem.ibdev, - "ib_umem_odp_map_dma_single_page failed with error %d\n", ret); - break; - } - range.hmm_pfns[pfn_index] |= HMM_PFN_STICKY; } /* upon success lock should stay on hold for the callee */ if (!ret) @@ -465,7 +416,6 @@ EXPORT_SYMBOL(ib_umem_odp_map_dma_and_lock); void ib_umem_odp_unmap_dma_pages(struct ib_umem_odp *umem_odp, u64 virt, u64 bound) { - dma_addr_t dma; int idx; u64 addr; struct ib_device *dev = umem_odp->umem.ibdev; @@ -479,15 +429,14 @@ void ib_umem_odp_unmap_dma_pages(struct ib_umem_odp *umem_odp, u64 virt, struct page *page = hmm_pfn_to_page(umem_odp->pfn_list[pfn_idx]); idx = (addr - ib_umem_start(umem_odp)) >> umem_odp->page_shift; - dma = umem_odp->dma_list[idx]; if (!(umem_odp->pfn_list[pfn_idx] & HMM_PFN_VALID)) continue; if (!(umem_odp->pfn_list[pfn_idx] & HMM_PFN_STICKY)) continue; - ib_dma_unmap_page(dev, dma, BIT(umem_odp->page_shift), - DMA_BIDIRECTIONAL); + ib_dma_unlink_range(dev, &umem_odp->iova, + idx * (1 << umem_odp->page_shift)); if (umem_odp->pfn_list[pfn_idx] & HMM_PFN_WRITE) { struct page *head_page = compound_head(page); /* diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index 5713fe25f4de..13d61f1ab40b 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -149,6 +149,7 @@ static void populate_mtt(__be64 *pas, size_t idx, size_t nentries, { struct ib_umem_odp *odp = to_ib_umem_odp(mr->umem); bool downgrade = flags & MLX5_IB_UPD_XLT_DOWNGRADE; + struct ib_device *dev = odp->umem.ibdev; unsigned long pfn; dma_addr_t pa; size_t i; @@ -162,12 +163,31 @@ static void populate_mtt(__be64 *pas, size_t idx, size_t nentries, /* Initial ODP init */ continue; - pa = odp->dma_list[idx + i]; + if (pfn & HMM_PFN_STICKY && odp->iova.addr) + /* + * We are in this flow when there is a need to resync flags, + * for example when page was already linked in prefetch call + * with READ flag and now we need to add WRITE flag + * + * This page was already programmed to HW and we don't want/need + * to unlink and link it again just to resync flags. + * + * The DMA address calculation below is based on the fact that + * RDMA UMEM doesn't work with swiotlb. + */ + pa = odp->iova.addr + (idx + i) * (1 << odp->page_shift); + else + pa = ib_dma_link_range(dev, hmm_pfn_to_page(pfn), 0, &odp->iova, + (idx + i) * (1 << odp->page_shift)); + WARN_ON_ONCE(ib_dma_mapping_error(dev, pa)); + pa |= MLX5_IB_MTT_READ; if ((pfn & HMM_PFN_WRITE) && !downgrade) pa |= MLX5_IB_MTT_WRITE; pas[i] = cpu_to_be64(pa); + odp->pfn_list[idx + i] |= HMM_PFN_STICKY; + odp->npages++; } } diff --git a/include/rdma/ib_umem_odp.h b/include/rdma/ib_umem_odp.h index 095b1297cfb1..a786556c65f9 100644 --- a/include/rdma/ib_umem_odp.h +++ b/include/rdma/ib_umem_odp.h @@ -17,15 +17,9 @@ struct ib_umem_odp { /* An array of the pfns included in the on-demand paging umem. */ unsigned long *pfn_list; - /* - * An array with DMA addresses mapped for pfns in pfn_list. - * The lower two bits designate access permissions. - * See ODP_READ_ALLOWED_BIT and ODP_WRITE_ALLOWED_BIT. - */ - dma_addr_t *dma_list; struct dma_iova_attrs iova; /* - * The umem_mutex protects the page_list and dma_list fields of an ODP + * The umem_mutex protects the page_list field of an ODP * umem, allowing only a single thread to map/unmap pages. The mutex * also protects access to the mmu notifier counters. */ diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index e71fa19187cc..c9e2bcd5268a 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -4160,6 +4160,42 @@ static inline void ib_dma_unmap_page(struct ib_device *dev, dma_unmap_page(dev->dma_device, addr, size, direction); } +/** + * ib_dma_link_range - Link a physical page to DMA address + * @dev: The device for which the dma_addr is to be created + * @page: The page to be mapped + * @offset: The offset within the page + * @iova: Preallocated IOVA attributes + * @dma_offset: DMA offset + */ +static inline dma_addr_t ib_dma_link_range(struct ib_device *dev, + struct page *page, + unsigned long offset, + struct dma_iova_attrs *iova, + dma_addr_t dma_offset) +{ + if (ib_uses_virt_dma(dev)) + return (uintptr_t)(page_address(page) + offset); + + return dma_link_range(page, offset, iova, dma_offset); +} + +/** + * ib_dma_unlink_range - Unlink a mapping created by ib_dma_link_page() + * @dev: The device for which the DMA address was created + * @iova: DMA IOVA properties + * @dma_offset: DMA offset + */ +static inline void ib_dma_unlink_range(struct ib_device *dev, + struct dma_iova_attrs *iova, + dma_addr_t dma_offset) +{ + if (ib_uses_virt_dma(dev)) + return; + + dma_unlink_range(iova, dma_offset); +} + int ib_dma_virt_map_sg(struct ib_device *dev, struct scatterlist *sg, int nents); static inline int ib_dma_map_sg_attrs(struct ib_device *dev, struct scatterlist *sg, int nents,