From patchwork Thu Sep 12 11:15:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13801923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08CCDEEB591 for ; Thu, 12 Sep 2024 11:16:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 689096B007B; Thu, 12 Sep 2024 07:16:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 638F66B0082; Thu, 12 Sep 2024 07:16:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 500DF6B0083; Thu, 12 Sep 2024 07:16:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 31F136B007B for ; Thu, 12 Sep 2024 07:16:08 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6B376161FA0 for ; Thu, 12 Sep 2024 11:16:07 +0000 (UTC) X-FDA: 82555831974.12.CD07A8F Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf01.hostedemail.com (Postfix) with ESMTP id D823C40006 for ; Thu, 12 Sep 2024 11:16:04 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=HnlDHW0a; spf=pass (imf01.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726139712; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=2DBCbfeL/jE2jfZvszJijVINvgiceTkfL13b/c+h6dQ=; b=ORgprSqi2Qcu1jCXqtDPfTuFipKBkQYlahClKYJy/lZmiuyGQnzJ3PVYOH/48V9Hdtqnso 44vwDccmU/05rt9alrAnwATZPWhFtuMc/yvKgZkLlMAvOJ23EoRbUq7Pm6OukCJeKc1AUT J6TjmxAIU5gUWvENJoGRa9sqMngH4Cc= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=HnlDHW0a; spf=pass (imf01.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726139712; a=rsa-sha256; cv=none; b=qf7TGlOAlR0ILha3rA7k4t6JIkVl9q/0QXGBHkx7EZAwgEyUSUVNr0mexwaTcD9wHsZaaH nmSfUs7/6yEflfSiB5LxgGpdNmuQbS0D1vsDXgrrKmCyMiOm/jaqBUW3w9zDZ2IPCzUgLE BzwPz/0pAzR24XaLOk1l6xMCNc99kHI= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 40588A451F7; Thu, 12 Sep 2024 11:15:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AE95BC4CECC; Thu, 12 Sep 2024 11:16:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1726139763; bh=xHiv+64ufjlfaZB+1birEmDSAEqJIKZbsr4qpKOqPrE=; h=From:To:Cc:Subject:Date:From; b=HnlDHW0aJOlZ/mQyNxg12Vfl1QyQgA+4ialoezfo+MkiMtzpo37W3A7XFZzNSS2gV nCy6S4qzhyg3yiWGrpjkJ7XXOwsjJGLduW17gDQ2j01fra/JxTwTC8bIEzsO37Jy3O tWtJI3x5KEzkYDPDEvpo4btQfTc/iM6jDN4b9cWi2x7dl8qZl6FekY1wFay7lNo6Ii IJZ7rjlAzG15afy2OtWqIvnGmh3J6ojSrsIxH92HPrMct8haQWjgPUjLGLnJK+FdOA egzwYBUFcoq09uv/i0x7cvVS0WkLeIYtdOhFJpeYR+kGOqMRlQvQJEwypNBVn2bnZB jTk09N6hrZ0Fw== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Keith Busch , Christoph Hellwig , "Zeng, Oak" , Chaitanya Kulkarni Cc: Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [RFC v2 00/21] Provide a new two step DMA API mapping API Date: Thu, 12 Sep 2024 14:15:35 +0300 Message-ID: X-Mailer: git-send-email 2.46.0 MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: D823C40006 X-Stat-Signature: yf1pn17epdnm1c5qbwxzy3qoidquum6s X-HE-Tag: 1726139764-414927 X-HE-Meta: U2FsdGVkX192SA9zlJfx/X/A/RD0vq9cI4+12/LvO/vzrIUXxhTSQkRG1wImoZfxwQbalX9gAEzFpfvQiQ356t4xkpWZ8vuQfDS8ITft/FbTokgMpLj3HvOoiXzaRVCHVm/nK4OqOqn/DCwhg7b/prKXS3/ANQaG2LHDm+DZ5FlVNOXaoPGjzOsvRTxMY385dMUoR3/KbZVZXyLkPl9d02F0laH1YZZbvM+9T8NhX0TTFnzk06wDpP+ovNbxZhln2ZQrgCm0kEprCMNqTcHH5XJIdD2WsQ/yVT/2tkV9nn5t9gHrheatREORrNNw0bOWztHazMmt6lYG6Kts/d7dI/y6OklgAbldJ/uBjdqg7k0vHFONh1bj1YqmkUI4f7KDgEpHLlghPXynCLEvPZrAAwqJINut3PFDBRE8HoAY24iiAiutwkNQgI/0hlu+JUxRuivsMS8zoDqZO4AyBMCmB0oqL1HDvEEcMY0OJn8lmtB83oMdkhGgYi/xAJfAyQ89h0eRGewFW/lsWoEwfq5knYpEmCfLXMiTB/7JArQZny9b2AjElAGOhHTjyKNQFkUXsxuNfWhr2QtpMqZzbyagAJ7sBTIEMjrmMxrvXJ0GaMmtRP0L75WVSiOz3LaIAthTwuIxe0/2JsZ9LCVnpywt7JD4fkUThTFK30XJDBHLkrpZhrXSwjDX3esFJcZnS9yuANVvVCuIRRtVa3dzgyiXVcIAQzLNxtdJpHsV8fh24Tj81EUnnaA4jcyykUlFqR/LYto7LIUJqcAfEfNjEvzM2p0CfaAPnZ1fkGqdytKIBvv2Fp+rQWJmrjQwCmAY5+7dVXeSReMv6wfPhrdohv3txhAytVkeoRA4JQOGRTlclBXuurpH1GyDD8vFH8RtS9zjYUZLvB9vfzEWhRNaHsApgugfJl0k8ZDLZYim+MbPeA7yDkCfiobTOpjdwC35F2OiakdbXJOLTY+3tfu1QTU uynFcaDV S97uJdNrIWrLHb2e/d+19w6LuYFo1/GKpFTCV1pazvaljPGINNnnhEgmQO5GtJQMcuKDA3c8D1OioT3eittETgWDayzPMuDRu0E7GgQf3imaM/jIRbq+X0xo6YmulPez9jKn+2a8W777BGqijk7qf0wvHZEYD3h4Xwn7Vogrn4kpfgmsXkIcSMV3ze4hhxiFJz0rSWeNmg3Y5UyWeIrEZ/VKqnHBiPmCaZED2G74UbkEMORANybBMxYyIQQedxitffWfm3kZ/oGyFoL4FZ0LvrOjCWAaXXSqIrHLL8AD9ZLgN3oSFuvgSpOm0T979lqABPpgdzbzwknAk7clz2KEBY6D1LqBMT2efwbte+bgxx5ZemAo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Instead of waiting and waiting when I finally fix NVMe patch which doesn't work correctly in DMA direct mode (it causes to failures in module reload), I decided to send the series as is to at least get a feedback about API and the overall direction. Thanks ------------------------------------------------------------------------- Changelog: v2: * Remove attr field from dma_iova_attrs * Added attr parameter to every new API call except alloc_iova * Embed use_iova boolean into struct dma_iova_state * Embed struct dma_memory_type into struct dma_iova_state * Changed function signatures and added new function to initialize state * Combined dma_set_memory_type() and dma_can_use_iova() to one function, they were used together anyway * Made dma_start_range/dma_end_range to be NOP for non-iommu case * Restructured the code to avoid inclusion of linux/pci.h in global header * Based on DMA static calls series * Changed iommu/dma functions to get parameters instead of dma_iova_state * Removed iommu domain pointer from dma_iova_state * dma_link_range now returns DMA address * Rewrote NVMe patch v1: https://lore.kernel.org/all/cover.1719909395.git.leon@kernel.org * Rewrote cover letter * Changed to API as proposed https://lore.kernel.org/linux-rdma/20240322184330.GL66976@ziepe.ca/ * Removed IB DMA wrappers and use DMA API directly v0: https://lore.kernel.org/all/cover.1709635535.git.leon@kernel.org ------------------------------------------------------------------------- Currently the only efficient way to map a complex memory description through the DMA API is by using the scatterlist APIs. The SG APIs are unique in that they efficiently combine the two fundamental operations of sizing and allocating a large IOVA window from the IOMMU and processing all the per-address swiotlb/flushing/p2p/map details. This uniqueness has been a long standing pain point as the scatterlist API is mandatory, but expensive to use. It prevents any kind of optimization or feature improvement (such as avoiding struct page for P2P) due to the impossibility of improving the scatterlist. Several approaches have been explored to expand the DMA API with additional scatterlist-like structures (BIO[1], rlist[2]), instead split up the DMAAPI to allow callers to bring their own data structure. The API is split up into parts: - dma_alloc_iova() / dma_free_iova() To do any pre-allocation required. This is done based on the caller supplying some details about how much IOMMU address space it would need in worst case. - dma_link_range() / dma_unlink_range() Perform the actual mapping into the pre-allocated IOVA. This is very similar to dma_map_page(). A driver will extent its mapping size using its own data structure, such as BIO, to request the required IOVA. Then it will iterate directly over it's data structure to DMA map each range. The result can then be stored directly into the HW specific DMA list. No intermediate scatterlist is required. In this series, examples of three users are converted to the new API to show the benefits. Each user has a unique flow: 1. RDMA ODP is an example of "SVA mirroring" using HMM that needs to dynamically map/unmap large numbers of single pages. This becomes significantly faster in the IOMMU case as the map/unmap is now just a page table walk, the IOVA allocation is pre-computed once. Significant amounts of memory are saved as there is no longer a need to store the dma_addr_t of each page. 2. VFIO PCI live migration code is building a very large "page list" for the device. Instead of allocating a scatter list entry per allocated page it can just allocate an array of 'struct page *', saving a large amount of memory. 3. NVMe PCI demonstrates how a BIO can be converted to a HW scatter list without having to allocate then populate an intermediate SG table. This step is first along a path to provide alternatives to scatterlist and solve some of the abuses and design mistakes, for instance in DMABUF's P2P support. The ODP and VFIO versions are complete and fully tested, they can be the users of the new API to merge it. The NVMe requires more work. [1] https://lore.kernel.org/all/169772852492.5232.17148564580779995849.stgit@klimt.1015granger.net/ [2] https://lore.kernel.org/all/ZD2lMvprVxu23BXZ@ziepe.ca/ Thanks Leon Romanovsky (21): iommu/dma: Provide an interface to allow preallocate IOVA iommu/dma: Implement link/unlink ranges callbacks iommu/dma: Add check if IOVA can be used dma-mapping: initialize IOVA state struct dma-mapping: provide an interface to allocate IOVA dma-mapping: set and query DMA IOVA state dma-mapping: implement link range API mm/hmm: let users to tag specific PFN with DMA mapped bit dma-mapping: provide callbacks to link/unlink HMM PFNs to specific IOVA RDMA/umem: Preallocate and cache IOVA for UMEM ODP RDMA/umem: Store ODP access mask information in PFN RDMA/core: Separate DMA mapping to caching IOVA and page linkage RDMA/umem: Prevent UMEM ODP creation with SWIOTLB vfio/mlx5: Explicitly use number of pages instead of allocated length vfio/mlx5: Rewrite create mkey flow to allow better code reuse vfio/mlx5: Explicitly store page list vfio/mlx5: Convert vfio to use DMA link API nvme-pci: remove optimizations for single DMA entry nvme-pci: precalculate number of DMA entries for each command nvme-pci: use new dma API nvme-pci: don't allow mapping of bvecs with offset drivers/infiniband/core/umem_odp.c | 216 +++++--------- drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 + drivers/infiniband/hw/mlx5/odp.c | 44 +-- drivers/iommu/dma-iommu.c | 165 ++++++++++- drivers/nvme/host/pci.c | 428 ++++++++++++++------------- drivers/pci/p2pdma.c | 4 +- drivers/vfio/pci/mlx5/cmd.c | 312 ++++++++++--------- drivers/vfio/pci/mlx5/cmd.h | 23 +- drivers/vfio/pci/mlx5/main.c | 89 +++--- include/linux/dma-map-ops.h | 7 + include/linux/dma-mapping.h | 85 ++++++ include/linux/hmm.h | 4 + include/linux/iommu-dma.h | 43 +++ include/rdma/ib_umem_odp.h | 23 +- kernel/dma/mapping.c | 232 +++++++++++++++ mm/hmm.c | 34 ++- 16 files changed, 1096 insertions(+), 614 deletions(-)