From patchwork Tue Oct 12 12:09:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gal Pressman X-Patchwork-Id: 12552343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34AE6C433FE for ; Tue, 12 Oct 2021 12:09:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1698B60E74 for ; Tue, 12 Oct 2021 12:09:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236196AbhJLMLa (ORCPT ); Tue, 12 Oct 2021 08:11:30 -0400 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:13763 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232665AbhJLML3 (ORCPT ); Tue, 12 Oct 2021 08:11:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1634040568; x=1665576568; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=Cu9aT9PKxubFeRU3gw1Gjtc6Ldb5NRA5s/4pefgZcj0=; b=QRa8JGzONRKvpfdfZGJxZ+Zljhht969ejd/GrHt3qSmtov5LQIJG9M9r KhWp5Y0VQ+n3j2IGRspJ1+DxIT3CenxA2gwTgYhMzACNL+zfUuq3HfI1a QkrC9VcDp52rPSRGAenripZUM0XG9Q2SngxAgm7cG29C7aYUXR2s5u0rK 4=; X-IronPort-AV: E=Sophos;i="5.85,367,1624320000"; d="scan'208";a="147112800" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-iad-1d-1c682de1.us-east-1.amazon.com) ([10.43.8.2]) by smtp-border-fw-6002.iad6.amazon.com with ESMTP; 12 Oct 2021 12:09:19 +0000 Received: from EX13D13EUA004.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan3.iad.amazon.com [10.40.163.38]) by email-inbound-relay-iad-1d-1c682de1.us-east-1.amazon.com (Postfix) with ESMTPS id 98058A20F0; Tue, 12 Oct 2021 12:09:16 +0000 (UTC) Received: from EX13MTAUEA002.ant.amazon.com (10.43.61.77) by EX13D13EUA004.ant.amazon.com (10.43.165.22) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Tue, 12 Oct 2021 12:09:15 +0000 Received: from 8c85908914bf.ant.amazon.com.com (10.1.212.21) by mail-relay.amazon.com (10.43.61.169) with Microsoft SMTP Server id 15.0.1497.23 via Frontend Transport; Tue, 12 Oct 2021 12:09:11 +0000 From: Gal Pressman To: Sumit Semwal , =?utf-8?q?Christian_K=C3=B6nig?= , Doug Ledford , Jason Gunthorpe CC: , , , , Oded Gabbay , Tomer Tayar , Yossi Leybovich , Alexander Matushevsky , Leon Romanovsky , Jianxin Xiong , Firas Jahjah , Gal Pressman Subject: [PATCH for-next 0/3] EFA dmabuf memory regions Date: Tue, 12 Oct 2021 15:09:00 +0300 Message-ID: <20211012120903.96933-1-galpress@amazon.com> X-Mailer: git-send-email 2.33.0 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Hey all, This is a followup to my previous RFCs [1][2], which now adds a new api to the RDMA subsystem that allows drivers to get a pinned dmabuf memory region without requiring an implementation of the move_notify callback. The new api makes use of the dynamic attachment api implemented in the RDMA subsystem, but calls dma_buf_pin() in order to make sure that the callback will not be called, as suggested by Christian. As explained in the previous RFC, move_notify requires the RDMA device to support on-demand-paging (ODP) which is not common on most devices (only supported by mlx5). While the dynamic requirement makes sense for certain GPUs, some devices (such as habanalabs) have device memory that is always "pinned" and do not need/use the move_notify operation. Patch #1 changes the dmabuf documentation to make it clear that pinning does not necessarily mean the memory must be moved to system memory, it is up to the exporter to decide. Patch #2 adds the RDMA api that allows drivers to get pinned dmabuf memory regions. Patch #3 adds the EFA implementation of the dmabuf importer. The motivation of this submission is to use habanalabs as the dmabuf exporter, and EFA as the importer to allow for peer2peer access through libibverbs. [1] https://lore.kernel.org/linux-rdma/20210818074352.29950-1-galpress@amazon.com/ [2] https://lore.kernel.org/linux-rdma/20211007104301.76693-1-galpress@amazon.com/ Thanks Gal Pressman (3): dma-buf: Fix pin callback comment RDMA/umem: Allow pinned dmabuf umem usage RDMA/efa: Add support for dmabuf memory regions drivers/infiniband/core/umem_dmabuf.c | 51 +++++++++++ drivers/infiniband/hw/efa/efa.h | 4 + drivers/infiniband/hw/efa/efa_main.c | 1 + drivers/infiniband/hw/efa/efa_verbs.c | 127 +++++++++++++++++++------- include/linux/dma-buf.h | 4 +- include/rdma/ib_umem.h | 12 +++ 6 files changed, 166 insertions(+), 33 deletions(-)