From patchwork Thu Oct 7 10:42:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gal Pressman X-Patchwork-Id: 12541537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73755C433FE for ; Thu, 7 Oct 2021 10:43:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5876F60F6C for ; Thu, 7 Oct 2021 10:43:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240726AbhJGKp0 (ORCPT ); Thu, 7 Oct 2021 06:45:26 -0400 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:3078 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240800AbhJGKpZ (ORCPT ); Thu, 7 Oct 2021 06:45:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1633603412; x=1665139412; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=NNoJ2JFFRziHZiTbB/PCsJ4bfqbQfqXZ170Oox+pA2I=; b=RXVJj24tWBGv7yhk3UwT8fGRA4OoVyNKyDFe8XGiXapmsAN7zOer8jl5 6AtXxHOFD2Pe/GEHJUOWLLInQGP3fsHgJjDxkXRddnCjPHiyM882S+iSc iEBRgWYh24Z24mdiz/J2116ovOP95E5Sbn5D81VWdJLVTjjzqW2wGHrou g=; X-IronPort-AV: E=Sophos;i="5.85,354,1624320000"; d="scan'208";a="165371252" Received: from pdx4-co-svc-p1-lb2-vlan3.amazon.com (HELO email-inbound-relay-iad-1a-4ba5c7da.us-east-1.amazon.com) ([10.25.36.214]) by smtp-border-fw-9102.sea19.amazon.com with ESMTP; 07 Oct 2021 10:43:22 +0000 Received: from EX13D19EUB002.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan2.iad.amazon.com [10.40.163.34]) by email-inbound-relay-iad-1a-4ba5c7da.us-east-1.amazon.com (Postfix) with ESMTPS id 73C5F8D8DB; Thu, 7 Oct 2021 10:43:18 +0000 (UTC) Received: from EX13MTAUWC001.ant.amazon.com (10.43.162.135) by EX13D19EUB002.ant.amazon.com (10.43.166.78) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Thu, 7 Oct 2021 10:43:17 +0000 Received: from 8c85908914bf.ant.amazon.com.com (10.1.213.27) by mail-relay.amazon.com (10.43.162.232) with Microsoft SMTP Server id 15.0.1497.23 via Frontend Transport; Thu, 7 Oct 2021 10:43:10 +0000 From: Gal Pressman To: Sumit Semwal , =?utf-8?q?Christian_K=C3=B6nig?= , Doug Ledford , Jason Gunthorpe CC: , , , , Oded Gabbay , Tomer Tayar , Yossi Leybovich , Alexander Matushevsky , Leon Romanovsky , Jianxin Xiong , Firas Jahjah , Gal Pressman Subject: [RFC PATCH 0/2] EFA dmabuf memory regions Date: Thu, 7 Oct 2021 13:42:58 +0300 Message-ID: <20211007104301.76693-1-galpress@amazon.com> X-Mailer: git-send-email 2.33.0 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Hey all, This is a followup to my previous RFC [1], which now makes use of the dynamic attachment API implemented in the RDMA subsystem, but calls dma_buf_pin() in order to make sure that the move_notify callback will not be used, as suggested by Christian. As explained in the previous RFC, move_notify requires the RDMA device to support on-demand-paging (ODP) which is not common on most devices (only supported by mlx5). While the dynamic requirement makes sense for certain GPUs, some devices (such as habanalabs) have device memory that is always "pinned" and do not need/use the move_notify operation. The first patch changes the dmabuf documentation to make it clear that pinning does not necessarily mean the memory must be moved to system memory, it is up to the exporter to decide. The motivation of this RFC is to use habanalabs as the dmabuf exporter, and EFA as the importer to allow for peer2peer access through libibverbs. [1] https://lore.kernel.org/linux-rdma/20210818074352.29950-1-galpress@amazon.com/ Thanks Gal Pressman (2): dma-buf: Fix pin callback comment RDMA/efa: Add support for dmabuf memory regions drivers/infiniband/hw/efa/efa.h | 4 + drivers/infiniband/hw/efa/efa_main.c | 1 + drivers/infiniband/hw/efa/efa_verbs.c | 166 +++++++++++++++++++++----- include/linux/dma-buf.h | 4 +- 4 files changed, 142 insertions(+), 33 deletions(-)