From patchwork Sun May 29 08:15:27 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 9139691 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id CDB7660221 for ; Sun, 29 May 2016 08:15:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BBF2B279E2 for ; Sun, 29 May 2016 08:15:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AF5C727F17; Sun, 29 May 2016 08:15:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_TVD_MIME_EPI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 53CA4279E2 for ; Sun, 29 May 2016 08:15:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752609AbcE2IPf (ORCPT ); Sun, 29 May 2016 04:15:35 -0400 Received: from bombadil.infradead.org ([198.137.202.9]:59805 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752430AbcE2IPe (ORCPT ); Sun, 29 May 2016 04:15:34 -0400 Received: from hch by bombadil.infradead.org with local (Exim 4.80.1 #2 (Red Hat Linux)) id 1b6vsd-0002t2-3U; Sun, 29 May 2016 08:15:27 +0000 Date: Sun, 29 May 2016 01:15:27 -0700 From: Christoph Hellwig To: Sagi Grimberg Cc: Christoph Hellwig , Christoph Lameter , Leon Romanovsky , Chuck Lever , Bart Van Assche , Yishai Hadas , Yishai Hadas , linux-rdma , Or Gerlitz , Joonsoo Kim , Haggai Eran , Majd Dibbiny Subject: Re: RDMA Read: Local protection error Message-ID: <20160529081527.GA5839@infradead.org> References: <8a3276bf-f716-3dca-9d54-369fc3bdcc39@dev.mellanox.co.il> <574728EC.9040802@grimberg.me> <57473025.5020801@grimberg.me> <20160526192351.GV25500@leon.nu> <574A941D.9050404@grimberg.me> <20160529071749.GB24347@infradead.org> <574AA4BE.2060207@grimberg.me> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <574AA4BE.2060207@grimberg.me> User-Agent: Mutt/1.5.24 (2015-08-30) X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Sun, May 29, 2016 at 11:13:50AM +0300, Sagi Grimberg wrote: > > >>Really? It's used all over the stack... > >> > >>So the suggestion is to restrict the allocation does not > >>cross the page boundary (which can easily be done with making > >>MLX4_MR_PAGES_ALIGN be PAGE_SIZE or larger than PAGE_SIZE/2)? > > > >We could simply switch to alloc_pages, e.g. something like the patch below: > > Patch below? :) Here we go. > > >But I have to admit I really like the coherent dma mapping version, > >for all sane architectures that works much better, although it sucks > >for some architetures where dma coherent mappings cause a bit of > >overhead. > > But this is specific to mlx4 which supports up 511 pages per MR, mlx5 > will need a coherent allocation anyways right (it supports up to 64K > pages per MR)? Why does that make a difference? diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h index ba32817..7adfa4b 100644 --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h @@ -129,8 +129,6 @@ struct mlx4_ib_cq { struct list_head recv_qp_list; }; -#define MLX4_MR_PAGES_ALIGN 0x40 - struct mlx4_ib_mr { struct ib_mr ibmr; __be64 *pages; @@ -139,7 +137,6 @@ struct mlx4_ib_mr { u32 max_pages; struct mlx4_mr mmr; struct ib_umem *umem; - void *pages_alloc; }; struct mlx4_ib_mw { diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c index b04f623..3c6a21f 100644 --- a/drivers/infiniband/hw/mlx4/mr.c +++ b/drivers/infiniband/hw/mlx4/mr.c @@ -278,17 +278,13 @@ mlx4_alloc_priv_pages(struct ib_device *device, int max_pages) { int size = max_pages * sizeof(u64); - int add_size; int ret; - add_size = max_t(int, MLX4_MR_PAGES_ALIGN - ARCH_KMALLOC_MINALIGN, 0); - - mr->pages_alloc = kzalloc(size + add_size, GFP_KERNEL); - if (!mr->pages_alloc) + mr->pages = (__be64 *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, + get_order(size)); + if (!mr->pages) return -ENOMEM; - - mr->pages = PTR_ALIGN(mr->pages_alloc, MLX4_MR_PAGES_ALIGN); - + mr->page_map = dma_map_single(device->dma_device, mr->pages, size, DMA_TO_DEVICE); @@ -299,7 +295,7 @@ mlx4_alloc_priv_pages(struct ib_device *device, return 0; err: - kfree(mr->pages_alloc); + free_pages((unsigned long)mr->pages, get_order(size)); return ret; } @@ -313,7 +309,7 @@ mlx4_free_priv_pages(struct mlx4_ib_mr *mr) dma_unmap_single(device->dma_device, mr->page_map, size, DMA_TO_DEVICE); - kfree(mr->pages_alloc); + free_pages((unsigned long)mr->pages, get_order(size)); mr->pages = NULL; } }