From patchwork Tue Oct 6 08:37:40 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 7333711 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 01A129F2F7 for ; Tue, 6 Oct 2015 08:37:57 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 2485C20616 for ; Tue, 6 Oct 2015 08:37:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 24F262062B for ; Tue, 6 Oct 2015 08:37:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752155AbbJFIhu (ORCPT ); Tue, 6 Oct 2015 04:37:50 -0400 Received: from mail-wi0-f173.google.com ([209.85.212.173]:35892 "EHLO mail-wi0-f173.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751137AbbJFIhq (ORCPT ); Tue, 6 Oct 2015 04:37:46 -0400 Received: by wicgb1 with SMTP id gb1so154246519wic.1 for ; Tue, 06 Oct 2015 01:37:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=Mj3PhAbVP21rsct4IwtlZnx28J1O6lsHjjwuV0AToOc=; b=Nrqnz+oVp2yc9Votyyeb/BGC6R+HllomFqgH0hyQHhd3xyFdZfUbJSpP4Pnv1DnPJR Q90pKYzTLJkFIJ2h9WX90YzJphoN3/9BO2j2xk7GHmyFHhA3o1FmUEllojDeXJc+n4El e/nsJVAgWah/0+Eoj1ePW3hdRdbpk9emna+Hv0MGkzPNuxG06Ho8HAsBmKWysOkM8ib/ 7iXOS7jrEnuKTYncEMpu6aG9Jbk4rbw1RXbLEhWb7ln/0BQTICefvXoeKVo6iGhD39xa q9Euzks4hC8FMmX4Mm684xosbTu3IvQHI/1fhQFKuihkwBeZwzuSTr3anuf/i3cpWU1k SkFA== X-Gm-Message-State: ALoCoQn0NfXC8F4Ts6k+2Jfp20g9Qe1BMn8FniJPlkSzWTj3S429ERNSHVpwZe6c06uoNVXiZAV+ X-Received: by 10.180.19.41 with SMTP id b9mr15279627wie.71.1444120665003; Tue, 06 Oct 2015 01:37:45 -0700 (PDT) Received: from [10.223.0.123] ([193.47.165.251]) by smtp.googlemail.com with ESMTPSA id qq4sm31139864wjc.14.2015.10.06.01.37.43 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 06 Oct 2015 01:37:44 -0700 (PDT) Subject: Re: [PATCH v1 00/24] New fast registration API To: Bart Van Assche References: <1442482947-27785-1-git-send-email-sagig@mellanox.com> <560AE099.2080004@sandisk.com> <560AFB71.3010003@dev.mellanox.co.il> <560C30F4.50900@sandisk.com> <560C42CE.6090000@sandisk.com> <560CDDBC.8000400@dev.mellanox.co.il> <560D730C.9000302@sandisk.com> <560D9E69.70604@sandisk.com> <35618B90-4D6E-4036-A69B-4405F020D440@dev.mellanox.co.il> <560EA4A1.3080709@sandisk.com> Cc: Sagi Grimberg , "linux-rdma@vger.kernel.org" , "linux-nfs@vger.kernel.org" , "Nicholas A. Bellinger" From: Sagi Grimberg Message-ID: <56138854.4040209@dev.mellanox.co.il> Date: Tue, 6 Oct 2015 11:37:40 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <560EA4A1.3080709@sandisk.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 10/2/2015 6:37 PM, Bart Van Assche wrote: > On 10/01/2015 11:14 PM, Sagi Grimberg wrote: >> Would you mind sending me your .config? > > Hello Sagi, Hi Bart, > > I just sent this .config file to you off-list. I see now the error you are referring to. The issue is that the device requires the MR page array to have an alignment (0x40 for mlx4 and 0x400 for mlx5). When I modified the page array allocation to be non-coherent I didn't take care of alignment. Taking care of this alignment may result in a higher order allocation as we'd need to add (alignment - 1) to the allocation size. e.g. a 512 pages on mlx4 will become: 512 * 8 + 0x40 - 1 = 4159 I'm leaning towards this approach. Any preference? I think this patch should take care of mlx4: } --- Sagi. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h index de6eab3..4c69247 100644 --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h @@ -129,6 +129,8 @@ struct mlx4_ib_cq { struct list_head recv_qp_list; }; +#define MLX4_MR_PAGES_ALIGN 0x40 + struct mlx4_ib_mr { struct ib_mr ibmr; __be64 *pages; @@ -137,6 +139,7 @@ struct mlx4_ib_mr { u32 max_pages; struct mlx4_mr mmr; struct ib_umem *umem; + void *pages_alloc; }; struct mlx4_ib_mw { diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c index fa01f75..d3f8175 100644 --- a/drivers/infiniband/hw/mlx4/mr.c +++ b/drivers/infiniband/hw/mlx4/mr.c @@ -279,10 +279,14 @@ mlx4_alloc_priv_pages(struct ib_device *device, int size = max_pages * sizeof(u64); int ret; - mr->pages = kzalloc(size, GFP_KERNEL); - if (!mr->pages) + size += max_t(int, MLX4_MR_PAGES_ALIGN - ARCH_KMALLOC_MINALIGN, 0); + + mr->pages_alloc = kzalloc(size, GFP_KERNEL); + if (!mr->pages_alloc) return -ENOMEM; + mr->pages = PTR_ALIGN(mr->pages_alloc, MLX4_MR_PAGES_ALIGN); + mr->page_map = dma_map_single(device->dma_device, mr->pages, size, DMA_TO_DEVICE); @@ -293,20 +297,22 @@ mlx4_alloc_priv_pages(struct ib_device *device, return 0; err: - kfree(mr->pages); + kfree(mr->pages_alloc); return ret; } static void mlx4_free_priv_pages(struct mlx4_ib_mr *mr) { - struct ib_device *device = mr->ibmr.device; - int size = mr->max_pages * sizeof(u64); - if (mr->pages) { + struct ib_device *device = mr->ibmr.device; + int size = mr->max_pages * sizeof(u64); + + size += max_t(int, MLX4_MR_PAGES_ALIGN - ARCH_KMALLOC_MINALIGN, 0); + dma_unmap_single(device->dma_device, mr->page_map, size, DMA_TO_DEVICE); - kfree(mr->pages); + kfree(mr->pages_alloc); mr->pages = NULL; }