From patchwork Thu May 26 17:19:33 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 9137289 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 0057C607D3 for ; Thu, 26 May 2016 17:19:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E3FAF21BED for ; Thu, 26 May 2016 17:19:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D8B31279C4; Thu, 26 May 2016 17:19:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5926C21BED for ; Thu, 26 May 2016 17:19:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754761AbcEZRTj (ORCPT ); Thu, 26 May 2016 13:19:39 -0400 Received: from mail-wm0-f67.google.com ([74.125.82.67]:34426 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753612AbcEZRTi (ORCPT ); Thu, 26 May 2016 13:19:38 -0400 Received: by mail-wm0-f67.google.com with SMTP id n129so6879420wmn.1 for ; Thu, 26 May 2016 10:19:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=/jReHpU68W7CIpncmsYpNlenaSFzhY+RGDf/2ExIztE=; b=kksF5932GElMpvzqurIW0cDidAH+bzhlTuU07OOWl7tz/kQjcTbi8rKtxZ9rcfXGHC 5QqyeKQKxnM3D+OGg1TIgOJDFokFWGQYz8M72KgNYP+8hzclM1vI2Lw3wEjPzoUs240Z h0obMI2emsz+SC+gaiRXR58JitCt0GWD0lkqGDSr9097/Nk0cjHoDYNf4Ln25lFInyHn xYtC4C2DTDt51KzC3GBnb0Rxk2SmimDgNLZO4uPh4smgq6uXFOavAW3t1v4ord/Xftx8 zaTJyrEoGjjcifFRRNrygcJP3czsbqZheTHoCoRkAYo4FQjLVOKvdT3IazSn5082bM8M Fa3w== X-Gm-Message-State: ALyK8tJUdcoCGUyFodq+gUtRw6NcBD334cPjOUXj+z6FYrkQpWjikS+sSor3mILBoIXQ3g== X-Received: by 10.194.204.233 with SMTP id lb9mr10399658wjc.147.1464283176968; Thu, 26 May 2016 10:19:36 -0700 (PDT) Received: from [192.168.1.24] (bzq-82-81-101-184.red.bezeqint.net. [82.81.101.184]) by smtp.gmail.com with ESMTPSA id f1sm3584017wmf.22.2016.05.26.10.19.34 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 26 May 2016 10:19:36 -0700 (PDT) Subject: Re: RDMA Read: Local protection error To: Chuck Lever , Bart Van Assche , Yishai Hadas References: <1A4F4C32-CE5A-44D9-9BFE-0E1F8D5DF44D@oracle.com> <57238F8C.70505@sandisk.com> <57277B63.8030506@sandisk.com> <6BBFD126-877C-4638-BB91-ABF715E29326@oracle.com> <1AFD636B-09FC-4736-B1C5-D1D9FA0B97B0@oracle.com> <8a3276bf-f716-3dca-9d54-369fc3bdcc39@dev.mellanox.co.il> <574728EC.9040802@grimberg.me> Cc: Yishai Hadas , linux-rdma , Or Gerlitz , Joonsoo Kim , Haggai Eran , Majd Dibbiny From: Sagi Grimberg Message-ID: <57473025.5020801@grimberg.me> Date: Thu, 26 May 2016 20:19:33 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.8.0 MIME-Version: 1.0 In-Reply-To: <574728EC.9040802@grimberg.me> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP >>>>> When debugging is disabled, kzalloc returns page-aligned >>>>> addresses: >>>> >>>> Is it defined some where that regular kzalloc/kmalloc guaranties to >>>> return a page-aligned address as you see in your testing ? if so the >>>> debug mode should behave the same. Otherwise we can consider using any >>>> flag allocation that can force that if such exists. >>>> Let's get other people's input here. >>> >>> My understanding is that the fact that k[mz]alloc() returns a >>> page-aligned buffer if the allocation size is > PAGE_SIZE / 2 is a >>> side effect of the implementation and not something callers of that >>> function should rely on. I think the only assumption k[mz]alloc() >>> callers should rely on is that the allocated memory respects >>> ARCH_KMALLOC_MINALIGN. >> >> I agree. mlx4_alloc_priv_pages() is carefully designed to >> correct the alignment of the buffer, so it already assumes >> that it is not getting a page-aligned buffer. >> >> The alignment isn't the problem here, though. It's that >> the buffer contains a page-boundary. That is guaranteed >> to be the case for HCAs that support more than 512 >> sges, so that will have to be addressed (at least in >> mlx5). > > rrr... > > I think we should make the pages allocations dma coherent > in order to fix that... > > Nice catch Chunk. Does this untested patch help (if so, mlx5 will need an identical patch)? Tested-by: Chuck Lever --- static void @@ -311,9 +294,8 @@ mlx4_free_priv_pages(struct mlx4_ib_mr *mr) struct ib_device *device = mr->ibmr.device; int size = mr->max_pages * sizeof(u64); - dma_unmap_single(device->dma_device, mr->page_map, - size, DMA_TO_DEVICE); - kfree(mr->pages_alloc); + dma_free_coherent(device->dma_device, size, + mr->pages, mr->page_map); mr->pages = NULL; } } @@ -532,19 +514,8 @@ int mlx4_ib_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents, unsigned int sg_offset) { struct mlx4_ib_mr *mr = to_mmr(ibmr); - int rc; mr->npages = 0; - ib_dma_sync_single_for_cpu(ibmr->device, mr->page_map, - sizeof(u64) * mr->max_pages, - DMA_TO_DEVICE); - - rc = ib_sg_to_pages(ibmr, sg, sg_nents, sg_offset, mlx4_set_page); - - ib_dma_sync_single_for_device(ibmr->device, mr->page_map, - sizeof(u64) * mr->max_pages, - DMA_TO_DEVICE); - - return rc; + return ib_sg_to_pages(ibmr, sg, sg_nents, sg_offset, mlx4_set_page); } -- -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h index ba328177eae9..78e9b3addfea 100644 --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h @@ -139,7 +139,6 @@ struct mlx4_ib_mr { u32 max_pages; struct mlx4_mr mmr; struct ib_umem *umem; - void *pages_alloc; }; struct mlx4_ib_mw { diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c index b04f6238e7e2..becb4a65c755 100644 --- a/drivers/infiniband/hw/mlx4/mr.c +++ b/drivers/infiniband/hw/mlx4/mr.c @@ -278,30 +278,13 @@ mlx4_alloc_priv_pages(struct ib_device *device, int max_pages) { int size = max_pages * sizeof(u64); - int add_size; - int ret; - - add_size = max_t(int, MLX4_MR_PAGES_ALIGN - ARCH_KMALLOC_MINALIGN, 0); - mr->pages_alloc = kzalloc(size + add_size, GFP_KERNEL); - if (!mr->pages_alloc) + mr->pages = dma_alloc_coherent(device->dma_device, size, + &mr->page_map, GFP_KERNEL); + if (!mr->pages) return -ENOMEM; - mr->pages = PTR_ALIGN(mr->pages_alloc, MLX4_MR_PAGES_ALIGN); - - mr->page_map = dma_map_single(device->dma_device, mr->pages, - size, DMA_TO_DEVICE); - - if (dma_mapping_error(device->dma_device, mr->page_map)) { - ret = -ENOMEM; - goto err; - } - return 0; -err: - kfree(mr->pages_alloc); - - return ret; }