From patchwork Tue Nov 28 07:10:28 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wei Hu (Xavier)" X-Patchwork-Id: 10078745 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B20CD6056A for ; Tue, 28 Nov 2017 06:41:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7947828E9F for ; Tue, 28 Nov 2017 06:41:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6DCAC29181; Tue, 28 Nov 2017 06:41:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ED02128E9F for ; Tue, 28 Nov 2017 06:41:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751036AbdK1Gl3 (ORCPT ); Tue, 28 Nov 2017 01:41:29 -0500 Received: from szxga04-in.huawei.com ([45.249.212.190]:11440 "EHLO szxga04-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750842AbdK1Gl2 (ORCPT ); Tue, 28 Nov 2017 01:41:28 -0500 Received: from 172.30.72.58 (EHLO DGGEMS411-HUB.china.huawei.com) ([172.30.72.58]) by dggrg04-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id DLN75787; Tue, 28 Nov 2017 14:41:23 +0800 (CST) Received: from linux-ioko.site (10.71.200.31) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.361.1; Tue, 28 Nov 2017 14:41:15 +0800 From: "Wei Hu (Xavier)" To: , CC: , , , , , Subject: [PATCH V2 rdma-rc 3/3] RDMA/hns: Get rid of page operation after dma_alloc_coherent Date: Tue, 28 Nov 2017 15:10:28 +0800 Message-ID: <1511853028-35191-4-git-send-email-xavier.huwei@huawei.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1511853028-35191-1-git-send-email-xavier.huwei@huawei.com> References: <1511853028-35191-1-git-send-email-xavier.huwei@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.71.200.31] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020204.5A1D0513.002D, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: fb5a0f4cfa6309bc7b9ae9d86e55f1b5 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In general dma_alloc_coherent() return a CPU virtual address and a DMA address, and we have no guarantee that the underlying memory even has an associated struct page at all. This patch gets rid of the page operation after dma_alloc_coherent, and records the VA returned form dma_alloc_coherent in the struct of hem in hns RoCE driver. Fixes: 9a44353("IB/hns: Add driver files for hns RoCE driver") Signed-off-by: Wei Hu (Xavier) Signed-off-by: Shaobo Xu Signed-off-by: Lijun Ou Signed-off-by: Yixian Liu Signed-off-by: Xiping Zhang (Francis) --- changelog: v1 -> v2: Revise the commit message and add Fixes line at Jason's comment. The related link: https://lkml.org/lkml/2017/11/27/841 --- drivers/infiniband/hw/hns/hns_roce_hem.c | 25 +++++++++++++------------ drivers/infiniband/hw/hns/hns_roce_hem.h | 1 + 2 files changed, 14 insertions(+), 12 deletions(-) diff --git a/drivers/infiniband/hw/hns/hns_roce_hem.c b/drivers/infiniband/hw/hns/hns_roce_hem.c index 8b733a6..0eeabfb 100644 --- a/drivers/infiniband/hw/hns/hns_roce_hem.c +++ b/drivers/infiniband/hw/hns/hns_roce_hem.c @@ -224,6 +224,7 @@ static struct hns_roce_hem *hns_roce_alloc_hem(struct hns_roce_dev *hr_dev, sg_init_table(chunk->mem, HNS_ROCE_HEM_CHUNK_LEN); chunk->npages = 0; chunk->nsg = 0; + memset(chunk->buf, 0, sizeof(chunk->buf)); list_add_tail(&chunk->list, &hem->chunk_list); } @@ -240,8 +241,7 @@ static struct hns_roce_hem *hns_roce_alloc_hem(struct hns_roce_dev *hr_dev, if (!buf) goto fail; - sg_set_buf(mem, buf, PAGE_SIZE << order); - WARN_ON(mem->offset); + chunk->buf[chunk->npages] = buf; sg_dma_len(mem) = PAGE_SIZE << order; ++chunk->npages; @@ -267,8 +267,8 @@ void hns_roce_free_hem(struct hns_roce_dev *hr_dev, struct hns_roce_hem *hem) list_for_each_entry_safe(chunk, tmp, &hem->chunk_list, list) { for (i = 0; i < chunk->npages; ++i) dma_free_coherent(hr_dev->dev, - chunk->mem[i].length, - lowmem_page_address(sg_page(&chunk->mem[i])), + sg_dma_len(&chunk->mem[i]), + chunk->buf[i], sg_dma_address(&chunk->mem[i])); kfree(chunk); } @@ -722,11 +722,12 @@ void *hns_roce_table_find(struct hns_roce_dev *hr_dev, struct hns_roce_hem_chunk *chunk; struct hns_roce_hem_mhop mhop; struct hns_roce_hem *hem; - struct page *page = NULL; + void *addr = NULL; unsigned long mhop_obj = obj; unsigned long obj_per_chunk; unsigned long idx_offset; int offset, dma_offset; + int length; int i, j; u32 hem_idx = 0; @@ -763,25 +764,25 @@ void *hns_roce_table_find(struct hns_roce_dev *hr_dev, list_for_each_entry(chunk, &hem->chunk_list, list) { for (i = 0; i < chunk->npages; ++i) { + length = sg_dma_len(&chunk->mem[i]); if (dma_handle && dma_offset >= 0) { - if (sg_dma_len(&chunk->mem[i]) > - (u32)dma_offset) + if (length > (u32)dma_offset) *dma_handle = sg_dma_address( &chunk->mem[i]) + dma_offset; - dma_offset -= sg_dma_len(&chunk->mem[i]); + dma_offset -= length; } - if (chunk->mem[i].length > (u32)offset) { - page = sg_page(&chunk->mem[i]); + if (length > (u32)offset) { + addr = chunk->buf[i] + offset; goto out; } - offset -= chunk->mem[i].length; + offset -= length; } } out: mutex_unlock(&table->mutex); - return page ? lowmem_page_address(page) + offset : NULL; + return addr; } EXPORT_SYMBOL_GPL(hns_roce_table_find); diff --git a/drivers/infiniband/hw/hns/hns_roce_hem.h b/drivers/infiniband/hw/hns/hns_roce_hem.h index db66db1..e8850d5 100644 --- a/drivers/infiniband/hw/hns/hns_roce_hem.h +++ b/drivers/infiniband/hw/hns/hns_roce_hem.h @@ -78,6 +78,7 @@ struct hns_roce_hem_chunk { int npages; int nsg; struct scatterlist mem[HNS_ROCE_HEM_CHUNK_LEN]; + void *buf[HNS_ROCE_HEM_CHUNK_LEN]; }; struct hns_roce_hem {