From patchwork Fri Aug 9 12:04:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Lucas Stach X-Patchwork-Id: 11086337 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6ECEE1709 for ; Fri, 9 Aug 2019 12:04:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5C57A28C73 for ; Fri, 9 Aug 2019 12:04:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 50A5F28C84; Fri, 9 Aug 2019 12:04:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id D68E728C76 for ; Fri, 9 Aug 2019 12:04:33 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C286289381; Fri, 9 Aug 2019 12:04:29 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from metis.ext.pengutronix.de (metis.ext.pengutronix.de [IPv6:2001:67c:670:201:290:27ff:fe1d:cc33]) by gabe.freedesktop.org (Postfix) with ESMTPS id E81C5894E7 for ; Fri, 9 Aug 2019 12:04:27 +0000 (UTC) Received: from dude02.hi.pengutronix.de ([2001:67c:670:100:1d::28] helo=dude02.pengutronix.de.) by metis.ext.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1hw3dA-00055N-4r; Fri, 09 Aug 2019 14:04:24 +0200 From: Lucas Stach To: etnaviv@lists.freedesktop.org Subject: [PATCH v3 1/8] drm/etnaviv: simplify unbind checks Date: Fri, 9 Aug 2019 14:04:17 +0200 Message-Id: <20190809120424.32679-1-l.stach@pengutronix.de> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-SA-Exim-Connect-IP: 2001:67c:670:100:1d::28 X-SA-Exim-Mail-From: l.stach@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: dri-devel@lists.freedesktop.org X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: patchwork-lst@pengutronix.de, kernel@pengutronix.de, dri-devel@lists.freedesktop.org, Russell King Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" X-Virus-Scanned: ClamAV using ClamSMTP Remember if the GPU has been sucessfully initialized. Only in that case do we need to clean up various structures in the unbind path. If the GPU hasn't been sucessfully initialized all the cleanups should happen in the failure paths of the init function. Signed-off-by: Lucas Stach Reviewed-by: Philipp Zabel Reviewed-by: Guido Günther --- v3: dropped superfluous whitespace --- drivers/gpu/drm/etnaviv/etnaviv_gpu.c | 19 ++++++------------- drivers/gpu/drm/etnaviv/etnaviv_gpu.h | 1 + 2 files changed, 7 insertions(+), 13 deletions(-) diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gpu.c b/drivers/gpu/drm/etnaviv/etnaviv_gpu.c index 4822549500ee..37f7dd69864e 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gpu.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gpu.c @@ -799,17 +799,16 @@ int etnaviv_gpu_init(struct etnaviv_gpu *gpu) pm_runtime_mark_last_busy(gpu->dev); pm_runtime_put_autosuspend(gpu->dev); + gpu->initialized = true; + return 0; free_buffer: etnaviv_cmdbuf_free(&gpu->buffer); - gpu->buffer.suballoc = NULL; destroy_suballoc: etnaviv_cmdbuf_suballoc_destroy(gpu->cmdbuf_suballoc); - gpu->cmdbuf_suballoc = NULL; destroy_iommu: etnaviv_iommu_destroy(gpu->mmu); - gpu->mmu = NULL; fail: pm_runtime_mark_last_busy(gpu->dev); pm_runtime_put_autosuspend(gpu->dev); @@ -1521,7 +1520,7 @@ int etnaviv_gpu_wait_idle(struct etnaviv_gpu *gpu, unsigned int timeout_ms) static int etnaviv_gpu_hw_suspend(struct etnaviv_gpu *gpu) { - if (gpu->buffer.suballoc) { + if (gpu->initialized) { /* Replace the last WAIT with END */ mutex_lock(&gpu->lock); etnaviv_buffer_end(gpu); @@ -1680,17 +1679,11 @@ static void etnaviv_gpu_unbind(struct device *dev, struct device *master, etnaviv_gpu_hw_suspend(gpu); #endif - if (gpu->buffer.suballoc) + if (gpu->initialized) { etnaviv_cmdbuf_free(&gpu->buffer); - - if (gpu->cmdbuf_suballoc) { etnaviv_cmdbuf_suballoc_destroy(gpu->cmdbuf_suballoc); - gpu->cmdbuf_suballoc = NULL; - } - - if (gpu->mmu) { etnaviv_iommu_destroy(gpu->mmu); - gpu->mmu = NULL; + gpu->initialized = false; } gpu->drm = NULL; @@ -1827,7 +1820,7 @@ static int etnaviv_gpu_rpm_resume(struct device *dev) return ret; /* Re-initialise the basic hardware state */ - if (gpu->drm && gpu->buffer.suballoc) { + if (gpu->drm && gpu->initialized) { ret = etnaviv_gpu_hw_resume(gpu); if (ret) { etnaviv_gpu_clk_disable(gpu); diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gpu.h b/drivers/gpu/drm/etnaviv/etnaviv_gpu.h index 9bcf151f706b..b06c7c98d522 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gpu.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_gpu.h @@ -99,6 +99,7 @@ struct etnaviv_gpu { enum etnaviv_sec_mode sec_mode; struct workqueue_struct *wq; struct drm_gpu_scheduler sched; + bool initialized; /* 'ring'-buffer: */ struct etnaviv_cmdbuf buffer; From patchwork Fri Aug 9 12:04:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Lucas Stach X-Patchwork-Id: 11086333 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 59E8C112C for ; Fri, 9 Aug 2019 12:04:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4619328C73 for ; Fri, 9 Aug 2019 12:04:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 39F0428C84; Fri, 9 Aug 2019 12:04:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 5EA7528C73 for ; Fri, 9 Aug 2019 12:04:40 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 725CE6EDCA; Fri, 9 Aug 2019 12:04:32 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from metis.ext.pengutronix.de (metis.ext.pengutronix.de [IPv6:2001:67c:670:201:290:27ff:fe1d:cc33]) by gabe.freedesktop.org (Postfix) with ESMTPS id 57DC3897F3 for ; Fri, 9 Aug 2019 12:04:28 +0000 (UTC) Received: from dude02.hi.pengutronix.de ([2001:67c:670:100:1d::28] helo=dude02.pengutronix.de.) by metis.ext.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1hw3dA-00055N-7L; Fri, 09 Aug 2019 14:04:24 +0200 From: Lucas Stach To: etnaviv@lists.freedesktop.org Subject: [PATCH v3 2/8] drm/etnaviv: split out cmdbuf mapping into address space Date: Fri, 9 Aug 2019 14:04:18 +0200 Message-Id: <20190809120424.32679-2-l.stach@pengutronix.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190809120424.32679-1-l.stach@pengutronix.de> References: <20190809120424.32679-1-l.stach@pengutronix.de> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 2001:67c:670:100:1d::28 X-SA-Exim-Mail-From: l.stach@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: dri-devel@lists.freedesktop.org X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: patchwork-lst@pengutronix.de, kernel@pengutronix.de, dri-devel@lists.freedesktop.org, Russell King Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" X-Virus-Scanned: ClamAV using ClamSMTP This allows to decouple the cmdbuf suballocator create and mapping the region into the GPU address space. Allowing multiple AS to share a single cmdbuf suballoc. Signed-off-by: Lucas Stach Reviewed-by: Philipp Zabel Reviewed-by: Guido Günther --- v3: moved adding mapping to MMU mmaping lsit to common path --- drivers/gpu/drm/etnaviv/etnaviv_buffer.c | 23 ++++---- drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.c | 35 ++++++------ drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.h | 11 +++- drivers/gpu/drm/etnaviv/etnaviv_dump.c | 6 ++- drivers/gpu/drm/etnaviv/etnaviv_gpu.c | 19 +++++-- drivers/gpu/drm/etnaviv/etnaviv_gpu.h | 3 +- drivers/gpu/drm/etnaviv/etnaviv_mmu.c | 69 +++++++++++++++--------- drivers/gpu/drm/etnaviv/etnaviv_mmu.h | 12 ++--- 8 files changed, 113 insertions(+), 65 deletions(-) diff --git a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c index fe0d2d67007d..6400a88cd778 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c @@ -118,7 +118,8 @@ static void etnaviv_buffer_dump(struct etnaviv_gpu *gpu, u32 *ptr = buf->vaddr + off; dev_info(gpu->dev, "virt %p phys 0x%08x free 0x%08x\n", - ptr, etnaviv_cmdbuf_get_va(buf) + off, size - len * 4 - off); + ptr, etnaviv_cmdbuf_get_va(buf, &gpu->cmdbuf_mapping) + + off, size - len * 4 - off); print_hex_dump(KERN_INFO, "cmd ", DUMP_PREFIX_OFFSET, 16, 4, ptr, len * 4, 0); @@ -151,7 +152,8 @@ static u32 etnaviv_buffer_reserve(struct etnaviv_gpu *gpu, if (buffer->user_size + cmd_dwords * sizeof(u64) > buffer->size) buffer->user_size = 0; - return etnaviv_cmdbuf_get_va(buffer) + buffer->user_size; + return etnaviv_cmdbuf_get_va(buffer, &gpu->cmdbuf_mapping) + + buffer->user_size; } u16 etnaviv_buffer_init(struct etnaviv_gpu *gpu) @@ -164,8 +166,8 @@ u16 etnaviv_buffer_init(struct etnaviv_gpu *gpu) buffer->user_size = 0; CMD_WAIT(buffer); - CMD_LINK(buffer, 2, etnaviv_cmdbuf_get_va(buffer) + - buffer->user_size - 4); + CMD_LINK(buffer, 2, etnaviv_cmdbuf_get_va(buffer, &gpu->cmdbuf_mapping) + + buffer->user_size - 4); return buffer->user_size / 8; } @@ -291,8 +293,8 @@ void etnaviv_sync_point_queue(struct etnaviv_gpu *gpu, unsigned int event) /* Append waitlink */ CMD_WAIT(buffer); - CMD_LINK(buffer, 2, etnaviv_cmdbuf_get_va(buffer) + - buffer->user_size - 4); + CMD_LINK(buffer, 2, etnaviv_cmdbuf_get_va(buffer, &gpu->cmdbuf_mapping) + + buffer->user_size - 4); /* * Kick off the 'sync point' command by replacing the previous @@ -319,7 +321,7 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, if (drm_debug & DRM_UT_DRIVER) etnaviv_buffer_dump(gpu, buffer, 0, 0x50); - link_target = etnaviv_cmdbuf_get_va(cmdbuf); + link_target = etnaviv_cmdbuf_get_va(cmdbuf, &gpu->cmdbuf_mapping); link_dwords = cmdbuf->size / 8; /* @@ -412,12 +414,13 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, CMD_LOAD_STATE(buffer, VIVS_GL_EVENT, VIVS_GL_EVENT_EVENT_ID(event) | VIVS_GL_EVENT_FROM_PE); CMD_WAIT(buffer); - CMD_LINK(buffer, 2, etnaviv_cmdbuf_get_va(buffer) + - buffer->user_size - 4); + CMD_LINK(buffer, 2, etnaviv_cmdbuf_get_va(buffer, &gpu->cmdbuf_mapping) + + buffer->user_size - 4); if (drm_debug & DRM_UT_DRIVER) pr_info("stream link to 0x%08x @ 0x%08x %p\n", - return_target, etnaviv_cmdbuf_get_va(cmdbuf), + return_target, + etnaviv_cmdbuf_get_va(cmdbuf, &gpu->cmdbuf_mapping), cmdbuf->vaddr); if (drm_debug & DRM_UT_DRIVER) { diff --git a/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.c b/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.c index 4d0f95452cf0..e842d988edd4 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.c @@ -8,6 +8,7 @@ #include #include "etnaviv_cmdbuf.h" +#include "etnaviv_gem.h" #include "etnaviv_gpu.h" #include "etnaviv_mmu.h" #include "etnaviv_perfmon.h" @@ -22,10 +23,6 @@ struct etnaviv_cmdbuf_suballoc { void *vaddr; dma_addr_t paddr; - /* GPU mapping */ - u32 iova; - struct drm_mm_node vram_node; /* only used on MMUv2 */ - /* allocation management */ struct mutex lock; DECLARE_BITMAP(granule_map, SUBALLOC_GRANULES); @@ -54,26 +51,31 @@ etnaviv_cmdbuf_suballoc_new(struct etnaviv_gpu * gpu) goto free_suballoc; } - ret = etnaviv_iommu_get_suballoc_va(gpu, suballoc->paddr, - &suballoc->vram_node, SUBALLOC_SIZE, - &suballoc->iova); - if (ret) - goto free_dma; - return suballoc; -free_dma: - dma_free_wc(gpu->dev, SUBALLOC_SIZE, suballoc->vaddr, suballoc->paddr); free_suballoc: kfree(suballoc); return ERR_PTR(ret); } +int etnaviv_cmdbuf_suballoc_map(struct etnaviv_cmdbuf_suballoc *suballoc, + struct etnaviv_iommu *mmu, + struct etnaviv_vram_mapping *mapping, + u32 memory_base) +{ + return etnaviv_iommu_get_suballoc_va(mmu, mapping, memory_base, + suballoc->paddr, SUBALLOC_SIZE); +} + +void etnaviv_cmdbuf_suballoc_unmap(struct etnaviv_iommu *mmu, + struct etnaviv_vram_mapping *mapping) +{ + etnaviv_iommu_put_suballoc_va(mmu, mapping); +} + void etnaviv_cmdbuf_suballoc_destroy(struct etnaviv_cmdbuf_suballoc *suballoc) { - etnaviv_iommu_put_suballoc_va(suballoc->gpu, &suballoc->vram_node, - SUBALLOC_SIZE, suballoc->iova); dma_free_wc(suballoc->gpu->dev, SUBALLOC_SIZE, suballoc->vaddr, suballoc->paddr); kfree(suballoc); @@ -127,9 +129,10 @@ void etnaviv_cmdbuf_free(struct etnaviv_cmdbuf *cmdbuf) wake_up_all(&suballoc->free_event); } -u32 etnaviv_cmdbuf_get_va(struct etnaviv_cmdbuf *buf) +u32 etnaviv_cmdbuf_get_va(struct etnaviv_cmdbuf *buf, + struct etnaviv_vram_mapping *mapping) { - return buf->suballoc->iova + buf->suballoc_offset; + return mapping->iova + buf->suballoc_offset; } dma_addr_t etnaviv_cmdbuf_get_pa(struct etnaviv_cmdbuf *buf) diff --git a/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.h b/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.h index 4d5d1a77eb2a..583f56bf011c 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.h @@ -9,6 +9,8 @@ #include struct etnaviv_gpu; +struct etnaviv_iommu; +struct etnaviv_vram_mapping; struct etnaviv_cmdbuf_suballoc; struct etnaviv_perfmon_request; @@ -25,13 +27,20 @@ struct etnaviv_cmdbuf { struct etnaviv_cmdbuf_suballoc * etnaviv_cmdbuf_suballoc_new(struct etnaviv_gpu * gpu); void etnaviv_cmdbuf_suballoc_destroy(struct etnaviv_cmdbuf_suballoc *suballoc); +int etnaviv_cmdbuf_suballoc_map(struct etnaviv_cmdbuf_suballoc *suballoc, + struct etnaviv_iommu *mmu, + struct etnaviv_vram_mapping *mapping, + u32 memory_base); +void etnaviv_cmdbuf_suballoc_unmap(struct etnaviv_iommu *mmu, + struct etnaviv_vram_mapping *mapping); int etnaviv_cmdbuf_init(struct etnaviv_cmdbuf_suballoc *suballoc, struct etnaviv_cmdbuf *cmdbuf, u32 size); void etnaviv_cmdbuf_free(struct etnaviv_cmdbuf *cmdbuf); -u32 etnaviv_cmdbuf_get_va(struct etnaviv_cmdbuf *buf); +u32 etnaviv_cmdbuf_get_va(struct etnaviv_cmdbuf *buf, + struct etnaviv_vram_mapping *mapping); dma_addr_t etnaviv_cmdbuf_get_pa(struct etnaviv_cmdbuf *buf); #endif /* __ETNAVIV_CMDBUF_H__ */ diff --git a/drivers/gpu/drm/etnaviv/etnaviv_dump.c b/drivers/gpu/drm/etnaviv/etnaviv_dump.c index cc808b0941c2..2ce60baa4ad9 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_dump.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_dump.c @@ -172,11 +172,13 @@ void etnaviv_core_dump(struct etnaviv_gem_submit *submit) etnaviv_core_dump_mmu(&iter, gpu->mmu, mmu_size); etnaviv_core_dump_mem(&iter, ETDUMP_BUF_RING, gpu->buffer.vaddr, gpu->buffer.size, - etnaviv_cmdbuf_get_va(&gpu->buffer)); + etnaviv_cmdbuf_get_va(&gpu->buffer, + &gpu->cmdbuf_mapping)); etnaviv_core_dump_mem(&iter, ETDUMP_BUF_CMD, submit->cmdbuf.vaddr, submit->cmdbuf.size, - etnaviv_cmdbuf_get_va(&submit->cmdbuf)); + etnaviv_cmdbuf_get_va(&submit->cmdbuf, + &gpu->cmdbuf_mapping)); /* Reserve space for the bomap */ if (n_bomap_pages) { diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gpu.c b/drivers/gpu/drm/etnaviv/etnaviv_gpu.c index 37f7dd69864e..f00547b88a13 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gpu.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gpu.c @@ -687,8 +687,8 @@ static void etnaviv_gpu_hw_init(struct etnaviv_gpu *gpu) prefetch = etnaviv_buffer_init(gpu); gpu_write(gpu, VIVS_HI_INTR_ENBL, ~0U); - etnaviv_gpu_start_fe(gpu, etnaviv_cmdbuf_get_va(&gpu->buffer), - prefetch); + etnaviv_gpu_start_fe(gpu, etnaviv_cmdbuf_get_va(&gpu->buffer, + &gpu->cmdbuf_mapping), prefetch); } int etnaviv_gpu_init(struct etnaviv_gpu *gpu) @@ -767,16 +767,24 @@ int etnaviv_gpu_init(struct etnaviv_gpu *gpu) goto destroy_iommu; } + ret = etnaviv_cmdbuf_suballoc_map(gpu->cmdbuf_suballoc, gpu->mmu, + &gpu->cmdbuf_mapping, + gpu->memory_base); + if (ret) { + dev_err(gpu->dev, "failed to map cmdbuf suballoc\n"); + goto destroy_suballoc; + } + /* Create buffer: */ ret = etnaviv_cmdbuf_init(gpu->cmdbuf_suballoc, &gpu->buffer, PAGE_SIZE); if (ret) { dev_err(gpu->dev, "could not create command buffer\n"); - goto destroy_suballoc; + goto unmap_suballoc; } if (gpu->mmu->version == ETNAVIV_IOMMU_V1 && - etnaviv_cmdbuf_get_va(&gpu->buffer) > 0x80000000) { + etnaviv_cmdbuf_get_va(&gpu->buffer, &gpu->cmdbuf_mapping) > 0x80000000) { ret = -EINVAL; dev_err(gpu->dev, "command buffer outside valid memory window\n"); @@ -805,6 +813,8 @@ int etnaviv_gpu_init(struct etnaviv_gpu *gpu) free_buffer: etnaviv_cmdbuf_free(&gpu->buffer); +unmap_suballoc: + etnaviv_cmdbuf_suballoc_unmap(gpu->mmu, &gpu->cmdbuf_mapping); destroy_suballoc: etnaviv_cmdbuf_suballoc_destroy(gpu->cmdbuf_suballoc); destroy_iommu: @@ -1681,6 +1691,7 @@ static void etnaviv_gpu_unbind(struct device *dev, struct device *master, if (gpu->initialized) { etnaviv_cmdbuf_free(&gpu->buffer); + etnaviv_cmdbuf_suballoc_unmap(gpu->mmu, &gpu->cmdbuf_mapping); etnaviv_cmdbuf_suballoc_destroy(gpu->cmdbuf_suballoc); etnaviv_iommu_destroy(gpu->mmu); gpu->initialized = false; diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gpu.h b/drivers/gpu/drm/etnaviv/etnaviv_gpu.h index b06c7c98d522..6a6add350d2d 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gpu.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_gpu.h @@ -7,6 +7,7 @@ #define __ETNAVIV_GPU_H__ #include "etnaviv_cmdbuf.h" +#include "etnaviv_gem.h" #include "etnaviv_drv.h" struct etnaviv_gem_submit; @@ -84,7 +85,6 @@ struct etnaviv_event { }; struct etnaviv_cmdbuf_suballoc; -struct etnaviv_cmdbuf; struct regulator; struct clk; @@ -102,6 +102,7 @@ struct etnaviv_gpu { bool initialized; /* 'ring'-buffer: */ + struct etnaviv_vram_mapping cmdbuf_mapping; struct etnaviv_cmdbuf buffer; int exec_state; diff --git a/drivers/gpu/drm/etnaviv/etnaviv_mmu.c b/drivers/gpu/drm/etnaviv/etnaviv_mmu.c index 731275999a57..09f516093b91 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_mmu.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_mmu.c @@ -334,52 +334,71 @@ void etnaviv_iommu_restore(struct etnaviv_gpu *gpu) etnaviv_iommuv2_restore(gpu); } -int etnaviv_iommu_get_suballoc_va(struct etnaviv_gpu *gpu, dma_addr_t paddr, - struct drm_mm_node *vram_node, size_t size, - u32 *iova) +int etnaviv_iommu_get_suballoc_va(struct etnaviv_iommu *mmu, + struct etnaviv_vram_mapping *mapping, + u32 memory_base, dma_addr_t paddr, + size_t size) { - struct etnaviv_iommu *mmu = gpu->mmu; + mutex_lock(&mmu->lock); + /* + * For MMUv1 we don't add the suballoc region to the pagetables, as + * those GPUs can only work with cmdbufs accessed through the linear + * window. Instead we manufacture a mapping to make it look uniform + * to the upper layers. + */ if (mmu->version == ETNAVIV_IOMMU_V1) { - *iova = paddr - gpu->memory_base; - return 0; + mapping->iova = paddr - memory_base; } else { + struct drm_mm_node *node = &mapping->vram_node; int ret; - mutex_lock(&mmu->lock); - ret = etnaviv_iommu_find_iova(mmu, vram_node, size); + ret = etnaviv_iommu_find_iova(mmu, node, size); if (ret < 0) { mutex_unlock(&mmu->lock); return ret; } - ret = etnaviv_domain_map(mmu->domain, vram_node->start, paddr, - size, ETNAVIV_PROT_READ); + + mapping->iova = node->start; + ret = etnaviv_domain_map(mmu->domain, node->start, paddr, size, + ETNAVIV_PROT_READ); + if (ret < 0) { - drm_mm_remove_node(vram_node); + drm_mm_remove_node(node); mutex_unlock(&mmu->lock); return ret; } - gpu->mmu->need_flush = true; - mutex_unlock(&mmu->lock); - *iova = (u32)vram_node->start; - return 0; + mmu->need_flush = true; } + + list_add_tail(&mapping->mmu_node, &mmu->mappings); + mapping->use = 1; + + mutex_unlock(&mmu->lock); + + return 0; } -void etnaviv_iommu_put_suballoc_va(struct etnaviv_gpu *gpu, - struct drm_mm_node *vram_node, size_t size, - u32 iova) +void etnaviv_iommu_put_suballoc_va(struct etnaviv_iommu *mmu, + struct etnaviv_vram_mapping *mapping) { - struct etnaviv_iommu *mmu = gpu->mmu; + struct drm_mm_node *node = &mapping->vram_node; - if (mmu->version == ETNAVIV_IOMMU_V2) { - mutex_lock(&mmu->lock); - etnaviv_domain_unmap(mmu->domain, iova, size); - drm_mm_remove_node(vram_node); - mutex_unlock(&mmu->lock); - } + if (!mapping->use) + return; + + mapping->use = 0; + + if (mmu->version == ETNAVIV_IOMMU_V1) + return; + + mutex_lock(&mmu->lock); + etnaviv_domain_unmap(mmu->domain, node->start, node->size); + drm_mm_remove_node(node); + mutex_unlock(&mmu->lock); } + size_t etnaviv_iommu_dump_size(struct etnaviv_iommu *iommu) { return iommu->domain->ops->dump_size(iommu->domain); diff --git a/drivers/gpu/drm/etnaviv/etnaviv_mmu.h b/drivers/gpu/drm/etnaviv/etnaviv_mmu.h index a0db17ffb686..fe1c9d6b9334 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_mmu.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_mmu.h @@ -59,12 +59,12 @@ int etnaviv_iommu_map_gem(struct etnaviv_iommu *mmu, void etnaviv_iommu_unmap_gem(struct etnaviv_iommu *mmu, struct etnaviv_vram_mapping *mapping); -int etnaviv_iommu_get_suballoc_va(struct etnaviv_gpu *gpu, dma_addr_t paddr, - struct drm_mm_node *vram_node, size_t size, - u32 *iova); -void etnaviv_iommu_put_suballoc_va(struct etnaviv_gpu *gpu, - struct drm_mm_node *vram_node, size_t size, - u32 iova); +int etnaviv_iommu_get_suballoc_va(struct etnaviv_iommu *mmu, + struct etnaviv_vram_mapping *mapping, + u32 memory_base, dma_addr_t paddr, + size_t size); +void etnaviv_iommu_put_suballoc_va(struct etnaviv_iommu *mmu, + struct etnaviv_vram_mapping *mapping); size_t etnaviv_iommu_dump_size(struct etnaviv_iommu *iommu); void etnaviv_iommu_dump(struct etnaviv_iommu *iommu, void *buf); From patchwork Fri Aug 9 12:04:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Lucas Stach X-Patchwork-Id: 11086327 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D41181399 for ; Fri, 9 Aug 2019 12:04:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C1A9328C75 for ; Fri, 9 Aug 2019 12:04:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B49F428C73; Fri, 9 Aug 2019 12:04:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 284ED28C73 for ; Fri, 9 Aug 2019 12:04:31 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 6D43489341; Fri, 9 Aug 2019 12:04:29 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from metis.ext.pengutronix.de (metis.ext.pengutronix.de [IPv6:2001:67c:670:201:290:27ff:fe1d:cc33]) by gabe.freedesktop.org (Postfix) with ESMTPS id 0825D89565 for ; Fri, 9 Aug 2019 12:04:28 +0000 (UTC) Received: from dude02.hi.pengutronix.de ([2001:67c:670:100:1d::28] helo=dude02.pengutronix.de.) by metis.ext.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1hw3dA-00055N-9s; Fri, 09 Aug 2019 14:04:24 +0200 From: Lucas Stach To: etnaviv@lists.freedesktop.org Subject: [PATCH v3 3/8] drm/etnaviv: share a single cmdbuf suballoc region across all GPUs Date: Fri, 9 Aug 2019 14:04:19 +0200 Message-Id: <20190809120424.32679-3-l.stach@pengutronix.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190809120424.32679-1-l.stach@pengutronix.de> References: <20190809120424.32679-1-l.stach@pengutronix.de> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 2001:67c:670:100:1d::28 X-SA-Exim-Mail-From: l.stach@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: dri-devel@lists.freedesktop.org X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: patchwork-lst@pengutronix.de, kernel@pengutronix.de, dri-devel@lists.freedesktop.org, Russell King Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" X-Virus-Scanned: ClamAV using ClamSMTP There is no need for each GPU to have it's own cmdbuf suballocation region. Only allocate a single one for the the etnaviv virtual device and share it across all GPUs. As the suballoc space is now potentially shared by more hardware jobs running in parallel, double its size to 512KB to avoid contention. Signed-off-by: Lucas Stach Reviewed-by: Philipp Zabel Reviewed-by: Guido Günther --- drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.c | 14 +++++++------- drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.h | 4 ++-- drivers/gpu/drm/etnaviv/etnaviv_drv.c | 19 +++++++++++++++---- drivers/gpu/drm/etnaviv/etnaviv_drv.h | 2 ++ drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c | 2 +- drivers/gpu/drm/etnaviv/etnaviv_gpu.c | 17 ++++------------- drivers/gpu/drm/etnaviv/etnaviv_gpu.h | 1 - 7 files changed, 31 insertions(+), 28 deletions(-) diff --git a/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.c b/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.c index e842d988edd4..b0babc0f7230 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.c @@ -13,13 +13,13 @@ #include "etnaviv_mmu.h" #include "etnaviv_perfmon.h" -#define SUBALLOC_SIZE SZ_256K +#define SUBALLOC_SIZE SZ_512K #define SUBALLOC_GRANULE SZ_4K #define SUBALLOC_GRANULES (SUBALLOC_SIZE / SUBALLOC_GRANULE) struct etnaviv_cmdbuf_suballoc { /* suballocated dma buffer properties */ - struct etnaviv_gpu *gpu; + struct device *dev; void *vaddr; dma_addr_t paddr; @@ -31,7 +31,7 @@ struct etnaviv_cmdbuf_suballoc { }; struct etnaviv_cmdbuf_suballoc * -etnaviv_cmdbuf_suballoc_new(struct etnaviv_gpu * gpu) +etnaviv_cmdbuf_suballoc_new(struct device *dev) { struct etnaviv_cmdbuf_suballoc *suballoc; int ret; @@ -40,11 +40,11 @@ etnaviv_cmdbuf_suballoc_new(struct etnaviv_gpu * gpu) if (!suballoc) return ERR_PTR(-ENOMEM); - suballoc->gpu = gpu; + suballoc->dev = dev; mutex_init(&suballoc->lock); init_waitqueue_head(&suballoc->free_event); - suballoc->vaddr = dma_alloc_wc(gpu->dev, SUBALLOC_SIZE, + suballoc->vaddr = dma_alloc_wc(dev, SUBALLOC_SIZE, &suballoc->paddr, GFP_KERNEL); if (!suballoc->vaddr) { ret = -ENOMEM; @@ -76,7 +76,7 @@ void etnaviv_cmdbuf_suballoc_unmap(struct etnaviv_iommu *mmu, void etnaviv_cmdbuf_suballoc_destroy(struct etnaviv_cmdbuf_suballoc *suballoc) { - dma_free_wc(suballoc->gpu->dev, SUBALLOC_SIZE, suballoc->vaddr, + dma_free_wc(suballoc->dev, SUBALLOC_SIZE, suballoc->vaddr, suballoc->paddr); kfree(suballoc); } @@ -101,7 +101,7 @@ int etnaviv_cmdbuf_init(struct etnaviv_cmdbuf_suballoc *suballoc, suballoc->free_space, msecs_to_jiffies(10 * 1000)); if (!ret) { - dev_err(suballoc->gpu->dev, + dev_err(suballoc->dev, "Timeout waiting for cmdbuf space\n"); return -ETIMEDOUT; } diff --git a/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.h b/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.h index 583f56bf011c..a28668e46e26 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.h @@ -8,7 +8,7 @@ #include -struct etnaviv_gpu; +struct device; struct etnaviv_iommu; struct etnaviv_vram_mapping; struct etnaviv_cmdbuf_suballoc; @@ -25,7 +25,7 @@ struct etnaviv_cmdbuf { }; struct etnaviv_cmdbuf_suballoc * -etnaviv_cmdbuf_suballoc_new(struct etnaviv_gpu * gpu); +etnaviv_cmdbuf_suballoc_new(struct device *dev); void etnaviv_cmdbuf_suballoc_destroy(struct etnaviv_cmdbuf_suballoc *suballoc); int etnaviv_cmdbuf_suballoc_map(struct etnaviv_cmdbuf_suballoc *suballoc, struct etnaviv_iommu *mmu, diff --git a/drivers/gpu/drm/etnaviv/etnaviv_drv.c b/drivers/gpu/drm/etnaviv/etnaviv_drv.c index 9d4404723489..5fa3aa7bdbc5 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_drv.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_drv.c @@ -530,23 +530,32 @@ static int etnaviv_bind(struct device *dev) INIT_LIST_HEAD(&priv->gem_list); priv->num_gpus = 0; + priv->cmdbuf_suballoc = etnaviv_cmdbuf_suballoc_new(drm->dev); + if (IS_ERR(priv->cmdbuf_suballoc)) { + dev_err(drm->dev, "Failed to create cmdbuf suballocator\n"); + ret = PTR_ERR(priv->cmdbuf_suballoc); + goto out_free_priv; + } + dev_set_drvdata(dev, drm); ret = component_bind_all(dev, drm); if (ret < 0) - goto out_bind; + goto out_destroy_suballoc; load_gpu(drm); ret = drm_dev_register(drm, 0); if (ret) - goto out_register; + goto out_unbind; return 0; -out_register: +out_unbind: component_unbind_all(dev, drm); -out_bind: +out_destroy_suballoc: + etnaviv_cmdbuf_suballoc_destroy(priv->cmdbuf_suballoc); +out_free_priv: kfree(priv); out_put: drm_dev_put(drm); @@ -565,6 +574,8 @@ static void etnaviv_unbind(struct device *dev) dev->dma_parms = NULL; + etnaviv_cmdbuf_suballoc_destroy(priv->cmdbuf_suballoc); + drm->dev_private = NULL; kfree(priv); diff --git a/drivers/gpu/drm/etnaviv/etnaviv_drv.h b/drivers/gpu/drm/etnaviv/etnaviv_drv.h index eabe394c4e25..e052d7db66ae 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_drv.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_drv.h @@ -36,6 +36,8 @@ struct etnaviv_drm_private { struct device_dma_parameters dma_parms; struct etnaviv_gpu *gpu[ETNA_MAX_PIPES]; + struct etnaviv_cmdbuf_suballoc *cmdbuf_suballoc; + /* list of GEM objects: */ struct mutex gem_lock; struct list_head gem_list; diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c index f535a627f297..3f4f6ab388de 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c @@ -496,7 +496,7 @@ int etnaviv_ioctl_gem_submit(struct drm_device *dev, void *data, goto err_submit_ww_acquire; } - ret = etnaviv_cmdbuf_init(gpu->cmdbuf_suballoc, &submit->cmdbuf, + ret = etnaviv_cmdbuf_init(priv->cmdbuf_suballoc, &submit->cmdbuf, ALIGN(args->stream_size, 8) + 8); if (ret) goto err_submit_objects; diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gpu.c b/drivers/gpu/drm/etnaviv/etnaviv_gpu.c index f00547b88a13..179bc6c544ca 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gpu.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gpu.c @@ -693,6 +693,7 @@ static void etnaviv_gpu_hw_init(struct etnaviv_gpu *gpu) int etnaviv_gpu_init(struct etnaviv_gpu *gpu) { + struct etnaviv_drm_private *priv = gpu->drm->dev_private; int ret, i; ret = pm_runtime_get_sync(gpu->dev); @@ -760,23 +761,16 @@ int etnaviv_gpu_init(struct etnaviv_gpu *gpu) goto fail; } - gpu->cmdbuf_suballoc = etnaviv_cmdbuf_suballoc_new(gpu); - if (IS_ERR(gpu->cmdbuf_suballoc)) { - dev_err(gpu->dev, "Failed to create cmdbuf suballocator\n"); - ret = PTR_ERR(gpu->cmdbuf_suballoc); - goto destroy_iommu; - } - - ret = etnaviv_cmdbuf_suballoc_map(gpu->cmdbuf_suballoc, gpu->mmu, + ret = etnaviv_cmdbuf_suballoc_map(priv->cmdbuf_suballoc, gpu->mmu, &gpu->cmdbuf_mapping, gpu->memory_base); if (ret) { dev_err(gpu->dev, "failed to map cmdbuf suballoc\n"); - goto destroy_suballoc; + goto destroy_iommu; } /* Create buffer: */ - ret = etnaviv_cmdbuf_init(gpu->cmdbuf_suballoc, &gpu->buffer, + ret = etnaviv_cmdbuf_init(priv->cmdbuf_suballoc, &gpu->buffer, PAGE_SIZE); if (ret) { dev_err(gpu->dev, "could not create command buffer\n"); @@ -815,8 +809,6 @@ int etnaviv_gpu_init(struct etnaviv_gpu *gpu) etnaviv_cmdbuf_free(&gpu->buffer); unmap_suballoc: etnaviv_cmdbuf_suballoc_unmap(gpu->mmu, &gpu->cmdbuf_mapping); -destroy_suballoc: - etnaviv_cmdbuf_suballoc_destroy(gpu->cmdbuf_suballoc); destroy_iommu: etnaviv_iommu_destroy(gpu->mmu); fail: @@ -1692,7 +1684,6 @@ static void etnaviv_gpu_unbind(struct device *dev, struct device *master, if (gpu->initialized) { etnaviv_cmdbuf_free(&gpu->buffer); etnaviv_cmdbuf_suballoc_unmap(gpu->mmu, &gpu->cmdbuf_mapping); - etnaviv_cmdbuf_suballoc_destroy(gpu->cmdbuf_suballoc); etnaviv_iommu_destroy(gpu->mmu); gpu->initialized = false; } diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gpu.h b/drivers/gpu/drm/etnaviv/etnaviv_gpu.h index 6a6add350d2d..933c8d016f11 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gpu.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_gpu.h @@ -137,7 +137,6 @@ struct etnaviv_gpu { int irq; struct etnaviv_iommu *mmu; - struct etnaviv_cmdbuf_suballoc *cmdbuf_suballoc; /* Power Control: */ struct clk *clk_bus; From patchwork Fri Aug 9 12:04:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Lucas Stach X-Patchwork-Id: 11086331 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 43E331399 for ; Fri, 9 Aug 2019 12:04:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 32D1528C73 for ; Fri, 9 Aug 2019 12:04:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2763C28C84; Fri, 9 Aug 2019 12:04:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id BEBD828C73 for ; Fri, 9 Aug 2019 12:04:37 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 6A87F6EDEC; Fri, 9 Aug 2019 12:04:30 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from metis.ext.pengutronix.de (metis.ext.pengutronix.de [IPv6:2001:67c:670:201:290:27ff:fe1d:cc33]) by gabe.freedesktop.org (Postfix) with ESMTPS id 660258989A for ; Fri, 9 Aug 2019 12:04:28 +0000 (UTC) Received: from dude02.hi.pengutronix.de ([2001:67c:670:100:1d::28] helo=dude02.pengutronix.de.) by metis.ext.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1hw3dA-00055N-Ac; Fri, 09 Aug 2019 14:04:24 +0200 From: Lucas Stach To: etnaviv@lists.freedesktop.org Subject: [PATCH v3 4/8] drm/etnaviv: replace MMU flush marker with flush sequence Date: Fri, 9 Aug 2019 14:04:20 +0200 Message-Id: <20190809120424.32679-4-l.stach@pengutronix.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190809120424.32679-1-l.stach@pengutronix.de> References: <20190809120424.32679-1-l.stach@pengutronix.de> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 2001:67c:670:100:1d::28 X-SA-Exim-Mail-From: l.stach@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: dri-devel@lists.freedesktop.org X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: patchwork-lst@pengutronix.de, kernel@pengutronix.de, dri-devel@lists.freedesktop.org, Russell King Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" X-Virus-Scanned: ClamAV using ClamSMTP If a MMU is shared between multiple GPUs, all of them need to flush their TLBs, so a single marker that gets reset on the first flush won't do. Replace the flush marker with a sequence number, so that it's possible to check if the TLB is in sync with the current page table state for each GPU. Signed-off-by: Lucas Stach Reviewed-by: Philipp Zabel Reviewed-by: Guido Günther --- v3: Avoid flush sequence update racing with the flush. --- drivers/gpu/drm/etnaviv/etnaviv_buffer.c | 10 ++++++---- drivers/gpu/drm/etnaviv/etnaviv_gpu.h | 1 + drivers/gpu/drm/etnaviv/etnaviv_mmu.c | 6 +++--- drivers/gpu/drm/etnaviv/etnaviv_mmu.h | 2 +- 4 files changed, 11 insertions(+), 8 deletions(-) diff --git a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c index 6400a88cd778..a3cdb20bfc5f 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c @@ -315,6 +315,8 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, u32 return_target, return_dwords; u32 link_target, link_dwords; bool switch_context = gpu->exec_state != exec_state; + unsigned int new_flush_seq = READ_ONCE(gpu->mmu->flush_seq); + bool need_flush = gpu->flush_seq != new_flush_seq; lockdep_assert_held(&gpu->lock); @@ -329,14 +331,14 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, * need to append a mmu flush load state, followed by a new * link to this buffer - a total of four additional words. */ - if (gpu->mmu->need_flush || switch_context) { + if (need_flush || switch_context) { u32 target, extra_dwords; /* link command */ extra_dwords = 1; /* flush command */ - if (gpu->mmu->need_flush) { + if (need_flush) { if (gpu->mmu->version == ETNAVIV_IOMMU_V1) extra_dwords += 1; else @@ -349,7 +351,7 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, target = etnaviv_buffer_reserve(gpu, buffer, extra_dwords); - if (gpu->mmu->need_flush) { + if (need_flush) { /* Add the MMU flush */ if (gpu->mmu->version == ETNAVIV_IOMMU_V1) { CMD_LOAD_STATE(buffer, VIVS_GL_FLUSH_MMU, @@ -369,7 +371,7 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, SYNC_RECIPIENT_PE); } - gpu->mmu->need_flush = false; + gpu->flush_seq = new_flush_seq; } if (switch_context) { diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gpu.h b/drivers/gpu/drm/etnaviv/etnaviv_gpu.h index 933c8d016f11..96380942cd8c 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gpu.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_gpu.h @@ -137,6 +137,7 @@ struct etnaviv_gpu { int irq; struct etnaviv_iommu *mmu; + unsigned int flush_seq; /* Power Control: */ struct clk *clk_bus; diff --git a/drivers/gpu/drm/etnaviv/etnaviv_mmu.c b/drivers/gpu/drm/etnaviv/etnaviv_mmu.c index 09f516093b91..bbd1624a3df8 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_mmu.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_mmu.c @@ -263,7 +263,7 @@ int etnaviv_iommu_map_gem(struct etnaviv_iommu *mmu, } list_add_tail(&mapping->mmu_node, &mmu->mappings); - mmu->need_flush = true; + mmu->flush_seq++; unlock: mutex_unlock(&mmu->lock); @@ -282,7 +282,7 @@ void etnaviv_iommu_unmap_gem(struct etnaviv_iommu *mmu, etnaviv_iommu_remove_mapping(mmu, mapping); list_del(&mapping->mmu_node); - mmu->need_flush = true; + mmu->flush_seq++; mutex_unlock(&mmu->lock); } @@ -369,7 +369,7 @@ int etnaviv_iommu_get_suballoc_va(struct etnaviv_iommu *mmu, return ret; } - mmu->need_flush = true; + mmu->flush_seq++; } list_add_tail(&mapping->mmu_node, &mmu->mappings); diff --git a/drivers/gpu/drm/etnaviv/etnaviv_mmu.h b/drivers/gpu/drm/etnaviv/etnaviv_mmu.h index fe1c9d6b9334..34afe25df9ca 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_mmu.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_mmu.h @@ -48,7 +48,7 @@ struct etnaviv_iommu { struct mutex lock; struct list_head mappings; struct drm_mm mm; - bool need_flush; + unsigned int flush_seq; }; struct etnaviv_gem_object; From patchwork Fri Aug 9 12:04:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Lucas Stach X-Patchwork-Id: 11086339 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A1A98112C for ; Fri, 9 Aug 2019 12:04:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8788128C73 for ; Fri, 9 Aug 2019 12:04:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7B93A28C84; Fri, 9 Aug 2019 12:04:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id CC57728C73 for ; Fri, 9 Aug 2019 12:04:44 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 652DA6EDF1; Fri, 9 Aug 2019 12:04:39 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from metis.ext.pengutronix.de (metis.ext.pengutronix.de [IPv6:2001:67c:670:201:290:27ff:fe1d:cc33]) by gabe.freedesktop.org (Postfix) with ESMTPS id 58A6789565 for ; Fri, 9 Aug 2019 12:04:29 +0000 (UTC) Received: from dude02.hi.pengutronix.de ([2001:67c:670:100:1d::28] helo=dude02.pengutronix.de.) by metis.ext.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1hw3dA-00055N-BQ; Fri, 09 Aug 2019 14:04:24 +0200 From: Lucas Stach To: etnaviv@lists.freedesktop.org Subject: [PATCH v3 5/8] drm/etnaviv: rework MMU handling Date: Fri, 9 Aug 2019 14:04:21 +0200 Message-Id: <20190809120424.32679-5-l.stach@pengutronix.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190809120424.32679-1-l.stach@pengutronix.de> References: <20190809120424.32679-1-l.stach@pengutronix.de> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 2001:67c:670:100:1d::28 X-SA-Exim-Mail-From: l.stach@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: dri-devel@lists.freedesktop.org X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: patchwork-lst@pengutronix.de, kernel@pengutronix.de, dri-devel@lists.freedesktop.org, Russell King Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" X-Virus-Scanned: ClamAV using ClamSMTP This reworks the MMU handling to make it possible to have multiple MMU contexts. A context is basically one instance of GPU page tables. Currently we have one set of page tables per GPU, which isn't all that clever, as it has the following two consequences: 1. All GPU clients (aka processes) are sharing the same pagetables, which means there is no isolation between clients, but only between GPU assigned memory spaces and the rest of the system. Better than nothing, but also not great. 2. Clients operating on the same set of buffers with different etnaviv GPU cores, e.g. a workload using both the 2D and 3D GPU, need to map the used buffers into the pagetable sets of each used GPU. This patch reworks all the MMU handling to introduce the abstraction of the MMU context. A context can be shared across different GPU cores, as long as they have compatible MMU implementations, which is the case for all systems with Vivante GPUs seen in the wild. As MMUv1 is not able to change pagetables on the fly, without a "stop the world" operation, which stops GPU, changes pagetables via CPU interaction, restarts GPU, the implementation introduces a shared context on MMUv1, which is returned whenever there is a request for a new context. This patch assigns a MMU context to each GPU, so on MMUv2 systems there is still one set of pagetables per GPU, but due to the shared context MMUv1 systems see a change in behavior as now a single pagetable set is used across all GPU cores. Signed-off-by: Lucas Stach Reviewed-by: Philipp Zabel Reviewed-by: Guido Günther --- v3: - move ops declaration to header. - rename gpu struct mmu member to mmu_context for consistency --- drivers/gpu/drm/etnaviv/etnaviv_buffer.c | 10 +- drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.c | 8 +- drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.h | 6 +- drivers/gpu/drm/etnaviv/etnaviv_drv.c | 6 +- drivers/gpu/drm/etnaviv/etnaviv_drv.h | 4 +- drivers/gpu/drm/etnaviv/etnaviv_dump.c | 12 +- drivers/gpu/drm/etnaviv/etnaviv_gem.c | 24 +- drivers/gpu/drm/etnaviv/etnaviv_gem.h | 2 +- drivers/gpu/drm/etnaviv/etnaviv_gpu.c | 31 ++- drivers/gpu/drm/etnaviv/etnaviv_gpu.h | 3 +- drivers/gpu/drm/etnaviv/etnaviv_iommu.c | 151 ++++++------ drivers/gpu/drm/etnaviv/etnaviv_iommu.h | 20 -- drivers/gpu/drm/etnaviv/etnaviv_iommu_v2.c | 264 +++++++++------------ drivers/gpu/drm/etnaviv/etnaviv_mmu.c | 259 ++++++++++++-------- drivers/gpu/drm/etnaviv/etnaviv_mmu.h | 91 +++++-- 15 files changed, 461 insertions(+), 430 deletions(-) delete mode 100644 drivers/gpu/drm/etnaviv/etnaviv_iommu.h diff --git a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c index a3cdb20bfc5f..4324b098689f 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c @@ -207,7 +207,7 @@ u16 etnaviv_buffer_config_mmuv2(struct etnaviv_gpu *gpu, u32 mtlb_addr, u32 safe return buffer->user_size / 8; } -u16 etnaviv_buffer_config_pta(struct etnaviv_gpu *gpu) +u16 etnaviv_buffer_config_pta(struct etnaviv_gpu *gpu, unsigned short id) { struct etnaviv_cmdbuf *buffer = &gpu->buffer; @@ -216,7 +216,7 @@ u16 etnaviv_buffer_config_pta(struct etnaviv_gpu *gpu) buffer->user_size = 0; CMD_LOAD_STATE(buffer, VIVS_MMUv2_PTA_CONFIG, - VIVS_MMUv2_PTA_CONFIG_INDEX(0)); + VIVS_MMUv2_PTA_CONFIG_INDEX(id)); CMD_END(buffer); @@ -315,7 +315,7 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, u32 return_target, return_dwords; u32 link_target, link_dwords; bool switch_context = gpu->exec_state != exec_state; - unsigned int new_flush_seq = READ_ONCE(gpu->mmu->flush_seq); + unsigned int new_flush_seq = READ_ONCE(gpu->mmu_context->flush_seq); bool need_flush = gpu->flush_seq != new_flush_seq; lockdep_assert_held(&gpu->lock); @@ -339,7 +339,7 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, /* flush command */ if (need_flush) { - if (gpu->mmu->version == ETNAVIV_IOMMU_V1) + if (gpu->mmu_context->global->version == ETNAVIV_IOMMU_V1) extra_dwords += 1; else extra_dwords += 3; @@ -353,7 +353,7 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, if (need_flush) { /* Add the MMU flush */ - if (gpu->mmu->version == ETNAVIV_IOMMU_V1) { + if (gpu->mmu_context->global->version == ETNAVIV_IOMMU_V1) { CMD_LOAD_STATE(buffer, VIVS_GL_FLUSH_MMU, VIVS_GL_FLUSH_MMU_FLUSH_FEMMU | VIVS_GL_FLUSH_MMU_FLUSH_UNK1 | diff --git a/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.c b/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.c index b0babc0f7230..f39430ba3593 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.c @@ -60,18 +60,18 @@ etnaviv_cmdbuf_suballoc_new(struct device *dev) } int etnaviv_cmdbuf_suballoc_map(struct etnaviv_cmdbuf_suballoc *suballoc, - struct etnaviv_iommu *mmu, + struct etnaviv_iommu_context *context, struct etnaviv_vram_mapping *mapping, u32 memory_base) { - return etnaviv_iommu_get_suballoc_va(mmu, mapping, memory_base, + return etnaviv_iommu_get_suballoc_va(context, mapping, memory_base, suballoc->paddr, SUBALLOC_SIZE); } -void etnaviv_cmdbuf_suballoc_unmap(struct etnaviv_iommu *mmu, +void etnaviv_cmdbuf_suballoc_unmap(struct etnaviv_iommu_context *context, struct etnaviv_vram_mapping *mapping) { - etnaviv_iommu_put_suballoc_va(mmu, mapping); + etnaviv_iommu_put_suballoc_va(context, mapping); } void etnaviv_cmdbuf_suballoc_destroy(struct etnaviv_cmdbuf_suballoc *suballoc) diff --git a/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.h b/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.h index a28668e46e26..ad6fd8eb0378 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_cmdbuf.h @@ -9,7 +9,7 @@ #include struct device; -struct etnaviv_iommu; +struct etnaviv_iommu_context; struct etnaviv_vram_mapping; struct etnaviv_cmdbuf_suballoc; struct etnaviv_perfmon_request; @@ -28,10 +28,10 @@ struct etnaviv_cmdbuf_suballoc * etnaviv_cmdbuf_suballoc_new(struct device *dev); void etnaviv_cmdbuf_suballoc_destroy(struct etnaviv_cmdbuf_suballoc *suballoc); int etnaviv_cmdbuf_suballoc_map(struct etnaviv_cmdbuf_suballoc *suballoc, - struct etnaviv_iommu *mmu, + struct etnaviv_iommu_context *context, struct etnaviv_vram_mapping *mapping, u32 memory_base); -void etnaviv_cmdbuf_suballoc_unmap(struct etnaviv_iommu *mmu, +void etnaviv_cmdbuf_suballoc_unmap(struct etnaviv_iommu_context *context, struct etnaviv_vram_mapping *mapping); diff --git a/drivers/gpu/drm/etnaviv/etnaviv_drv.c b/drivers/gpu/drm/etnaviv/etnaviv_drv.c index 5fa3aa7bdbc5..eb0c23fe979a 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_drv.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_drv.c @@ -119,9 +119,9 @@ static int etnaviv_mmu_show(struct etnaviv_gpu *gpu, struct seq_file *m) seq_printf(m, "Active Objects (%s):\n", dev_name(gpu->dev)); - mutex_lock(&gpu->mmu->lock); - drm_mm_print(&gpu->mmu->mm, &p); - mutex_unlock(&gpu->mmu->lock); + mutex_lock(&gpu->mmu_context->lock); + drm_mm_print(&gpu->mmu_context->mm, &p); + mutex_unlock(&gpu->mmu_context->lock); return 0; } diff --git a/drivers/gpu/drm/etnaviv/etnaviv_drv.h b/drivers/gpu/drm/etnaviv/etnaviv_drv.h index e052d7db66ae..5f8db08f1c17 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_drv.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_drv.h @@ -22,6 +22,7 @@ struct etnaviv_gpu; struct etnaviv_mmu; struct etnaviv_gem_object; struct etnaviv_gem_submit; +struct etnaviv_iommu_global; struct etnaviv_file_private { /* @@ -37,6 +38,7 @@ struct etnaviv_drm_private { struct etnaviv_gpu *gpu[ETNA_MAX_PIPES]; struct etnaviv_cmdbuf_suballoc *cmdbuf_suballoc; + struct etnaviv_iommu_global *mmu_global; /* list of GEM objects: */ struct mutex gem_lock; @@ -69,7 +71,7 @@ int etnaviv_gem_new_userptr(struct drm_device *dev, struct drm_file *file, uintptr_t ptr, u32 size, u32 flags, u32 *handle); u16 etnaviv_buffer_init(struct etnaviv_gpu *gpu); u16 etnaviv_buffer_config_mmuv2(struct etnaviv_gpu *gpu, u32 mtlb_addr, u32 safe_addr); -u16 etnaviv_buffer_config_pta(struct etnaviv_gpu *gpu); +u16 etnaviv_buffer_config_pta(struct etnaviv_gpu *gpu, unsigned short id); void etnaviv_buffer_end(struct etnaviv_gpu *gpu); void etnaviv_sync_point_queue(struct etnaviv_gpu *gpu, unsigned int event); void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, diff --git a/drivers/gpu/drm/etnaviv/etnaviv_dump.c b/drivers/gpu/drm/etnaviv/etnaviv_dump.c index 2ce60baa4ad9..7e6791517693 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_dump.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_dump.c @@ -93,7 +93,7 @@ static void etnaviv_core_dump_registers(struct core_dump_iterator *iter, } static void etnaviv_core_dump_mmu(struct core_dump_iterator *iter, - struct etnaviv_iommu *mmu, size_t mmu_size) + struct etnaviv_iommu_context *mmu, size_t mmu_size) { etnaviv_iommu_dump(mmu, iter->data); @@ -125,9 +125,9 @@ void etnaviv_core_dump(struct etnaviv_gem_submit *submit) return; etnaviv_dump_core = false; - mutex_lock(&gpu->mmu->lock); + mutex_lock(&gpu->mmu_context->lock); - mmu_size = etnaviv_iommu_dump_size(gpu->mmu); + mmu_size = etnaviv_iommu_dump_size(gpu->mmu_context); /* We always dump registers, mmu, ring, hanging cmdbuf and end marker */ n_obj = 5; @@ -157,7 +157,7 @@ void etnaviv_core_dump(struct etnaviv_gem_submit *submit) iter.start = __vmalloc(file_size, GFP_KERNEL | __GFP_NOWARN | __GFP_NORETRY, PAGE_KERNEL); if (!iter.start) { - mutex_unlock(&gpu->mmu->lock); + mutex_unlock(&gpu->mmu_context->lock); dev_warn(gpu->dev, "failed to allocate devcoredump file\n"); return; } @@ -169,7 +169,7 @@ void etnaviv_core_dump(struct etnaviv_gem_submit *submit) memset(iter.hdr, 0, iter.data - iter.start); etnaviv_core_dump_registers(&iter, gpu); - etnaviv_core_dump_mmu(&iter, gpu->mmu, mmu_size); + etnaviv_core_dump_mmu(&iter, gpu->mmu_context, mmu_size); etnaviv_core_dump_mem(&iter, ETDUMP_BUF_RING, gpu->buffer.vaddr, gpu->buffer.size, etnaviv_cmdbuf_get_va(&gpu->buffer, @@ -221,7 +221,7 @@ void etnaviv_core_dump(struct etnaviv_gem_submit *submit) obj->base.size); } - mutex_unlock(&gpu->mmu->lock); + mutex_unlock(&gpu->mmu_context->lock); etnaviv_core_dump_header(&iter, ETDUMP_BUF_END, iter.data); diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem.c b/drivers/gpu/drm/etnaviv/etnaviv_gem.c index e199a6833ff0..0ccc3c4dffc4 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gem.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem.c @@ -223,12 +223,12 @@ int etnaviv_gem_mmap_offset(struct drm_gem_object *obj, u64 *offset) static struct etnaviv_vram_mapping * etnaviv_gem_get_vram_mapping(struct etnaviv_gem_object *obj, - struct etnaviv_iommu *mmu) + struct etnaviv_iommu_context *context) { struct etnaviv_vram_mapping *mapping; list_for_each_entry(mapping, &obj->vram_list, obj_node) { - if (mapping->mmu == mmu) + if (mapping->context == context) return mapping; } @@ -256,7 +256,7 @@ struct etnaviv_vram_mapping *etnaviv_gem_mapping_get( int ret = 0; mutex_lock(&etnaviv_obj->lock); - mapping = etnaviv_gem_get_vram_mapping(etnaviv_obj, gpu->mmu); + mapping = etnaviv_gem_get_vram_mapping(etnaviv_obj, gpu->mmu_context); if (mapping) { /* * Holding the object lock prevents the use count changing @@ -265,12 +265,12 @@ struct etnaviv_vram_mapping *etnaviv_gem_mapping_get( * the MMU owns this mapping to close this race. */ if (mapping->use == 0) { - mutex_lock(&gpu->mmu->lock); - if (mapping->mmu == gpu->mmu) + mutex_lock(&gpu->mmu_context->lock); + if (mapping->context == gpu->mmu_context) mapping->use += 1; else mapping = NULL; - mutex_unlock(&gpu->mmu->lock); + mutex_unlock(&gpu->mmu_context->lock); if (mapping) goto out; } else { @@ -303,11 +303,11 @@ struct etnaviv_vram_mapping *etnaviv_gem_mapping_get( list_del(&mapping->obj_node); } - mapping->mmu = gpu->mmu; + mapping->context = gpu->mmu_context; mapping->use = 1; - ret = etnaviv_iommu_map_gem(gpu->mmu, etnaviv_obj, gpu->memory_base, - mapping); + ret = etnaviv_iommu_map_gem(gpu->mmu_context, etnaviv_obj, + gpu->memory_base, mapping); if (ret < 0) kfree(mapping); else @@ -525,12 +525,12 @@ void etnaviv_gem_free_object(struct drm_gem_object *obj) list_for_each_entry_safe(mapping, tmp, &etnaviv_obj->vram_list, obj_node) { - struct etnaviv_iommu *mmu = mapping->mmu; + struct etnaviv_iommu_context *context = mapping->context; WARN_ON(mapping->use); - if (mmu) - etnaviv_iommu_unmap_gem(mmu, mapping); + if (context) + etnaviv_iommu_unmap_gem(context, mapping); list_del(&mapping->obj_node); kfree(mapping); diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem.h b/drivers/gpu/drm/etnaviv/etnaviv_gem.h index d7d8a835f379..5a004d5e4eaa 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gem.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem.h @@ -25,7 +25,7 @@ struct etnaviv_vram_mapping { struct list_head scan_node; struct list_head mmu_node; struct etnaviv_gem_object *object; - struct etnaviv_iommu *mmu; + struct etnaviv_iommu_context *context; struct drm_mm_node vram_node; unsigned int use; u32 iova; diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gpu.c b/drivers/gpu/drm/etnaviv/etnaviv_gpu.c index 179bc6c544ca..885ca8f92338 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gpu.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gpu.c @@ -681,7 +681,7 @@ static void etnaviv_gpu_hw_init(struct etnaviv_gpu *gpu) etnaviv_gpu_setup_pulse_eater(gpu); /* setup the MMU */ - etnaviv_iommu_restore(gpu); + etnaviv_iommu_restore(gpu, gpu->mmu_context); /* Start command processor */ prefetch = etnaviv_buffer_init(gpu); @@ -754,14 +754,19 @@ int etnaviv_gpu_init(struct etnaviv_gpu *gpu) goto fail; } - gpu->mmu = etnaviv_iommu_new(gpu); - if (IS_ERR(gpu->mmu)) { - dev_err(gpu->dev, "Failed to instantiate GPU IOMMU\n"); - ret = PTR_ERR(gpu->mmu); + ret = etnaviv_iommu_global_init(gpu); + if (ret) goto fail; + + gpu->mmu_context = etnaviv_iommu_context_init(priv->mmu_global); + if (IS_ERR(gpu->mmu_context)) { + dev_err(gpu->dev, "Failed to instantiate GPU IOMMU\n"); + ret = PTR_ERR(gpu->mmu_context); + goto iommu_global_fini; } - ret = etnaviv_cmdbuf_suballoc_map(priv->cmdbuf_suballoc, gpu->mmu, + ret = etnaviv_cmdbuf_suballoc_map(priv->cmdbuf_suballoc, + gpu->mmu_context, &gpu->cmdbuf_mapping, gpu->memory_base); if (ret) { @@ -777,7 +782,7 @@ int etnaviv_gpu_init(struct etnaviv_gpu *gpu) goto unmap_suballoc; } - if (gpu->mmu->version == ETNAVIV_IOMMU_V1 && + if (!(gpu->identity.minor_features1 & chipMinorFeatures1_MMU_VERSION) && etnaviv_cmdbuf_get_va(&gpu->buffer, &gpu->cmdbuf_mapping) > 0x80000000) { ret = -EINVAL; dev_err(gpu->dev, @@ -808,9 +813,11 @@ int etnaviv_gpu_init(struct etnaviv_gpu *gpu) free_buffer: etnaviv_cmdbuf_free(&gpu->buffer); unmap_suballoc: - etnaviv_cmdbuf_suballoc_unmap(gpu->mmu, &gpu->cmdbuf_mapping); + etnaviv_cmdbuf_suballoc_unmap(gpu->mmu_context, &gpu->cmdbuf_mapping); destroy_iommu: - etnaviv_iommu_destroy(gpu->mmu); + etnaviv_iommu_context_put(gpu->mmu_context); +iommu_global_fini: + etnaviv_iommu_global_fini(gpu); fail: pm_runtime_mark_last_busy(gpu->dev); pm_runtime_put_autosuspend(gpu->dev); @@ -1683,8 +1690,10 @@ static void etnaviv_gpu_unbind(struct device *dev, struct device *master, if (gpu->initialized) { etnaviv_cmdbuf_free(&gpu->buffer); - etnaviv_cmdbuf_suballoc_unmap(gpu->mmu, &gpu->cmdbuf_mapping); - etnaviv_iommu_destroy(gpu->mmu); + etnaviv_cmdbuf_suballoc_unmap(gpu->mmu_context, + &gpu->cmdbuf_mapping); + etnaviv_iommu_context_put(gpu->mmu_context); + etnaviv_iommu_global_fini(gpu); gpu->initialized = false; } diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gpu.h b/drivers/gpu/drm/etnaviv/etnaviv_gpu.h index 96380942cd8c..c0bd6018d53b 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gpu.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_gpu.h @@ -8,6 +8,7 @@ #include "etnaviv_cmdbuf.h" #include "etnaviv_gem.h" +#include "etnaviv_mmu.h" #include "etnaviv_drv.h" struct etnaviv_gem_submit; @@ -136,7 +137,7 @@ struct etnaviv_gpu { void __iomem *mmio; int irq; - struct etnaviv_iommu *mmu; + struct etnaviv_iommu_context *mmu_context; unsigned int flush_seq; /* Power Control: */ diff --git a/drivers/gpu/drm/etnaviv/etnaviv_iommu.c b/drivers/gpu/drm/etnaviv/etnaviv_iommu.c index 18c627c5cae1..a2f1ff151822 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_iommu.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_iommu.c @@ -11,7 +11,6 @@ #include "etnaviv_gpu.h" #include "etnaviv_mmu.h" -#include "etnaviv_iommu.h" #include "state_hi.xml.h" #define PT_SIZE SZ_2M @@ -19,113 +18,78 @@ #define GPU_MEM_START 0x80000000 -struct etnaviv_iommuv1_domain { - struct etnaviv_iommu_domain base; +struct etnaviv_iommuv1_context { + struct etnaviv_iommu_context base; u32 *pgtable_cpu; dma_addr_t pgtable_dma; }; -static struct etnaviv_iommuv1_domain * -to_etnaviv_domain(struct etnaviv_iommu_domain *domain) +static struct etnaviv_iommuv1_context * +to_v1_context(struct etnaviv_iommu_context *context) { - return container_of(domain, struct etnaviv_iommuv1_domain, base); + return container_of(context, struct etnaviv_iommuv1_context, base); } -static int __etnaviv_iommu_init(struct etnaviv_iommuv1_domain *etnaviv_domain) +static void etnaviv_iommuv1_free(struct etnaviv_iommu_context *context) { - u32 *p; - int i; - - etnaviv_domain->base.bad_page_cpu = - dma_alloc_wc(etnaviv_domain->base.dev, SZ_4K, - &etnaviv_domain->base.bad_page_dma, - GFP_KERNEL); - if (!etnaviv_domain->base.bad_page_cpu) - return -ENOMEM; - - p = etnaviv_domain->base.bad_page_cpu; - for (i = 0; i < SZ_4K / 4; i++) - *p++ = 0xdead55aa; - - etnaviv_domain->pgtable_cpu = dma_alloc_wc(etnaviv_domain->base.dev, - PT_SIZE, - &etnaviv_domain->pgtable_dma, - GFP_KERNEL); - if (!etnaviv_domain->pgtable_cpu) { - dma_free_wc(etnaviv_domain->base.dev, SZ_4K, - etnaviv_domain->base.bad_page_cpu, - etnaviv_domain->base.bad_page_dma); - return -ENOMEM; - } - - memset32(etnaviv_domain->pgtable_cpu, etnaviv_domain->base.bad_page_dma, - PT_ENTRIES); - - return 0; -} + struct etnaviv_iommuv1_context *v1_context = to_v1_context(context); -static void etnaviv_iommuv1_domain_free(struct etnaviv_iommu_domain *domain) -{ - struct etnaviv_iommuv1_domain *etnaviv_domain = - to_etnaviv_domain(domain); + drm_mm_takedown(&context->mm); - dma_free_wc(etnaviv_domain->base.dev, PT_SIZE, - etnaviv_domain->pgtable_cpu, etnaviv_domain->pgtable_dma); + dma_free_wc(context->global->dev, PT_SIZE, v1_context->pgtable_cpu, + v1_context->pgtable_dma); - dma_free_wc(etnaviv_domain->base.dev, SZ_4K, - etnaviv_domain->base.bad_page_cpu, - etnaviv_domain->base.bad_page_dma); + context->global->v1.shared_context = NULL; - kfree(etnaviv_domain); + kfree(v1_context); } -static int etnaviv_iommuv1_map(struct etnaviv_iommu_domain *domain, +static int etnaviv_iommuv1_map(struct etnaviv_iommu_context *context, unsigned long iova, phys_addr_t paddr, size_t size, int prot) { - struct etnaviv_iommuv1_domain *etnaviv_domain = to_etnaviv_domain(domain); + struct etnaviv_iommuv1_context *v1_context = to_v1_context(context); unsigned int index = (iova - GPU_MEM_START) / SZ_4K; if (size != SZ_4K) return -EINVAL; - etnaviv_domain->pgtable_cpu[index] = paddr; + v1_context->pgtable_cpu[index] = paddr; return 0; } -static size_t etnaviv_iommuv1_unmap(struct etnaviv_iommu_domain *domain, +static size_t etnaviv_iommuv1_unmap(struct etnaviv_iommu_context *context, unsigned long iova, size_t size) { - struct etnaviv_iommuv1_domain *etnaviv_domain = - to_etnaviv_domain(domain); + struct etnaviv_iommuv1_context *v1_context = to_v1_context(context); unsigned int index = (iova - GPU_MEM_START) / SZ_4K; if (size != SZ_4K) return -EINVAL; - etnaviv_domain->pgtable_cpu[index] = etnaviv_domain->base.bad_page_dma; + v1_context->pgtable_cpu[index] = context->global->bad_page_dma; return SZ_4K; } -static size_t etnaviv_iommuv1_dump_size(struct etnaviv_iommu_domain *domain) +static size_t etnaviv_iommuv1_dump_size(struct etnaviv_iommu_context *context) { return PT_SIZE; } -static void etnaviv_iommuv1_dump(struct etnaviv_iommu_domain *domain, void *buf) +static void etnaviv_iommuv1_dump(struct etnaviv_iommu_context *context, + void *buf) { - struct etnaviv_iommuv1_domain *etnaviv_domain = - to_etnaviv_domain(domain); + struct etnaviv_iommuv1_context *v1_context = to_v1_context(context); - memcpy(buf, etnaviv_domain->pgtable_cpu, PT_SIZE); + memcpy(buf, v1_context->pgtable_cpu, PT_SIZE); } -void etnaviv_iommuv1_restore(struct etnaviv_gpu *gpu) +static void etnaviv_iommuv1_restore(struct etnaviv_gpu *gpu, + struct etnaviv_iommu_context *context) { - struct etnaviv_iommuv1_domain *etnaviv_domain = - to_etnaviv_domain(gpu->mmu->domain); + struct etnaviv_iommuv1_context *v1_context = to_v1_context(context); u32 pgtable; /* set base addresses */ @@ -136,7 +100,7 @@ void etnaviv_iommuv1_restore(struct etnaviv_gpu *gpu) gpu_write(gpu, VIVS_MC_MEMORY_BASE_ADDR_PE, gpu->memory_base); /* set page table address in MC */ - pgtable = (u32)etnaviv_domain->pgtable_dma; + pgtable = (u32)v1_context->pgtable_dma; gpu_write(gpu, VIVS_MC_MMU_FE_PAGE_TABLE, pgtable); gpu_write(gpu, VIVS_MC_MMU_TX_PAGE_TABLE, pgtable); @@ -145,39 +109,62 @@ void etnaviv_iommuv1_restore(struct etnaviv_gpu *gpu) gpu_write(gpu, VIVS_MC_MMU_RA_PAGE_TABLE, pgtable); } -static const struct etnaviv_iommu_domain_ops etnaviv_iommuv1_ops = { - .free = etnaviv_iommuv1_domain_free, + +const struct etnaviv_iommu_ops etnaviv_iommuv1_ops = { + .free = etnaviv_iommuv1_free, .map = etnaviv_iommuv1_map, .unmap = etnaviv_iommuv1_unmap, .dump_size = etnaviv_iommuv1_dump_size, .dump = etnaviv_iommuv1_dump, + .restore = etnaviv_iommuv1_restore, }; -struct etnaviv_iommu_domain * -etnaviv_iommuv1_domain_alloc(struct etnaviv_gpu *gpu) +struct etnaviv_iommu_context * +etnaviv_iommuv1_context_alloc(struct etnaviv_iommu_global *global) { - struct etnaviv_iommuv1_domain *etnaviv_domain; - struct etnaviv_iommu_domain *domain; - int ret; + struct etnaviv_iommuv1_context *v1_context; + struct etnaviv_iommu_context *context; + + mutex_lock(&global->lock); + + /* + * MMUv1 does not support switching between different contexts without + * a stop the world operation, so we only support a single shared + * context with this version. + */ + if (global->v1.shared_context) { + context = global->v1.shared_context; + etnaviv_iommu_context_get(context); + mutex_unlock(&global->lock); + return context; + } - etnaviv_domain = kzalloc(sizeof(*etnaviv_domain), GFP_KERNEL); - if (!etnaviv_domain) + v1_context = kzalloc(sizeof(*v1_context), GFP_KERNEL); + if (!v1_context) return NULL; - domain = &etnaviv_domain->base; + v1_context->pgtable_cpu = dma_alloc_wc(global->dev, PT_SIZE, + &v1_context->pgtable_dma, + GFP_KERNEL); + if (!v1_context->pgtable_cpu) + goto out_free; - domain->dev = gpu->dev; - domain->base = GPU_MEM_START; - domain->size = PT_ENTRIES * SZ_4K; - domain->ops = &etnaviv_iommuv1_ops; + memset32(v1_context->pgtable_cpu, global->bad_page_dma, PT_ENTRIES); - ret = __etnaviv_iommu_init(etnaviv_domain); - if (ret) - goto out_free; + context = &v1_context->base; + context->global = global; + kref_init(&context->refcount); + mutex_init(&context->lock); + INIT_LIST_HEAD(&context->mappings); + drm_mm_init(&context->mm, GPU_MEM_START, PT_ENTRIES * SZ_4K); + context->global->v1.shared_context = context; + + mutex_unlock(&global->lock); - return &etnaviv_domain->base; + return context; out_free: - kfree(etnaviv_domain); + mutex_unlock(&global->lock); + kfree(v1_context); return NULL; } diff --git a/drivers/gpu/drm/etnaviv/etnaviv_iommu.h b/drivers/gpu/drm/etnaviv/etnaviv_iommu.h deleted file mode 100644 index b279404ce91a..000000000000 --- a/drivers/gpu/drm/etnaviv/etnaviv_iommu.h +++ /dev/null @@ -1,20 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* - * Copyright (C) 2014-2018 Etnaviv Project - */ - -#ifndef __ETNAVIV_IOMMU_H__ -#define __ETNAVIV_IOMMU_H__ - -struct etnaviv_gpu; -struct etnaviv_iommu_domain; - -struct etnaviv_iommu_domain * -etnaviv_iommuv1_domain_alloc(struct etnaviv_gpu *gpu); -void etnaviv_iommuv1_restore(struct etnaviv_gpu *gpu); - -struct etnaviv_iommu_domain * -etnaviv_iommuv2_domain_alloc(struct etnaviv_gpu *gpu); -void etnaviv_iommuv2_restore(struct etnaviv_gpu *gpu); - -#endif /* __ETNAVIV_IOMMU_H__ */ diff --git a/drivers/gpu/drm/etnaviv/etnaviv_iommu_v2.c b/drivers/gpu/drm/etnaviv/etnaviv_iommu_v2.c index d7cc184da571..5ca2077c148d 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_iommu_v2.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_iommu_v2.c @@ -13,7 +13,6 @@ #include "etnaviv_cmdbuf.h" #include "etnaviv_gpu.h" #include "etnaviv_mmu.h" -#include "etnaviv_iommu.h" #include "state.xml.h" #include "state_hi.xml.h" @@ -28,11 +27,9 @@ #define MMUv2_MAX_STLB_ENTRIES 1024 -struct etnaviv_iommuv2_domain { - struct etnaviv_iommu_domain base; - /* P(age) T(able) A(rray) */ - u64 *pta_cpu; - dma_addr_t pta_dma; +struct etnaviv_iommuv2_context { + struct etnaviv_iommu_context base; + unsigned short id; /* M(aster) TLB aka first level pagetable */ u32 *mtlb_cpu; dma_addr_t mtlb_dma; @@ -41,41 +38,62 @@ struct etnaviv_iommuv2_domain { dma_addr_t stlb_dma[MMUv2_MAX_STLB_ENTRIES]; }; -static struct etnaviv_iommuv2_domain * -to_etnaviv_domain(struct etnaviv_iommu_domain *domain) +static struct etnaviv_iommuv2_context * +to_v2_context(struct etnaviv_iommu_context *context) { - return container_of(domain, struct etnaviv_iommuv2_domain, base); + return container_of(context, struct etnaviv_iommuv2_context, base); } +static void etnaviv_iommuv2_free(struct etnaviv_iommu_context *context) +{ + struct etnaviv_iommuv2_context *v2_context = to_v2_context(context); + int i; + + drm_mm_takedown(&context->mm); + + for (i = 0; i < MMUv2_MAX_STLB_ENTRIES; i++) { + if (v2_context->stlb_cpu[i]) + dma_free_wc(context->global->dev, SZ_4K, + v2_context->stlb_cpu[i], + v2_context->stlb_dma[i]); + } + + dma_free_wc(context->global->dev, SZ_4K, v2_context->mtlb_cpu, + v2_context->mtlb_dma); + + clear_bit(v2_context->id, context->global->v2.pta_alloc); + + vfree(v2_context); +} static int -etnaviv_iommuv2_ensure_stlb(struct etnaviv_iommuv2_domain *etnaviv_domain, +etnaviv_iommuv2_ensure_stlb(struct etnaviv_iommuv2_context *v2_context, int stlb) { - if (etnaviv_domain->stlb_cpu[stlb]) + if (v2_context->stlb_cpu[stlb]) return 0; - etnaviv_domain->stlb_cpu[stlb] = - dma_alloc_wc(etnaviv_domain->base.dev, SZ_4K, - &etnaviv_domain->stlb_dma[stlb], + v2_context->stlb_cpu[stlb] = + dma_alloc_wc(v2_context->base.global->dev, SZ_4K, + &v2_context->stlb_dma[stlb], GFP_KERNEL); - if (!etnaviv_domain->stlb_cpu[stlb]) + if (!v2_context->stlb_cpu[stlb]) return -ENOMEM; - memset32(etnaviv_domain->stlb_cpu[stlb], MMUv2_PTE_EXCEPTION, + memset32(v2_context->stlb_cpu[stlb], MMUv2_PTE_EXCEPTION, SZ_4K / sizeof(u32)); - etnaviv_domain->mtlb_cpu[stlb] = etnaviv_domain->stlb_dma[stlb] | - MMUv2_PTE_PRESENT; + v2_context->mtlb_cpu[stlb] = + v2_context->stlb_dma[stlb] | MMUv2_PTE_PRESENT; + return 0; } -static int etnaviv_iommuv2_map(struct etnaviv_iommu_domain *domain, +static int etnaviv_iommuv2_map(struct etnaviv_iommu_context *context, unsigned long iova, phys_addr_t paddr, size_t size, int prot) { - struct etnaviv_iommuv2_domain *etnaviv_domain = - to_etnaviv_domain(domain); + struct etnaviv_iommuv2_context *v2_context = to_v2_context(context); int mtlb_entry, stlb_entry, ret; u32 entry = lower_32_bits(paddr) | MMUv2_PTE_PRESENT; @@ -91,20 +109,19 @@ static int etnaviv_iommuv2_map(struct etnaviv_iommu_domain *domain, mtlb_entry = (iova & MMUv2_MTLB_MASK) >> MMUv2_MTLB_SHIFT; stlb_entry = (iova & MMUv2_STLB_MASK) >> MMUv2_STLB_SHIFT; - ret = etnaviv_iommuv2_ensure_stlb(etnaviv_domain, mtlb_entry); + ret = etnaviv_iommuv2_ensure_stlb(v2_context, mtlb_entry); if (ret) return ret; - etnaviv_domain->stlb_cpu[mtlb_entry][stlb_entry] = entry; + v2_context->stlb_cpu[mtlb_entry][stlb_entry] = entry; return 0; } -static size_t etnaviv_iommuv2_unmap(struct etnaviv_iommu_domain *domain, +static size_t etnaviv_iommuv2_unmap(struct etnaviv_iommu_context *context, unsigned long iova, size_t size) { - struct etnaviv_iommuv2_domain *etnaviv_domain = - to_etnaviv_domain(domain); + struct etnaviv_iommuv2_context *etnaviv_domain = to_v2_context(context); int mtlb_entry, stlb_entry; if (size != SZ_4K) @@ -118,118 +135,35 @@ static size_t etnaviv_iommuv2_unmap(struct etnaviv_iommu_domain *domain, return SZ_4K; } -static int etnaviv_iommuv2_init(struct etnaviv_iommuv2_domain *etnaviv_domain) -{ - int ret; - - /* allocate scratch page */ - etnaviv_domain->base.bad_page_cpu = - dma_alloc_wc(etnaviv_domain->base.dev, SZ_4K, - &etnaviv_domain->base.bad_page_dma, - GFP_KERNEL); - if (!etnaviv_domain->base.bad_page_cpu) { - ret = -ENOMEM; - goto fail_mem; - } - - memset32(etnaviv_domain->base.bad_page_cpu, 0xdead55aa, - SZ_4K / sizeof(u32)); - - etnaviv_domain->pta_cpu = dma_alloc_wc(etnaviv_domain->base.dev, - SZ_4K, &etnaviv_domain->pta_dma, - GFP_KERNEL); - if (!etnaviv_domain->pta_cpu) { - ret = -ENOMEM; - goto fail_mem; - } - - etnaviv_domain->mtlb_cpu = dma_alloc_wc(etnaviv_domain->base.dev, - SZ_4K, &etnaviv_domain->mtlb_dma, - GFP_KERNEL); - if (!etnaviv_domain->mtlb_cpu) { - ret = -ENOMEM; - goto fail_mem; - } - - memset32(etnaviv_domain->mtlb_cpu, MMUv2_PTE_EXCEPTION, - MMUv2_MAX_STLB_ENTRIES); - - return 0; - -fail_mem: - if (etnaviv_domain->base.bad_page_cpu) - dma_free_wc(etnaviv_domain->base.dev, SZ_4K, - etnaviv_domain->base.bad_page_cpu, - etnaviv_domain->base.bad_page_dma); - - if (etnaviv_domain->pta_cpu) - dma_free_wc(etnaviv_domain->base.dev, SZ_4K, - etnaviv_domain->pta_cpu, etnaviv_domain->pta_dma); - - if (etnaviv_domain->mtlb_cpu) - dma_free_wc(etnaviv_domain->base.dev, SZ_4K, - etnaviv_domain->mtlb_cpu, etnaviv_domain->mtlb_dma); - - return ret; -} - -static void etnaviv_iommuv2_domain_free(struct etnaviv_iommu_domain *domain) -{ - struct etnaviv_iommuv2_domain *etnaviv_domain = - to_etnaviv_domain(domain); - int i; - - dma_free_wc(etnaviv_domain->base.dev, SZ_4K, - etnaviv_domain->base.bad_page_cpu, - etnaviv_domain->base.bad_page_dma); - - dma_free_wc(etnaviv_domain->base.dev, SZ_4K, - etnaviv_domain->pta_cpu, etnaviv_domain->pta_dma); - - dma_free_wc(etnaviv_domain->base.dev, SZ_4K, - etnaviv_domain->mtlb_cpu, etnaviv_domain->mtlb_dma); - - for (i = 0; i < MMUv2_MAX_STLB_ENTRIES; i++) { - if (etnaviv_domain->stlb_cpu[i]) - dma_free_wc(etnaviv_domain->base.dev, SZ_4K, - etnaviv_domain->stlb_cpu[i], - etnaviv_domain->stlb_dma[i]); - } - - vfree(etnaviv_domain); -} - -static size_t etnaviv_iommuv2_dump_size(struct etnaviv_iommu_domain *domain) +static size_t etnaviv_iommuv2_dump_size(struct etnaviv_iommu_context *context) { - struct etnaviv_iommuv2_domain *etnaviv_domain = - to_etnaviv_domain(domain); + struct etnaviv_iommuv2_context *v2_context = to_v2_context(context); size_t dump_size = SZ_4K; int i; for (i = 0; i < MMUv2_MAX_STLB_ENTRIES; i++) - if (etnaviv_domain->mtlb_cpu[i] & MMUv2_PTE_PRESENT) + if (v2_context->mtlb_cpu[i] & MMUv2_PTE_PRESENT) dump_size += SZ_4K; return dump_size; } -static void etnaviv_iommuv2_dump(struct etnaviv_iommu_domain *domain, void *buf) +static void etnaviv_iommuv2_dump(struct etnaviv_iommu_context *context, void *buf) { - struct etnaviv_iommuv2_domain *etnaviv_domain = - to_etnaviv_domain(domain); + struct etnaviv_iommuv2_context *v2_context = to_v2_context(context); int i; - memcpy(buf, etnaviv_domain->mtlb_cpu, SZ_4K); + memcpy(buf, v2_context->mtlb_cpu, SZ_4K); buf += SZ_4K; for (i = 0; i < MMUv2_MAX_STLB_ENTRIES; i++, buf += SZ_4K) - if (etnaviv_domain->mtlb_cpu[i] & MMUv2_PTE_PRESENT) - memcpy(buf, etnaviv_domain->stlb_cpu[i], SZ_4K); + if (v2_context->mtlb_cpu[i] & MMUv2_PTE_PRESENT) + memcpy(buf, v2_context->stlb_cpu[i], SZ_4K); } -static void etnaviv_iommuv2_restore_nonsec(struct etnaviv_gpu *gpu) +static void etnaviv_iommuv2_restore_nonsec(struct etnaviv_gpu *gpu, + struct etnaviv_iommu_context *context) { - struct etnaviv_iommuv2_domain *etnaviv_domain = - to_etnaviv_domain(gpu->mmu->domain); + struct etnaviv_iommuv2_context *v2_context = to_v2_context(context); u16 prefetch; /* If the MMU is already enabled the state is still there. */ @@ -237,8 +171,8 @@ static void etnaviv_iommuv2_restore_nonsec(struct etnaviv_gpu *gpu) return; prefetch = etnaviv_buffer_config_mmuv2(gpu, - (u32)etnaviv_domain->mtlb_dma, - (u32)etnaviv_domain->base.bad_page_dma); + (u32)v2_context->mtlb_dma, + (u32)context->global->bad_page_dma); etnaviv_gpu_start_fe(gpu, (u32)etnaviv_cmdbuf_get_pa(&gpu->buffer), prefetch); etnaviv_gpu_wait_idle(gpu, 100); @@ -246,10 +180,10 @@ static void etnaviv_iommuv2_restore_nonsec(struct etnaviv_gpu *gpu) gpu_write(gpu, VIVS_MMUv2_CONTROL, VIVS_MMUv2_CONTROL_ENABLE); } -static void etnaviv_iommuv2_restore_sec(struct etnaviv_gpu *gpu) +static void etnaviv_iommuv2_restore_sec(struct etnaviv_gpu *gpu, + struct etnaviv_iommu_context *context) { - struct etnaviv_iommuv2_domain *etnaviv_domain = - to_etnaviv_domain(gpu->mmu->domain); + struct etnaviv_iommuv2_context *v2_context = to_v2_context(context); u16 prefetch; /* If the MMU is already enabled the state is still there. */ @@ -257,26 +191,26 @@ static void etnaviv_iommuv2_restore_sec(struct etnaviv_gpu *gpu) return; gpu_write(gpu, VIVS_MMUv2_PTA_ADDRESS_LOW, - lower_32_bits(etnaviv_domain->pta_dma)); + lower_32_bits(context->global->v2.pta_dma)); gpu_write(gpu, VIVS_MMUv2_PTA_ADDRESS_HIGH, - upper_32_bits(etnaviv_domain->pta_dma)); + upper_32_bits(context->global->v2.pta_dma)); gpu_write(gpu, VIVS_MMUv2_PTA_CONTROL, VIVS_MMUv2_PTA_CONTROL_ENABLE); gpu_write(gpu, VIVS_MMUv2_NONSEC_SAFE_ADDR_LOW, - lower_32_bits(etnaviv_domain->base.bad_page_dma)); + lower_32_bits(context->global->bad_page_dma)); gpu_write(gpu, VIVS_MMUv2_SEC_SAFE_ADDR_LOW, - lower_32_bits(etnaviv_domain->base.bad_page_dma)); + lower_32_bits(context->global->bad_page_dma)); gpu_write(gpu, VIVS_MMUv2_SAFE_ADDRESS_CONFIG, VIVS_MMUv2_SAFE_ADDRESS_CONFIG_NON_SEC_SAFE_ADDR_HIGH( - upper_32_bits(etnaviv_domain->base.bad_page_dma)) | + upper_32_bits(context->global->bad_page_dma)) | VIVS_MMUv2_SAFE_ADDRESS_CONFIG_SEC_SAFE_ADDR_HIGH( - upper_32_bits(etnaviv_domain->base.bad_page_dma))); + upper_32_bits(context->global->bad_page_dma))); - etnaviv_domain->pta_cpu[0] = etnaviv_domain->mtlb_dma | - VIVS_MMUv2_CONFIGURATION_MODE_MODE4_K; + context->global->v2.pta_cpu[0] = v2_context->mtlb_dma | + VIVS_MMUv2_CONFIGURATION_MODE_MODE4_K; /* trigger a PTA load through the FE */ - prefetch = etnaviv_buffer_config_pta(gpu); + prefetch = etnaviv_buffer_config_pta(gpu, v2_context->id); etnaviv_gpu_start_fe(gpu, (u32)etnaviv_cmdbuf_get_pa(&gpu->buffer), prefetch); etnaviv_gpu_wait_idle(gpu, 100); @@ -284,14 +218,15 @@ static void etnaviv_iommuv2_restore_sec(struct etnaviv_gpu *gpu) gpu_write(gpu, VIVS_MMUv2_SEC_CONTROL, VIVS_MMUv2_SEC_CONTROL_ENABLE); } -void etnaviv_iommuv2_restore(struct etnaviv_gpu *gpu) +static void etnaviv_iommuv2_restore(struct etnaviv_gpu *gpu, + struct etnaviv_iommu_context *context) { switch (gpu->sec_mode) { case ETNA_SEC_NONE: - etnaviv_iommuv2_restore_nonsec(gpu); + etnaviv_iommuv2_restore_nonsec(gpu, context); break; case ETNA_SEC_KERNEL: - etnaviv_iommuv2_restore_sec(gpu); + etnaviv_iommuv2_restore_sec(gpu, context); break; default: WARN(1, "unhandled GPU security mode\n"); @@ -299,39 +234,56 @@ void etnaviv_iommuv2_restore(struct etnaviv_gpu *gpu) } } -static const struct etnaviv_iommu_domain_ops etnaviv_iommuv2_ops = { - .free = etnaviv_iommuv2_domain_free, +const struct etnaviv_iommu_ops etnaviv_iommuv2_ops = { + .free = etnaviv_iommuv2_free, .map = etnaviv_iommuv2_map, .unmap = etnaviv_iommuv2_unmap, .dump_size = etnaviv_iommuv2_dump_size, .dump = etnaviv_iommuv2_dump, + .restore = etnaviv_iommuv2_restore, }; -struct etnaviv_iommu_domain * -etnaviv_iommuv2_domain_alloc(struct etnaviv_gpu *gpu) +struct etnaviv_iommu_context * +etnaviv_iommuv2_context_alloc(struct etnaviv_iommu_global *global) { - struct etnaviv_iommuv2_domain *etnaviv_domain; - struct etnaviv_iommu_domain *domain; - int ret; + struct etnaviv_iommuv2_context *v2_context; + struct etnaviv_iommu_context *context; - etnaviv_domain = vzalloc(sizeof(*etnaviv_domain)); - if (!etnaviv_domain) + v2_context = vzalloc(sizeof(*v2_context)); + if (!v2_context) return NULL; - domain = &etnaviv_domain->base; + mutex_lock(&global->lock); + v2_context->id = find_first_zero_bit(global->v2.pta_alloc, + ETNAVIV_PTA_ENTRIES); + if (v2_context->id < ETNAVIV_PTA_ENTRIES) { + set_bit(v2_context->id, global->v2.pta_alloc); + } else { + mutex_unlock(&global->lock); + goto out_free; + } + mutex_unlock(&global->lock); - domain->dev = gpu->dev; - domain->base = SZ_4K; - domain->size = (u64)SZ_1G * 4 - SZ_4K; - domain->ops = &etnaviv_iommuv2_ops; + v2_context->mtlb_cpu = dma_alloc_wc(global->dev, SZ_4K, + &v2_context->mtlb_dma, GFP_KERNEL); + if (!v2_context->mtlb_cpu) + goto out_free_id; - ret = etnaviv_iommuv2_init(etnaviv_domain); - if (ret) - goto out_free; + memset32(v2_context->mtlb_cpu, MMUv2_PTE_EXCEPTION, + MMUv2_MAX_STLB_ENTRIES); + + context = &v2_context->base; + context->global = global; + kref_init(&context->refcount); + mutex_init(&context->lock); + INIT_LIST_HEAD(&context->mappings); + drm_mm_init(&context->mm, SZ_4K, (u64)SZ_1G * 4 - SZ_4K); - return &etnaviv_domain->base; + return context; +out_free_id: + clear_bit(v2_context->id, global->v2.pta_alloc); out_free: - vfree(etnaviv_domain); + vfree(v2_context); return NULL; } diff --git a/drivers/gpu/drm/etnaviv/etnaviv_mmu.c b/drivers/gpu/drm/etnaviv/etnaviv_mmu.c index bbd1624a3df8..2f64eef773ed 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_mmu.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_mmu.c @@ -3,6 +3,7 @@ * Copyright (C) 2015-2018 Etnaviv Project */ +#include #include #include "common.xml.h" @@ -10,10 +11,9 @@ #include "etnaviv_drv.h" #include "etnaviv_gem.h" #include "etnaviv_gpu.h" -#include "etnaviv_iommu.h" #include "etnaviv_mmu.h" -static void etnaviv_domain_unmap(struct etnaviv_iommu_domain *domain, +static void etnaviv_context_unmap(struct etnaviv_iommu_context *context, unsigned long iova, size_t size) { size_t unmapped_page, unmapped = 0; @@ -26,7 +26,8 @@ static void etnaviv_domain_unmap(struct etnaviv_iommu_domain *domain, } while (unmapped < size) { - unmapped_page = domain->ops->unmap(domain, iova, pgsize); + unmapped_page = context->global->ops->unmap(context, iova, + pgsize); if (!unmapped_page) break; @@ -35,7 +36,7 @@ static void etnaviv_domain_unmap(struct etnaviv_iommu_domain *domain, } } -static int etnaviv_domain_map(struct etnaviv_iommu_domain *domain, +static int etnaviv_context_map(struct etnaviv_iommu_context *context, unsigned long iova, phys_addr_t paddr, size_t size, int prot) { @@ -51,7 +52,8 @@ static int etnaviv_domain_map(struct etnaviv_iommu_domain *domain, } while (size) { - ret = domain->ops->map(domain, iova, paddr, pgsize, prot); + ret = context->global->ops->map(context, iova, paddr, pgsize, + prot); if (ret) break; @@ -62,21 +64,19 @@ static int etnaviv_domain_map(struct etnaviv_iommu_domain *domain, /* unroll mapping in case something went wrong */ if (ret) - etnaviv_domain_unmap(domain, orig_iova, orig_size - size); + etnaviv_context_unmap(context, orig_iova, orig_size - size); return ret; } -static int etnaviv_iommu_map(struct etnaviv_iommu *iommu, u32 iova, +static int etnaviv_iommu_map(struct etnaviv_iommu_context *context, u32 iova, struct sg_table *sgt, unsigned len, int prot) -{ - struct etnaviv_iommu_domain *domain = iommu->domain; - struct scatterlist *sg; +{ struct scatterlist *sg; unsigned int da = iova; unsigned int i, j; int ret; - if (!domain || !sgt) + if (!context || !sgt) return -EINVAL; for_each_sg(sgt->sgl, sg, sgt->nents, i) { @@ -85,7 +85,7 @@ static int etnaviv_iommu_map(struct etnaviv_iommu *iommu, u32 iova, VERB("map[%d]: %08x %08x(%zx)", i, iova, pa, bytes); - ret = etnaviv_domain_map(domain, da, pa, bytes, prot); + ret = etnaviv_context_map(context, da, pa, bytes, prot); if (ret) goto fail; @@ -100,16 +100,15 @@ static int etnaviv_iommu_map(struct etnaviv_iommu *iommu, u32 iova, for_each_sg(sgt->sgl, sg, i, j) { size_t bytes = sg_dma_len(sg) + sg->offset; - etnaviv_domain_unmap(domain, da, bytes); + etnaviv_context_unmap(context, da, bytes); da += bytes; } return ret; } -static void etnaviv_iommu_unmap(struct etnaviv_iommu *iommu, u32 iova, +static void etnaviv_iommu_unmap(struct etnaviv_iommu_context *context, u32 iova, struct sg_table *sgt, unsigned len) { - struct etnaviv_iommu_domain *domain = iommu->domain; struct scatterlist *sg; unsigned int da = iova; int i; @@ -117,7 +116,7 @@ static void etnaviv_iommu_unmap(struct etnaviv_iommu *iommu, u32 iova, for_each_sg(sgt->sgl, sg, sgt->nents, i) { size_t bytes = sg_dma_len(sg) + sg->offset; - etnaviv_domain_unmap(domain, da, bytes); + etnaviv_context_unmap(context, da, bytes); VERB("unmap[%d]: %08x(%zx)", i, iova, bytes); @@ -127,24 +126,24 @@ static void etnaviv_iommu_unmap(struct etnaviv_iommu *iommu, u32 iova, } } -static void etnaviv_iommu_remove_mapping(struct etnaviv_iommu *mmu, +static void etnaviv_iommu_remove_mapping(struct etnaviv_iommu_context *context, struct etnaviv_vram_mapping *mapping) { struct etnaviv_gem_object *etnaviv_obj = mapping->object; - etnaviv_iommu_unmap(mmu, mapping->vram_node.start, + etnaviv_iommu_unmap(context, mapping->vram_node.start, etnaviv_obj->sgt, etnaviv_obj->base.size); drm_mm_remove_node(&mapping->vram_node); } -static int etnaviv_iommu_find_iova(struct etnaviv_iommu *mmu, +static int etnaviv_iommu_find_iova(struct etnaviv_iommu_context *context, struct drm_mm_node *node, size_t size) { struct etnaviv_vram_mapping *free = NULL; enum drm_mm_insert_mode mode = DRM_MM_INSERT_LOW; int ret; - lockdep_assert_held(&mmu->lock); + lockdep_assert_held(&context->lock); while (1) { struct etnaviv_vram_mapping *m, *n; @@ -152,17 +151,17 @@ static int etnaviv_iommu_find_iova(struct etnaviv_iommu *mmu, struct list_head list; bool found; - ret = drm_mm_insert_node_in_range(&mmu->mm, node, + ret = drm_mm_insert_node_in_range(&context->mm, node, size, 0, 0, 0, U64_MAX, mode); if (ret != -ENOSPC) break; /* Try to retire some entries */ - drm_mm_scan_init(&scan, &mmu->mm, size, 0, 0, mode); + drm_mm_scan_init(&scan, &context->mm, size, 0, 0, mode); found = 0; INIT_LIST_HEAD(&list); - list_for_each_entry(free, &mmu->mappings, mmu_node) { + list_for_each_entry(free, &context->mappings, mmu_node) { /* If this vram node has not been used, skip this. */ if (!free->vram_node.mm) continue; @@ -204,8 +203,8 @@ static int etnaviv_iommu_find_iova(struct etnaviv_iommu *mmu, * this mapping. */ list_for_each_entry_safe(m, n, &list, scan_node) { - etnaviv_iommu_remove_mapping(mmu, m); - m->mmu = NULL; + etnaviv_iommu_remove_mapping(context, m); + m->context = NULL; list_del_init(&m->mmu_node); list_del_init(&m->scan_node); } @@ -221,7 +220,7 @@ static int etnaviv_iommu_find_iova(struct etnaviv_iommu *mmu, return ret; } -int etnaviv_iommu_map_gem(struct etnaviv_iommu *mmu, +int etnaviv_iommu_map_gem(struct etnaviv_iommu_context *context, struct etnaviv_gem_object *etnaviv_obj, u32 memory_base, struct etnaviv_vram_mapping *mapping) { @@ -231,17 +230,17 @@ int etnaviv_iommu_map_gem(struct etnaviv_iommu *mmu, lockdep_assert_held(&etnaviv_obj->lock); - mutex_lock(&mmu->lock); + mutex_lock(&context->lock); /* v1 MMU can optimize single entry (contiguous) scatterlists */ - if (mmu->version == ETNAVIV_IOMMU_V1 && + if (context->global->version == ETNAVIV_IOMMU_V1 && sgt->nents == 1 && !(etnaviv_obj->flags & ETNA_BO_FORCE_MMU)) { u32 iova; iova = sg_dma_address(sgt->sgl) - memory_base; if (iova < 0x80000000 - sg_dma_len(sgt->sgl)) { mapping->iova = iova; - list_add_tail(&mapping->mmu_node, &mmu->mappings); + list_add_tail(&mapping->mmu_node, &context->mappings); ret = 0; goto unlock; } @@ -249,12 +248,12 @@ int etnaviv_iommu_map_gem(struct etnaviv_iommu *mmu, node = &mapping->vram_node; - ret = etnaviv_iommu_find_iova(mmu, node, etnaviv_obj->base.size); + ret = etnaviv_iommu_find_iova(context, node, etnaviv_obj->base.size); if (ret < 0) goto unlock; mapping->iova = node->start; - ret = etnaviv_iommu_map(mmu, node->start, sgt, etnaviv_obj->base.size, + ret = etnaviv_iommu_map(context, node->start, sgt, etnaviv_obj->base.size, ETNAVIV_PROT_READ | ETNAVIV_PROT_WRITE); if (ret < 0) { @@ -262,84 +261,63 @@ int etnaviv_iommu_map_gem(struct etnaviv_iommu *mmu, goto unlock; } - list_add_tail(&mapping->mmu_node, &mmu->mappings); - mmu->flush_seq++; + list_add_tail(&mapping->mmu_node, &context->mappings); + context->flush_seq++; unlock: - mutex_unlock(&mmu->lock); + mutex_unlock(&context->lock); return ret; } -void etnaviv_iommu_unmap_gem(struct etnaviv_iommu *mmu, +void etnaviv_iommu_unmap_gem(struct etnaviv_iommu_context *context, struct etnaviv_vram_mapping *mapping) { WARN_ON(mapping->use); - mutex_lock(&mmu->lock); + mutex_lock(&context->lock); /* If the vram node is on the mm, unmap and remove the node */ - if (mapping->vram_node.mm == &mmu->mm) - etnaviv_iommu_remove_mapping(mmu, mapping); + if (mapping->vram_node.mm == &context->mm) + etnaviv_iommu_remove_mapping(context, mapping); list_del(&mapping->mmu_node); - mmu->flush_seq++; - mutex_unlock(&mmu->lock); + context->flush_seq++; + mutex_unlock(&context->lock); } -void etnaviv_iommu_destroy(struct etnaviv_iommu *mmu) +static void etnaviv_iommu_context_free(struct kref *kref) { - drm_mm_takedown(&mmu->mm); - mmu->domain->ops->free(mmu->domain); - kfree(mmu); -} + struct etnaviv_iommu_context *context = + container_of(kref, struct etnaviv_iommu_context, refcount); -struct etnaviv_iommu *etnaviv_iommu_new(struct etnaviv_gpu *gpu) + context->global->ops->free(context); +} +void etnaviv_iommu_context_put(struct etnaviv_iommu_context *context) { - enum etnaviv_iommu_version version; - struct etnaviv_iommu *mmu; - - mmu = kzalloc(sizeof(*mmu), GFP_KERNEL); - if (!mmu) - return ERR_PTR(-ENOMEM); - - if (!(gpu->identity.minor_features1 & chipMinorFeatures1_MMU_VERSION)) { - mmu->domain = etnaviv_iommuv1_domain_alloc(gpu); - version = ETNAVIV_IOMMU_V1; - } else { - mmu->domain = etnaviv_iommuv2_domain_alloc(gpu); - version = ETNAVIV_IOMMU_V2; - } - - if (!mmu->domain) { - dev_err(gpu->dev, "Failed to allocate GPU IOMMU domain\n"); - kfree(mmu); - return ERR_PTR(-ENOMEM); - } - - mmu->gpu = gpu; - mmu->version = version; - mutex_init(&mmu->lock); - INIT_LIST_HEAD(&mmu->mappings); - - drm_mm_init(&mmu->mm, mmu->domain->base, mmu->domain->size); - - return mmu; + kref_put(&context->refcount, etnaviv_iommu_context_free); } -void etnaviv_iommu_restore(struct etnaviv_gpu *gpu) +struct etnaviv_iommu_context * +etnaviv_iommu_context_init(struct etnaviv_iommu_global *global) { - if (gpu->mmu->version == ETNAVIV_IOMMU_V1) - etnaviv_iommuv1_restore(gpu); + if (global->version == ETNAVIV_IOMMU_V1) + return etnaviv_iommuv1_context_alloc(global); else - etnaviv_iommuv2_restore(gpu); + return etnaviv_iommuv2_context_alloc(global); +} + +void etnaviv_iommu_restore(struct etnaviv_gpu *gpu, + struct etnaviv_iommu_context *context) +{ + context->global->ops->restore(gpu, context); } -int etnaviv_iommu_get_suballoc_va(struct etnaviv_iommu *mmu, +int etnaviv_iommu_get_suballoc_va(struct etnaviv_iommu_context *context, struct etnaviv_vram_mapping *mapping, u32 memory_base, dma_addr_t paddr, size_t size) { - mutex_lock(&mmu->lock); + mutex_lock(&context->lock); /* * For MMUv1 we don't add the suballoc region to the pagetables, as @@ -347,40 +325,40 @@ int etnaviv_iommu_get_suballoc_va(struct etnaviv_iommu *mmu, * window. Instead we manufacture a mapping to make it look uniform * to the upper layers. */ - if (mmu->version == ETNAVIV_IOMMU_V1) { + if (context->global->version == ETNAVIV_IOMMU_V1) { mapping->iova = paddr - memory_base; } else { struct drm_mm_node *node = &mapping->vram_node; int ret; - ret = etnaviv_iommu_find_iova(mmu, node, size); + ret = etnaviv_iommu_find_iova(context, node, size); if (ret < 0) { - mutex_unlock(&mmu->lock); + mutex_unlock(&context->lock); return ret; } mapping->iova = node->start; - ret = etnaviv_domain_map(mmu->domain, node->start, paddr, size, - ETNAVIV_PROT_READ); + ret = etnaviv_context_map(context, node->start, paddr, size, + ETNAVIV_PROT_READ); if (ret < 0) { drm_mm_remove_node(node); - mutex_unlock(&mmu->lock); + mutex_unlock(&context->lock); return ret; } - mmu->flush_seq++; + context->flush_seq++; } - list_add_tail(&mapping->mmu_node, &mmu->mappings); + list_add_tail(&mapping->mmu_node, &context->mappings); mapping->use = 1; - mutex_unlock(&mmu->lock); + mutex_unlock(&context->lock); return 0; } -void etnaviv_iommu_put_suballoc_va(struct etnaviv_iommu *mmu, +void etnaviv_iommu_put_suballoc_va(struct etnaviv_iommu_context *context, struct etnaviv_vram_mapping *mapping) { struct drm_mm_node *node = &mapping->vram_node; @@ -390,21 +368,104 @@ void etnaviv_iommu_put_suballoc_va(struct etnaviv_iommu *mmu, mapping->use = 0; - if (mmu->version == ETNAVIV_IOMMU_V1) + if (context->global->version == ETNAVIV_IOMMU_V1) return; - mutex_lock(&mmu->lock); - etnaviv_domain_unmap(mmu->domain, node->start, node->size); + mutex_lock(&context->lock); + etnaviv_context_unmap(context, node->start, node->size); drm_mm_remove_node(node); - mutex_unlock(&mmu->lock); + mutex_unlock(&context->lock); +} + +size_t etnaviv_iommu_dump_size(struct etnaviv_iommu_context *context) +{ + return context->global->ops->dump_size(context); +} + +void etnaviv_iommu_dump(struct etnaviv_iommu_context *context, void *buf) +{ + context->global->ops->dump(context, buf); } -size_t etnaviv_iommu_dump_size(struct etnaviv_iommu *iommu) +int etnaviv_iommu_global_init(struct etnaviv_gpu *gpu) { - return iommu->domain->ops->dump_size(iommu->domain); + enum etnaviv_iommu_version version = ETNAVIV_IOMMU_V1; + struct etnaviv_drm_private *priv = gpu->drm->dev_private; + struct etnaviv_iommu_global *global; + struct device *dev = gpu->drm->dev; + + if (gpu->identity.minor_features1 & chipMinorFeatures1_MMU_VERSION) + version = ETNAVIV_IOMMU_V2; + + if (priv->mmu_global) { + if (priv->mmu_global->version != version) { + dev_err(gpu->dev, + "MMU version doesn't match global version\n"); + return -ENXIO; + } + + priv->mmu_global->use++; + return 0; + } + + global = kzalloc(sizeof(*global), GFP_KERNEL); + if (!global) + return -ENOMEM; + + global->bad_page_cpu = dma_alloc_wc(dev, SZ_4K, &global->bad_page_dma, + GFP_KERNEL); + if (!global->bad_page_cpu) + goto free_global; + + memset32(global->bad_page_cpu, 0xdead55aa, SZ_4K / sizeof(u32)); + + if (version == ETNAVIV_IOMMU_V2) { + global->v2.pta_cpu = dma_alloc_wc(dev, ETNAVIV_PTA_SIZE, + &global->v2.pta_dma, GFP_KERNEL); + if (!global->v2.pta_cpu) + goto free_bad_page; + } + + global->dev = dev; + global->version = version; + global->use = 1; + mutex_init(&global->lock); + + if (version == ETNAVIV_IOMMU_V1) + global->ops = &etnaviv_iommuv1_ops; + else + global->ops = &etnaviv_iommuv2_ops; + + priv->mmu_global = global; + + return 0; + +free_bad_page: + dma_free_wc(dev, SZ_4K, global->bad_page_cpu, global->bad_page_dma); +free_global: + kfree(global); + + return -ENOMEM; } -void etnaviv_iommu_dump(struct etnaviv_iommu *iommu, void *buf) +void etnaviv_iommu_global_fini(struct etnaviv_gpu *gpu) { - iommu->domain->ops->dump(iommu->domain, buf); + struct etnaviv_drm_private *priv = gpu->drm->dev_private; + struct etnaviv_iommu_global *global = priv->mmu_global; + + if (--global->use > 0) + return; + + if (global->v2.pta_cpu) + dma_free_wc(global->dev, ETNAVIV_PTA_SIZE, + global->v2.pta_cpu, global->v2.pta_dma); + + if (global->bad_page_cpu) + dma_free_wc(global->dev, SZ_4K, + global->bad_page_cpu, global->bad_page_dma); + + mutex_destroy(&global->lock); + kfree(global); + + priv->mmu_global = NULL; } diff --git a/drivers/gpu/drm/etnaviv/etnaviv_mmu.h b/drivers/gpu/drm/etnaviv/etnaviv_mmu.h index 34afe25df9ca..4438d66db6ab 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_mmu.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_mmu.h @@ -16,33 +16,58 @@ enum etnaviv_iommu_version { struct etnaviv_gpu; struct etnaviv_vram_mapping; -struct etnaviv_iommu_domain; +struct etnaviv_iommu_global; +struct etnaviv_iommu_context; -struct etnaviv_iommu_domain_ops { - void (*free)(struct etnaviv_iommu_domain *); - int (*map)(struct etnaviv_iommu_domain *domain, unsigned long iova, +struct etnaviv_iommu_ops { + struct etnaviv_iommu_context *(*init)(struct etnaviv_iommu_global *); + void (*free)(struct etnaviv_iommu_context *); + int (*map)(struct etnaviv_iommu_context *context, unsigned long iova, phys_addr_t paddr, size_t size, int prot); - size_t (*unmap)(struct etnaviv_iommu_domain *domain, unsigned long iova, + size_t (*unmap)(struct etnaviv_iommu_context *context, unsigned long iova, size_t size); - size_t (*dump_size)(struct etnaviv_iommu_domain *); - void (*dump)(struct etnaviv_iommu_domain *, void *); + size_t (*dump_size)(struct etnaviv_iommu_context *); + void (*dump)(struct etnaviv_iommu_context *, void *); + void (*restore)(struct etnaviv_gpu *, struct etnaviv_iommu_context *); }; -struct etnaviv_iommu_domain { +extern const struct etnaviv_iommu_ops etnaviv_iommuv1_ops; +extern const struct etnaviv_iommu_ops etnaviv_iommuv2_ops; + +#define ETNAVIV_PTA_SIZE SZ_4K +#define ETNAVIV_PTA_ENTRIES (ETNAVIV_PTA_SIZE / sizeof(u64)) + +struct etnaviv_iommu_global { struct device *dev; + enum etnaviv_iommu_version version; + const struct etnaviv_iommu_ops *ops; + unsigned int use; + struct mutex lock; + void *bad_page_cpu; dma_addr_t bad_page_dma; - u64 base; - u64 size; - const struct etnaviv_iommu_domain_ops *ops; + /* + * This union holds members needed by either MMUv1 or MMUv2, which + * can not exist at the same time. + */ + union { + struct { + struct etnaviv_iommu_context *shared_context; + } v1; + struct { + /* P(age) T(able) A(rray) */ + u64 *pta_cpu; + dma_addr_t pta_dma; + struct spinlock pta_lock; + DECLARE_BITMAP(pta_alloc, ETNAVIV_PTA_ENTRIES); + } v2; + }; }; -struct etnaviv_iommu { - struct etnaviv_gpu *gpu; - struct etnaviv_iommu_domain *domain; - - enum etnaviv_iommu_version version; +struct etnaviv_iommu_context { + struct kref refcount; + struct etnaviv_iommu_global *global; /* memory manager for GPU address area */ struct mutex lock; @@ -51,26 +76,40 @@ struct etnaviv_iommu { unsigned int flush_seq; }; +int etnaviv_iommu_global_init(struct etnaviv_gpu *gpu); +void etnaviv_iommu_global_fini(struct etnaviv_gpu *gpu); + struct etnaviv_gem_object; -int etnaviv_iommu_map_gem(struct etnaviv_iommu *mmu, +int etnaviv_iommu_map_gem(struct etnaviv_iommu_context *context, struct etnaviv_gem_object *etnaviv_obj, u32 memory_base, struct etnaviv_vram_mapping *mapping); -void etnaviv_iommu_unmap_gem(struct etnaviv_iommu *mmu, +void etnaviv_iommu_unmap_gem(struct etnaviv_iommu_context *context, struct etnaviv_vram_mapping *mapping); -int etnaviv_iommu_get_suballoc_va(struct etnaviv_iommu *mmu, +int etnaviv_iommu_get_suballoc_va(struct etnaviv_iommu_context *ctx, struct etnaviv_vram_mapping *mapping, u32 memory_base, dma_addr_t paddr, size_t size); -void etnaviv_iommu_put_suballoc_va(struct etnaviv_iommu *mmu, +void etnaviv_iommu_put_suballoc_va(struct etnaviv_iommu_context *ctx, struct etnaviv_vram_mapping *mapping); -size_t etnaviv_iommu_dump_size(struct etnaviv_iommu *iommu); -void etnaviv_iommu_dump(struct etnaviv_iommu *iommu, void *buf); - -struct etnaviv_iommu *etnaviv_iommu_new(struct etnaviv_gpu *gpu); -void etnaviv_iommu_destroy(struct etnaviv_iommu *iommu); -void etnaviv_iommu_restore(struct etnaviv_gpu *gpu); +size_t etnaviv_iommu_dump_size(struct etnaviv_iommu_context *ctx); +void etnaviv_iommu_dump(struct etnaviv_iommu_context *ctx, void *buf); + +struct etnaviv_iommu_context * +etnaviv_iommu_context_init(struct etnaviv_iommu_global *global); +static inline void etnaviv_iommu_context_get(struct etnaviv_iommu_context *ctx) +{ + kref_get(&ctx->refcount); +} +void etnaviv_iommu_context_put(struct etnaviv_iommu_context *ctx); +void etnaviv_iommu_restore(struct etnaviv_gpu *gpu, + struct etnaviv_iommu_context *ctx); + +struct etnaviv_iommu_context * +etnaviv_iommuv1_context_alloc(struct etnaviv_iommu_global *global); +struct etnaviv_iommu_context * +etnaviv_iommuv2_context_alloc(struct etnaviv_iommu_global *global); #endif /* __ETNAVIV_MMU_H__ */ From patchwork Fri Aug 9 12:04:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Lucas Stach X-Patchwork-Id: 11086341 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0F512112C for ; Fri, 9 Aug 2019 12:04:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EFBF428C75 for ; Fri, 9 Aug 2019 12:04:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E476F28C84; Fri, 9 Aug 2019 12:04:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 8922128C75 for ; Fri, 9 Aug 2019 12:04:43 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 67FC86EDEE; Fri, 9 Aug 2019 12:04:37 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from metis.ext.pengutronix.de (metis.ext.pengutronix.de [IPv6:2001:67c:670:201:290:27ff:fe1d:cc33]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8278B89565 for ; Fri, 9 Aug 2019 12:04:28 +0000 (UTC) Received: from dude02.hi.pengutronix.de ([2001:67c:670:100:1d::28] helo=dude02.pengutronix.de.) by metis.ext.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1hw3dA-00055N-Cl; Fri, 09 Aug 2019 14:04:24 +0200 From: Lucas Stach To: etnaviv@lists.freedesktop.org Subject: [PATCH v3 6/8] drm/etnaviv: split out starting of FE idle loop Date: Fri, 9 Aug 2019 14:04:22 +0200 Message-Id: <20190809120424.32679-6-l.stach@pengutronix.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190809120424.32679-1-l.stach@pengutronix.de> References: <20190809120424.32679-1-l.stach@pengutronix.de> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 2001:67c:670:100:1d::28 X-SA-Exim-Mail-From: l.stach@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: dri-devel@lists.freedesktop.org X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: patchwork-lst@pengutronix.de, kernel@pengutronix.de, dri-devel@lists.freedesktop.org, Russell King Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" X-Virus-Scanned: ClamAV using ClamSMTP Move buffer setup and starting of the FE loop in the kernel ringbuffer into a separate function. This is a preparation to start the FE later in the submit process. Signed-off-by: Lucas Stach Reviewed-by: Philipp Zabel Reviewed-by: Guido Günther --- drivers/gpu/drm/etnaviv/etnaviv_gpu.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gpu.c b/drivers/gpu/drm/etnaviv/etnaviv_gpu.c index 885ca8f92338..3af72a4127b0 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gpu.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gpu.c @@ -600,6 +600,20 @@ void etnaviv_gpu_start_fe(struct etnaviv_gpu *gpu, u32 address, u16 prefetch) } } +static void etnaviv_gpu_start_fe_idleloop(struct etnaviv_gpu *gpu) +{ + u32 address = etnaviv_cmdbuf_get_va(&gpu->buffer, &gpu->cmdbuf_mapping); + u16 prefetch; + + /* setup the MMU */ + etnaviv_iommu_restore(gpu, gpu->mmu_context); + + /* Start command processor */ + prefetch = etnaviv_buffer_init(gpu); + + etnaviv_gpu_start_fe(gpu, address, prefetch); +} + static void etnaviv_gpu_setup_pulse_eater(struct etnaviv_gpu *gpu) { /* @@ -633,8 +647,6 @@ static void etnaviv_gpu_setup_pulse_eater(struct etnaviv_gpu *gpu) static void etnaviv_gpu_hw_init(struct etnaviv_gpu *gpu) { - u16 prefetch; - if ((etnaviv_is_model_rev(gpu, GC320, 0x5007) || etnaviv_is_model_rev(gpu, GC320, 0x5220)) && gpu_read(gpu, VIVS_HI_CHIP_TIME) != 0x2062400) { @@ -680,15 +692,9 @@ static void etnaviv_gpu_hw_init(struct etnaviv_gpu *gpu) /* setup the pulse eater */ etnaviv_gpu_setup_pulse_eater(gpu); - /* setup the MMU */ - etnaviv_iommu_restore(gpu, gpu->mmu_context); - - /* Start command processor */ - prefetch = etnaviv_buffer_init(gpu); - gpu_write(gpu, VIVS_HI_INTR_ENBL, ~0U); - etnaviv_gpu_start_fe(gpu, etnaviv_cmdbuf_get_va(&gpu->buffer, - &gpu->cmdbuf_mapping), prefetch); + + etnaviv_gpu_start_fe_idleloop(gpu); } int etnaviv_gpu_init(struct etnaviv_gpu *gpu) From patchwork Fri Aug 9 12:04:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Lucas Stach X-Patchwork-Id: 11086329 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 32DA81399 for ; Fri, 9 Aug 2019 12:04:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2274228C73 for ; Fri, 9 Aug 2019 12:04:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1644528C84; Fri, 9 Aug 2019 12:04:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B052D28C73 for ; Fri, 9 Aug 2019 12:04:35 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 590F16EDEA; Fri, 9 Aug 2019 12:04:30 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from metis.ext.pengutronix.de (metis.ext.pengutronix.de [IPv6:2001:67c:670:201:290:27ff:fe1d:cc33]) by gabe.freedesktop.org (Postfix) with ESMTPS id 9AE5C89949 for ; Fri, 9 Aug 2019 12:04:28 +0000 (UTC) Received: from dude02.hi.pengutronix.de ([2001:67c:670:100:1d::28] helo=dude02.pengutronix.de.) by metis.ext.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1hw3dA-00055N-DU; Fri, 09 Aug 2019 14:04:24 +0200 From: Lucas Stach To: etnaviv@lists.freedesktop.org Subject: [PATCH v3 7/8] drm/etnaviv: provide MMU context to etnaviv_gem_mapping_get Date: Fri, 9 Aug 2019 14:04:23 +0200 Message-Id: <20190809120424.32679-7-l.stach@pengutronix.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190809120424.32679-1-l.stach@pengutronix.de> References: <20190809120424.32679-1-l.stach@pengutronix.de> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 2001:67c:670:100:1d::28 X-SA-Exim-Mail-From: l.stach@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: dri-devel@lists.freedesktop.org X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: patchwork-lst@pengutronix.de, kernel@pengutronix.de, dri-devel@lists.freedesktop.org, Russell King Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" X-Virus-Scanned: ClamAV using ClamSMTP In preparation to having a context per process, etnaviv_gem_mapping_get should not use the current GPU context, but needs to be told which context to use. Signed-off-by: Lucas Stach Reviewed-by: Philipp Zabel Reviewed-by: Guido Günther --- drivers/gpu/drm/etnaviv/etnaviv_gem.c | 22 ++++++++++++-------- drivers/gpu/drm/etnaviv/etnaviv_gem.h | 3 ++- drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c | 3 ++- 3 files changed, 17 insertions(+), 11 deletions(-) diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem.c b/drivers/gpu/drm/etnaviv/etnaviv_gem.c index 0ccc3c4dffc4..04c8170f76cd 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gem.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem.c @@ -248,7 +248,8 @@ void etnaviv_gem_mapping_unreference(struct etnaviv_vram_mapping *mapping) } struct etnaviv_vram_mapping *etnaviv_gem_mapping_get( - struct drm_gem_object *obj, struct etnaviv_gpu *gpu) + struct drm_gem_object *obj, struct etnaviv_gpu *gpu, + struct etnaviv_iommu_context *mmu_context) { struct etnaviv_gem_object *etnaviv_obj = to_etnaviv_bo(obj); struct etnaviv_vram_mapping *mapping; @@ -256,7 +257,7 @@ struct etnaviv_vram_mapping *etnaviv_gem_mapping_get( int ret = 0; mutex_lock(&etnaviv_obj->lock); - mapping = etnaviv_gem_get_vram_mapping(etnaviv_obj, gpu->mmu_context); + mapping = etnaviv_gem_get_vram_mapping(etnaviv_obj, mmu_context); if (mapping) { /* * Holding the object lock prevents the use count changing @@ -265,12 +266,12 @@ struct etnaviv_vram_mapping *etnaviv_gem_mapping_get( * the MMU owns this mapping to close this race. */ if (mapping->use == 0) { - mutex_lock(&gpu->mmu_context->lock); - if (mapping->context == gpu->mmu_context) + mutex_lock(&mmu_context->lock); + if (mapping->context == mmu_context) mapping->use += 1; else mapping = NULL; - mutex_unlock(&gpu->mmu_context->lock); + mutex_unlock(&mmu_context->lock); if (mapping) goto out; } else { @@ -303,11 +304,12 @@ struct etnaviv_vram_mapping *etnaviv_gem_mapping_get( list_del(&mapping->obj_node); } - mapping->context = gpu->mmu_context; + etnaviv_iommu_context_get(mmu_context); + mapping->context = mmu_context; mapping->use = 1; - ret = etnaviv_iommu_map_gem(gpu->mmu_context, etnaviv_obj, - gpu->memory_base, mapping); + ret = etnaviv_iommu_map_gem(mmu_context, etnaviv_obj, gpu->memory_base, + mapping); if (ret < 0) kfree(mapping); else @@ -529,8 +531,10 @@ void etnaviv_gem_free_object(struct drm_gem_object *obj) WARN_ON(mapping->use); - if (context) + if (context) { etnaviv_iommu_unmap_gem(context, mapping); + etnaviv_iommu_context_put(context); + } list_del(&mapping->obj_node); kfree(mapping); diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem.h b/drivers/gpu/drm/etnaviv/etnaviv_gem.h index 5a004d5e4eaa..f342560b5938 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gem.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem.h @@ -119,7 +119,8 @@ struct page **etnaviv_gem_get_pages(struct etnaviv_gem_object *obj); void etnaviv_gem_put_pages(struct etnaviv_gem_object *obj); struct etnaviv_vram_mapping *etnaviv_gem_mapping_get( - struct drm_gem_object *obj, struct etnaviv_gpu *gpu); + struct drm_gem_object *obj, struct etnaviv_gpu *gpu, + struct etnaviv_iommu_context *mmu_context); void etnaviv_gem_mapping_unreference(struct etnaviv_vram_mapping *mapping); #endif /* __ETNAVIV_GEM_H__ */ diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c index 3f4f6ab388de..7929d258daf8 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c @@ -224,7 +224,8 @@ static int submit_pin_objects(struct etnaviv_gem_submit *submit) struct etnaviv_vram_mapping *mapping; mapping = etnaviv_gem_mapping_get(&etnaviv_obj->base, - submit->gpu); + submit->gpu, + submit->gpu->mmu_context); if (IS_ERR(mapping)) { ret = PTR_ERR(mapping); break; From patchwork Fri Aug 9 12:04:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Lucas Stach X-Patchwork-Id: 11086335 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AD4E81399 for ; Fri, 9 Aug 2019 12:04:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9A91128C73 for ; Fri, 9 Aug 2019 12:04:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8E58B28C8B; Fri, 9 Aug 2019 12:04:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id F3F0228C73 for ; Fri, 9 Aug 2019 12:04:41 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 418676E1EC; Fri, 9 Aug 2019 12:04:37 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from metis.ext.pengutronix.de (metis.ext.pengutronix.de [IPv6:2001:67c:670:201:290:27ff:fe1d:cc33]) by gabe.freedesktop.org (Postfix) with ESMTPS id 9BDD689958 for ; Fri, 9 Aug 2019 12:04:28 +0000 (UTC) Received: from dude02.hi.pengutronix.de ([2001:67c:670:100:1d::28] helo=dude02.pengutronix.de.) by metis.ext.pengutronix.de with esmtp (Exim 4.92) (envelope-from ) id 1hw3dA-00055N-Fa; Fri, 09 Aug 2019 14:04:24 +0200 From: Lucas Stach To: etnaviv@lists.freedesktop.org Subject: [PATCH v3 8/8] drm/etnaviv: implement per-process address spaces on MMUv2 Date: Fri, 9 Aug 2019 14:04:24 +0200 Message-Id: <20190809120424.32679-8-l.stach@pengutronix.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190809120424.32679-1-l.stach@pengutronix.de> References: <20190809120424.32679-1-l.stach@pengutronix.de> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 2001:67c:670:100:1d::28 X-SA-Exim-Mail-From: l.stach@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: dri-devel@lists.freedesktop.org X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: patchwork-lst@pengutronix.de, kernel@pengutronix.de, dri-devel@lists.freedesktop.org, Russell King Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" X-Virus-Scanned: ClamAV using ClamSMTP This builds on top of the MMU contexts introduced earlier. Instead of having one context per GPU core, each GPU client receives its own context. On MMUv1 this still means a single shared pagetable set is used by all clients, but on MMUv2 there is now a distinct set of pagetables for each client. As the command fetch is also translated via the MMU on MMUv2 the kernel command ringbuffer is mapped into each of the client pagetables. As the MMU context switch is a bit of a heavy operation, due to the needed cache and TLB flushing, this patch implements a lazy way of switching the MMU context. The kernel does not have its own MMU context, but reuses the last client context for all of its operations. This has some visible impact, as the GPU can now only be started once a client has submitted some work and we got the client MMU context assigned. Also the MMU context has a different lifetime than the general client context, as the GPU might still execute the kernel command buffer in the context of a client even after the client has completed all GPU work and has been terminated. Only when the GPU is runtime suspended or switches to another clients MMU context is the old context freed up. Signed-off-by: Lucas Stach Reviewed-by: Philipp Zabel Reviewed-by: Guido Günther --- v3: Don't call etnaviv_cmdbuf_suballoc_unmap when mapping failed. --- drivers/gpu/drm/etnaviv/etnaviv_buffer.c | 64 ++++++++--- drivers/gpu/drm/etnaviv/etnaviv_drv.c | 38 ++++++- drivers/gpu/drm/etnaviv/etnaviv_drv.h | 6 +- drivers/gpu/drm/etnaviv/etnaviv_dump.c | 4 +- drivers/gpu/drm/etnaviv/etnaviv_gem.c | 7 +- drivers/gpu/drm/etnaviv/etnaviv_gem.h | 4 +- drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c | 11 +- drivers/gpu/drm/etnaviv/etnaviv_gpu.c | 105 ++++++++----------- drivers/gpu/drm/etnaviv/etnaviv_gpu.h | 4 - drivers/gpu/drm/etnaviv/etnaviv_iommu.c | 10 +- drivers/gpu/drm/etnaviv/etnaviv_iommu_v2.c | 17 ++- drivers/gpu/drm/etnaviv/etnaviv_mmu.c | 42 ++++++-- drivers/gpu/drm/etnaviv/etnaviv_mmu.h | 11 +- 13 files changed, 208 insertions(+), 115 deletions(-) diff --git a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c index 4324b098689f..876a035ee1a2 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c @@ -118,7 +118,8 @@ static void etnaviv_buffer_dump(struct etnaviv_gpu *gpu, u32 *ptr = buf->vaddr + off; dev_info(gpu->dev, "virt %p phys 0x%08x free 0x%08x\n", - ptr, etnaviv_cmdbuf_get_va(buf, &gpu->cmdbuf_mapping) + + ptr, etnaviv_cmdbuf_get_va(buf, + &gpu->mmu_context->cmdbuf_mapping) + off, size - len * 4 - off); print_hex_dump(KERN_INFO, "cmd ", DUMP_PREFIX_OFFSET, 16, 4, @@ -152,7 +153,8 @@ static u32 etnaviv_buffer_reserve(struct etnaviv_gpu *gpu, if (buffer->user_size + cmd_dwords * sizeof(u64) > buffer->size) buffer->user_size = 0; - return etnaviv_cmdbuf_get_va(buffer, &gpu->cmdbuf_mapping) + + return etnaviv_cmdbuf_get_va(buffer, + &gpu->mmu_context->cmdbuf_mapping) + buffer->user_size; } @@ -166,7 +168,8 @@ u16 etnaviv_buffer_init(struct etnaviv_gpu *gpu) buffer->user_size = 0; CMD_WAIT(buffer); - CMD_LINK(buffer, 2, etnaviv_cmdbuf_get_va(buffer, &gpu->cmdbuf_mapping) + CMD_LINK(buffer, 2, + etnaviv_cmdbuf_get_va(buffer, &gpu->mmu_context->cmdbuf_mapping) + buffer->user_size - 4); return buffer->user_size / 8; @@ -293,7 +296,8 @@ void etnaviv_sync_point_queue(struct etnaviv_gpu *gpu, unsigned int event) /* Append waitlink */ CMD_WAIT(buffer); - CMD_LINK(buffer, 2, etnaviv_cmdbuf_get_va(buffer, &gpu->cmdbuf_mapping) + CMD_LINK(buffer, 2, + etnaviv_cmdbuf_get_va(buffer, &gpu->mmu_context->cmdbuf_mapping) + buffer->user_size - 4); /* @@ -308,7 +312,8 @@ void etnaviv_sync_point_queue(struct etnaviv_gpu *gpu, unsigned int event) /* Append a command buffer to the ring buffer. */ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, - unsigned int event, struct etnaviv_cmdbuf *cmdbuf) + struct etnaviv_iommu_context *mmu_context, unsigned int event, + struct etnaviv_cmdbuf *cmdbuf) { struct etnaviv_cmdbuf *buffer = &gpu->buffer; unsigned int waitlink_offset = buffer->user_size - 16; @@ -317,17 +322,19 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, bool switch_context = gpu->exec_state != exec_state; unsigned int new_flush_seq = READ_ONCE(gpu->mmu_context->flush_seq); bool need_flush = gpu->flush_seq != new_flush_seq; + bool switch_mmu_context = gpu->mmu_context != mmu_context; lockdep_assert_held(&gpu->lock); if (drm_debug & DRM_UT_DRIVER) etnaviv_buffer_dump(gpu, buffer, 0, 0x50); - link_target = etnaviv_cmdbuf_get_va(cmdbuf, &gpu->cmdbuf_mapping); + link_target = etnaviv_cmdbuf_get_va(cmdbuf, + &gpu->mmu_context->cmdbuf_mapping); link_dwords = cmdbuf->size / 8; /* - * If we need maintanence prior to submitting this buffer, we will + * If we need maintenance prior to submitting this buffer, we will * need to append a mmu flush load state, followed by a new * link to this buffer - a total of four additional words. */ @@ -349,7 +356,24 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, if (switch_context) extra_dwords += 4; + /* PTA load command */ + if (switch_mmu_context && gpu->sec_mode == ETNA_SEC_KERNEL) + extra_dwords += 1; + target = etnaviv_buffer_reserve(gpu, buffer, extra_dwords); + /* + * Switch MMU context if necessary. Must be done after the + * link target has been calculated, as the jump forward in the + * kernel ring still uses the last active MMU context before + * the switch. + */ + if (switch_mmu_context) { + struct etnaviv_iommu_context *old_context = gpu->mmu_context; + + etnaviv_iommu_context_get(mmu_context); + gpu->mmu_context = mmu_context; + etnaviv_iommu_context_put(old_context); + } if (need_flush) { /* Add the MMU flush */ @@ -361,10 +385,23 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, VIVS_GL_FLUSH_MMU_FLUSH_PEMMU | VIVS_GL_FLUSH_MMU_FLUSH_UNK4); } else { + u32 flush = VIVS_MMUv2_CONFIGURATION_MODE_MASK | + VIVS_MMUv2_CONFIGURATION_FLUSH_FLUSH; + + if (switch_mmu_context && + gpu->sec_mode == ETNA_SEC_KERNEL) { + unsigned short id = + etnaviv_iommuv2_get_pta_id(gpu->mmu_context); + CMD_LOAD_STATE(buffer, + VIVS_MMUv2_PTA_CONFIG, + VIVS_MMUv2_PTA_CONFIG_INDEX(id)); + } + + if (gpu->sec_mode == ETNA_SEC_NONE) + flush |= etnaviv_iommuv2_get_mtlb_addr(gpu->mmu_context); + CMD_LOAD_STATE(buffer, VIVS_MMUv2_CONFIGURATION, - VIVS_MMUv2_CONFIGURATION_MODE_MASK | - VIVS_MMUv2_CONFIGURATION_ADDRESS_MASK | - VIVS_MMUv2_CONFIGURATION_FLUSH_FLUSH); + flush); CMD_SEM(buffer, SYNC_RECIPIENT_FE, SYNC_RECIPIENT_PE); CMD_STALL(buffer, SYNC_RECIPIENT_FE, @@ -380,6 +417,8 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, } /* And the link to the submitted buffer */ + link_target = etnaviv_cmdbuf_get_va(cmdbuf, + &gpu->mmu_context->cmdbuf_mapping); CMD_LINK(buffer, link_dwords, link_target); /* Update the link target to point to above instructions */ @@ -416,13 +455,14 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, CMD_LOAD_STATE(buffer, VIVS_GL_EVENT, VIVS_GL_EVENT_EVENT_ID(event) | VIVS_GL_EVENT_FROM_PE); CMD_WAIT(buffer); - CMD_LINK(buffer, 2, etnaviv_cmdbuf_get_va(buffer, &gpu->cmdbuf_mapping) + CMD_LINK(buffer, 2, + etnaviv_cmdbuf_get_va(buffer, &gpu->mmu_context->cmdbuf_mapping) + buffer->user_size - 4); if (drm_debug & DRM_UT_DRIVER) pr_info("stream link to 0x%08x @ 0x%08x %p\n", return_target, - etnaviv_cmdbuf_get_va(cmdbuf, &gpu->cmdbuf_mapping), + etnaviv_cmdbuf_get_va(cmdbuf, &gpu->mmu_context->cmdbuf_mapping), cmdbuf->vaddr); if (drm_debug & DRM_UT_DRIVER) { diff --git a/drivers/gpu/drm/etnaviv/etnaviv_drv.c b/drivers/gpu/drm/etnaviv/etnaviv_drv.c index eb0c23fe979a..80f1edcbbea0 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_drv.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_drv.c @@ -50,12 +50,19 @@ static int etnaviv_open(struct drm_device *dev, struct drm_file *file) { struct etnaviv_drm_private *priv = dev->dev_private; struct etnaviv_file_private *ctx; - int i; + int ret, i; ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); if (!ctx) return -ENOMEM; + ctx->mmu = etnaviv_iommu_context_init(priv->mmu_global, + priv->cmdbuf_suballoc); + if (!ctx->mmu) { + ret = -ENOMEM; + goto out_free; + } + for (i = 0; i < ETNA_MAX_PIPES; i++) { struct etnaviv_gpu *gpu = priv->gpu[i]; struct drm_sched_rq *rq; @@ -70,6 +77,10 @@ static int etnaviv_open(struct drm_device *dev, struct drm_file *file) file->driver_priv = ctx; return 0; + +out_free: + kfree(ctx); + return ret; } static void etnaviv_postclose(struct drm_device *dev, struct drm_file *file) @@ -85,6 +96,8 @@ static void etnaviv_postclose(struct drm_device *dev, struct drm_file *file) drm_sched_entity_destroy(&ctx->sched_entity[i]); } + etnaviv_iommu_context_put(ctx->mmu); + kfree(ctx); } @@ -116,12 +129,29 @@ static int etnaviv_mm_show(struct drm_device *dev, struct seq_file *m) static int etnaviv_mmu_show(struct etnaviv_gpu *gpu, struct seq_file *m) { struct drm_printer p = drm_seq_file_printer(m); + struct etnaviv_iommu_context *mmu_context; seq_printf(m, "Active Objects (%s):\n", dev_name(gpu->dev)); - mutex_lock(&gpu->mmu_context->lock); - drm_mm_print(&gpu->mmu_context->mm, &p); - mutex_unlock(&gpu->mmu_context->lock); + /* + * Lock the GPU to avoid a MMU context switch just now and elevate + * the refcount of the current context to avoid it disappearing from + * under our feet. + */ + mutex_lock(&gpu->lock); + mmu_context = gpu->mmu_context; + if (mmu_context) + etnaviv_iommu_context_get(mmu_context); + mutex_unlock(&gpu->lock); + + if (!mmu_context) + return 0; + + mutex_lock(&mmu_context->lock); + drm_mm_print(&mmu_context->mm, &p); + mutex_unlock(&mmu_context->lock); + + etnaviv_iommu_context_put(mmu_context); return 0; } diff --git a/drivers/gpu/drm/etnaviv/etnaviv_drv.h b/drivers/gpu/drm/etnaviv/etnaviv_drv.h index 5f8db08f1c17..a488cfdb6bbf 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_drv.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_drv.h @@ -25,10 +25,7 @@ struct etnaviv_gem_submit; struct etnaviv_iommu_global; struct etnaviv_file_private { - /* - * When per-context address spaces are supported we'd keep track of - * the context's page-tables here. - */ + struct etnaviv_iommu_context *mmu; struct drm_sched_entity sched_entity[ETNA_MAX_PIPES]; }; @@ -75,6 +72,7 @@ u16 etnaviv_buffer_config_pta(struct etnaviv_gpu *gpu, unsigned short id); void etnaviv_buffer_end(struct etnaviv_gpu *gpu); void etnaviv_sync_point_queue(struct etnaviv_gpu *gpu, unsigned int event); void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, + struct etnaviv_iommu_context *mmu, unsigned int event, struct etnaviv_cmdbuf *cmdbuf); void etnaviv_validate_init(void); bool etnaviv_cmd_validate_one(struct etnaviv_gpu *gpu, diff --git a/drivers/gpu/drm/etnaviv/etnaviv_dump.c b/drivers/gpu/drm/etnaviv/etnaviv_dump.c index 7e6791517693..698db540972c 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_dump.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_dump.c @@ -173,12 +173,12 @@ void etnaviv_core_dump(struct etnaviv_gem_submit *submit) etnaviv_core_dump_mem(&iter, ETDUMP_BUF_RING, gpu->buffer.vaddr, gpu->buffer.size, etnaviv_cmdbuf_get_va(&gpu->buffer, - &gpu->cmdbuf_mapping)); + &gpu->mmu_context->cmdbuf_mapping)); etnaviv_core_dump_mem(&iter, ETDUMP_BUF_CMD, submit->cmdbuf.vaddr, submit->cmdbuf.size, etnaviv_cmdbuf_get_va(&submit->cmdbuf, - &gpu->cmdbuf_mapping)); + &gpu->mmu_context->cmdbuf_mapping)); /* Reserve space for the bomap */ if (n_bomap_pages) { diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem.c b/drivers/gpu/drm/etnaviv/etnaviv_gem.c index 04c8170f76cd..e79f6ef3659a 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gem.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem.c @@ -248,8 +248,7 @@ void etnaviv_gem_mapping_unreference(struct etnaviv_vram_mapping *mapping) } struct etnaviv_vram_mapping *etnaviv_gem_mapping_get( - struct drm_gem_object *obj, struct etnaviv_gpu *gpu, - struct etnaviv_iommu_context *mmu_context) + struct drm_gem_object *obj, struct etnaviv_iommu_context *mmu_context) { struct etnaviv_gem_object *etnaviv_obj = to_etnaviv_bo(obj); struct etnaviv_vram_mapping *mapping; @@ -308,8 +307,8 @@ struct etnaviv_vram_mapping *etnaviv_gem_mapping_get( mapping->context = mmu_context; mapping->use = 1; - ret = etnaviv_iommu_map_gem(mmu_context, etnaviv_obj, gpu->memory_base, - mapping); + ret = etnaviv_iommu_map_gem(mmu_context, etnaviv_obj, + mmu_context->global->memory_base, mapping); if (ret < 0) kfree(mapping); else diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem.h b/drivers/gpu/drm/etnaviv/etnaviv_gem.h index f342560b5938..1e11659a8842 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gem.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem.h @@ -93,6 +93,7 @@ struct etnaviv_gem_submit { struct kref refcount; struct etnaviv_file_private *ctx; struct etnaviv_gpu *gpu; + struct etnaviv_iommu_context *mmu_context, *prev_mmu_context; struct dma_fence *out_fence, *in_fence; int out_fence_id; struct list_head node; /* GPU active submit list */ @@ -119,8 +120,7 @@ struct page **etnaviv_gem_get_pages(struct etnaviv_gem_object *obj); void etnaviv_gem_put_pages(struct etnaviv_gem_object *obj); struct etnaviv_vram_mapping *etnaviv_gem_mapping_get( - struct drm_gem_object *obj, struct etnaviv_gpu *gpu, - struct etnaviv_iommu_context *mmu_context); + struct drm_gem_object *obj, struct etnaviv_iommu_context *mmu_context); void etnaviv_gem_mapping_unreference(struct etnaviv_vram_mapping *mapping); #endif /* __ETNAVIV_GEM_H__ */ diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c index 7929d258daf8..16e7d371a7ef 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c @@ -224,8 +224,7 @@ static int submit_pin_objects(struct etnaviv_gem_submit *submit) struct etnaviv_vram_mapping *mapping; mapping = etnaviv_gem_mapping_get(&etnaviv_obj->base, - submit->gpu, - submit->gpu->mmu_context); + submit->mmu_context); if (IS_ERR(mapping)) { ret = PTR_ERR(mapping); break; @@ -362,6 +361,12 @@ static void submit_cleanup(struct kref *kref) if (submit->cmdbuf.suballoc) etnaviv_cmdbuf_free(&submit->cmdbuf); + if (submit->mmu_context) + etnaviv_iommu_context_put(submit->mmu_context); + + if (submit->prev_mmu_context) + etnaviv_iommu_context_put(submit->prev_mmu_context); + for (i = 0; i < submit->nr_bos; i++) { struct etnaviv_gem_object *etnaviv_obj = submit->bos[i].obj; @@ -503,6 +508,8 @@ int etnaviv_ioctl_gem_submit(struct drm_device *dev, void *data, goto err_submit_objects; submit->ctx = file->driver_priv; + etnaviv_iommu_context_get(submit->ctx->mmu); + submit->mmu_context = submit->ctx->mmu; submit->exec_state = args->exec_state; submit->flags = args->flags; diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gpu.c b/drivers/gpu/drm/etnaviv/etnaviv_gpu.c index 3af72a4127b0..d8a83ebfce47 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gpu.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gpu.c @@ -602,7 +602,8 @@ void etnaviv_gpu_start_fe(struct etnaviv_gpu *gpu, u32 address, u16 prefetch) static void etnaviv_gpu_start_fe_idleloop(struct etnaviv_gpu *gpu) { - u32 address = etnaviv_cmdbuf_get_va(&gpu->buffer, &gpu->cmdbuf_mapping); + u32 address = etnaviv_cmdbuf_get_va(&gpu->buffer, + &gpu->mmu_context->cmdbuf_mapping); u16 prefetch; /* setup the MMU */ @@ -693,8 +694,6 @@ static void etnaviv_gpu_hw_init(struct etnaviv_gpu *gpu) etnaviv_gpu_setup_pulse_eater(gpu); gpu_write(gpu, VIVS_HI_INTR_ENBL, ~0U); - - etnaviv_gpu_start_fe_idleloop(gpu); } int etnaviv_gpu_init(struct etnaviv_gpu *gpu) @@ -724,28 +723,6 @@ int etnaviv_gpu_init(struct etnaviv_gpu *gpu) goto fail; } - /* - * Set the GPU linear window to be at the end of the DMA window, where - * the CMA area is likely to reside. This ensures that we are able to - * map the command buffers while having the linear window overlap as - * much RAM as possible, so we can optimize mappings for other buffers. - * - * For 3D cores only do this if MC2.0 is present, as with MC1.0 it leads - * to different views of the memory on the individual engines. - */ - if (!(gpu->identity.features & chipFeatures_PIPE_3D) || - (gpu->identity.minor_features0 & chipMinorFeatures0_MC20)) { - u32 dma_mask = (u32)dma_get_required_mask(gpu->dev); - if (dma_mask < PHYS_OFFSET + SZ_2G) - gpu->memory_base = PHYS_OFFSET; - else - gpu->memory_base = dma_mask - SZ_2G + 1; - } else if (PHYS_OFFSET >= SZ_2G) { - dev_info(gpu->dev, "Need to move linear window on MC1.0, disabling TS\n"); - gpu->memory_base = PHYS_OFFSET; - gpu->identity.features &= ~chipFeatures_FAST_CLEAR; - } - /* * On cores with security features supported, we claim control over the * security states. @@ -764,20 +741,26 @@ int etnaviv_gpu_init(struct etnaviv_gpu *gpu) if (ret) goto fail; - gpu->mmu_context = etnaviv_iommu_context_init(priv->mmu_global); - if (IS_ERR(gpu->mmu_context)) { - dev_err(gpu->dev, "Failed to instantiate GPU IOMMU\n"); - ret = PTR_ERR(gpu->mmu_context); - goto iommu_global_fini; - } - - ret = etnaviv_cmdbuf_suballoc_map(priv->cmdbuf_suballoc, - gpu->mmu_context, - &gpu->cmdbuf_mapping, - gpu->memory_base); - if (ret) { - dev_err(gpu->dev, "failed to map cmdbuf suballoc\n"); - goto destroy_iommu; + /* + * Set the GPU linear window to be at the end of the DMA window, where + * the CMA area is likely to reside. This ensures that we are able to + * map the command buffers while having the linear window overlap as + * much RAM as possible, so we can optimize mappings for other buffers. + * + * For 3D cores only do this if MC2.0 is present, as with MC1.0 it leads + * to different views of the memory on the individual engines. + */ + if (!(gpu->identity.features & chipFeatures_PIPE_3D) || + (gpu->identity.minor_features0 & chipMinorFeatures0_MC20)) { + u32 dma_mask = (u32)dma_get_required_mask(gpu->dev); + if (dma_mask < PHYS_OFFSET + SZ_2G) + priv->mmu_global->memory_base = PHYS_OFFSET; + else + priv->mmu_global->memory_base = dma_mask - SZ_2G + 1; + } else if (PHYS_OFFSET >= SZ_2G) { + dev_info(gpu->dev, "Need to move linear window on MC1.0, disabling TS\n"); + priv->mmu_global->memory_base = PHYS_OFFSET; + gpu->identity.features &= ~chipFeatures_FAST_CLEAR; } /* Create buffer: */ @@ -785,15 +768,7 @@ int etnaviv_gpu_init(struct etnaviv_gpu *gpu) PAGE_SIZE); if (ret) { dev_err(gpu->dev, "could not create command buffer\n"); - goto unmap_suballoc; - } - - if (!(gpu->identity.minor_features1 & chipMinorFeatures1_MMU_VERSION) && - etnaviv_cmdbuf_get_va(&gpu->buffer, &gpu->cmdbuf_mapping) > 0x80000000) { - ret = -EINVAL; - dev_err(gpu->dev, - "command buffer outside valid memory window\n"); - goto free_buffer; + goto fail; } /* Setup event management */ @@ -816,14 +791,6 @@ int etnaviv_gpu_init(struct etnaviv_gpu *gpu) return 0; -free_buffer: - etnaviv_cmdbuf_free(&gpu->buffer); -unmap_suballoc: - etnaviv_cmdbuf_suballoc_unmap(gpu->mmu_context, &gpu->cmdbuf_mapping); -destroy_iommu: - etnaviv_iommu_context_put(gpu->mmu_context); -iommu_global_fini: - etnaviv_iommu_global_fini(gpu); fail: pm_runtime_mark_last_busy(gpu->dev); pm_runtime_put_autosuspend(gpu->dev); @@ -1017,6 +984,7 @@ void etnaviv_gpu_recover_hang(struct etnaviv_gpu *gpu) etnaviv_gpu_hw_init(gpu); gpu->exec_state = -1; + gpu->mmu_context = NULL; mutex_unlock(&gpu->lock); pm_runtime_mark_last_busy(gpu->dev); @@ -1323,6 +1291,15 @@ struct dma_fence *etnaviv_gpu_submit(struct etnaviv_gem_submit *submit) goto out_unlock; } + if (!gpu->mmu_context) { + etnaviv_iommu_context_get(submit->mmu_context); + gpu->mmu_context = submit->mmu_context; + etnaviv_gpu_start_fe_idleloop(gpu); + } else { + etnaviv_iommu_context_get(gpu->mmu_context); + submit->prev_mmu_context = gpu->mmu_context; + } + if (submit->nr_pmrs) { gpu->event[event[1]].sync_point = &sync_point_perfmon_sample_pre; kref_get(&submit->refcount); @@ -1332,8 +1309,8 @@ struct dma_fence *etnaviv_gpu_submit(struct etnaviv_gem_submit *submit) gpu->event[event[0]].fence = gpu_fence; submit->cmdbuf.user_size = submit->cmdbuf.size - 8; - etnaviv_buffer_queue(gpu, submit->exec_state, event[0], - &submit->cmdbuf); + etnaviv_buffer_queue(gpu, submit->exec_state, submit->mmu_context, + event[0], &submit->cmdbuf); if (submit->nr_pmrs) { gpu->event[event[2]].sync_point = &sync_point_perfmon_sample_post; @@ -1535,7 +1512,7 @@ int etnaviv_gpu_wait_idle(struct etnaviv_gpu *gpu, unsigned int timeout_ms) static int etnaviv_gpu_hw_suspend(struct etnaviv_gpu *gpu) { - if (gpu->initialized) { + if (gpu->initialized && gpu->mmu_context) { /* Replace the last WAIT with END */ mutex_lock(&gpu->lock); etnaviv_buffer_end(gpu); @@ -1547,8 +1524,13 @@ static int etnaviv_gpu_hw_suspend(struct etnaviv_gpu *gpu) * we fail, just warn and continue. */ etnaviv_gpu_wait_idle(gpu, 100); + + etnaviv_iommu_context_put(gpu->mmu_context); + gpu->mmu_context = NULL; } + gpu->exec_state = -1; + return etnaviv_gpu_clk_disable(gpu); } @@ -1564,8 +1546,6 @@ static int etnaviv_gpu_hw_resume(struct etnaviv_gpu *gpu) etnaviv_gpu_update_clock(gpu); etnaviv_gpu_hw_init(gpu); - gpu->exec_state = -1; - mutex_unlock(&gpu->lock); return 0; @@ -1696,9 +1676,6 @@ static void etnaviv_gpu_unbind(struct device *dev, struct device *master, if (gpu->initialized) { etnaviv_cmdbuf_free(&gpu->buffer); - etnaviv_cmdbuf_suballoc_unmap(gpu->mmu_context, - &gpu->cmdbuf_mapping); - etnaviv_iommu_context_put(gpu->mmu_context); etnaviv_iommu_global_fini(gpu); gpu->initialized = false; } diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gpu.h b/drivers/gpu/drm/etnaviv/etnaviv_gpu.h index c0bd6018d53b..8f9bd4edc96a 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gpu.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_gpu.h @@ -103,13 +103,9 @@ struct etnaviv_gpu { bool initialized; /* 'ring'-buffer: */ - struct etnaviv_vram_mapping cmdbuf_mapping; struct etnaviv_cmdbuf buffer; int exec_state; - /* bus base address of memory */ - u32 memory_base; - /* event management: */ DECLARE_BITMAP(event_bitmap, ETNA_NR_EVENTS); struct etnaviv_event event[ETNA_NR_EVENTS]; diff --git a/drivers/gpu/drm/etnaviv/etnaviv_iommu.c b/drivers/gpu/drm/etnaviv/etnaviv_iommu.c index a2f1ff151822..aac8dbf3ea56 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_iommu.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_iommu.c @@ -93,11 +93,11 @@ static void etnaviv_iommuv1_restore(struct etnaviv_gpu *gpu, u32 pgtable; /* set base addresses */ - gpu_write(gpu, VIVS_MC_MEMORY_BASE_ADDR_RA, gpu->memory_base); - gpu_write(gpu, VIVS_MC_MEMORY_BASE_ADDR_FE, gpu->memory_base); - gpu_write(gpu, VIVS_MC_MEMORY_BASE_ADDR_TX, gpu->memory_base); - gpu_write(gpu, VIVS_MC_MEMORY_BASE_ADDR_PEZ, gpu->memory_base); - gpu_write(gpu, VIVS_MC_MEMORY_BASE_ADDR_PE, gpu->memory_base); + gpu_write(gpu, VIVS_MC_MEMORY_BASE_ADDR_RA, context->global->memory_base); + gpu_write(gpu, VIVS_MC_MEMORY_BASE_ADDR_FE, context->global->memory_base); + gpu_write(gpu, VIVS_MC_MEMORY_BASE_ADDR_TX, context->global->memory_base); + gpu_write(gpu, VIVS_MC_MEMORY_BASE_ADDR_PEZ, context->global->memory_base); + gpu_write(gpu, VIVS_MC_MEMORY_BASE_ADDR_PE, context->global->memory_base); /* set page table address in MC */ pgtable = (u32)v1_context->pgtable_dma; diff --git a/drivers/gpu/drm/etnaviv/etnaviv_iommu_v2.c b/drivers/gpu/drm/etnaviv/etnaviv_iommu_v2.c index 5ca2077c148d..043111a1d60c 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_iommu_v2.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_iommu_v2.c @@ -206,7 +206,7 @@ static void etnaviv_iommuv2_restore_sec(struct etnaviv_gpu *gpu, VIVS_MMUv2_SAFE_ADDRESS_CONFIG_SEC_SAFE_ADDR_HIGH( upper_32_bits(context->global->bad_page_dma))); - context->global->v2.pta_cpu[0] = v2_context->mtlb_dma | + context->global->v2.pta_cpu[v2_context->id] = v2_context->mtlb_dma | VIVS_MMUv2_CONFIGURATION_MODE_MODE4_K; /* trigger a PTA load through the FE */ @@ -218,6 +218,19 @@ static void etnaviv_iommuv2_restore_sec(struct etnaviv_gpu *gpu, gpu_write(gpu, VIVS_MMUv2_SEC_CONTROL, VIVS_MMUv2_SEC_CONTROL_ENABLE); } +u32 etnaviv_iommuv2_get_mtlb_addr(struct etnaviv_iommu_context *context) +{ + struct etnaviv_iommuv2_context *v2_context = to_v2_context(context); + + return v2_context->mtlb_dma; +} + +unsigned short etnaviv_iommuv2_get_pta_id(struct etnaviv_iommu_context *context) +{ + struct etnaviv_iommuv2_context *v2_context = to_v2_context(context); + + return v2_context->id; +} static void etnaviv_iommuv2_restore(struct etnaviv_gpu *gpu, struct etnaviv_iommu_context *context) { @@ -272,6 +285,8 @@ etnaviv_iommuv2_context_alloc(struct etnaviv_iommu_global *global) memset32(v2_context->mtlb_cpu, MMUv2_PTE_EXCEPTION, MMUv2_MAX_STLB_ENTRIES); + global->v2.pta_cpu[v2_context->id] = v2_context->mtlb_dma; + context = &v2_context->base; context->global = global; kref_init(&context->refcount); diff --git a/drivers/gpu/drm/etnaviv/etnaviv_mmu.c b/drivers/gpu/drm/etnaviv/etnaviv_mmu.c index 2f64eef773ed..82822e30bf30 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_mmu.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_mmu.c @@ -290,6 +290,8 @@ static void etnaviv_iommu_context_free(struct kref *kref) struct etnaviv_iommu_context *context = container_of(kref, struct etnaviv_iommu_context, refcount); + etnaviv_cmdbuf_suballoc_unmap(context, &context->cmdbuf_mapping); + context->global->ops->free(context); } void etnaviv_iommu_context_put(struct etnaviv_iommu_context *context) @@ -298,12 +300,28 @@ void etnaviv_iommu_context_put(struct etnaviv_iommu_context *context) } struct etnaviv_iommu_context * -etnaviv_iommu_context_init(struct etnaviv_iommu_global *global) +etnaviv_iommu_context_init(struct etnaviv_iommu_global *global, + struct etnaviv_cmdbuf_suballoc *suballoc) { + struct etnaviv_iommu_context *ctx; + int ret; + if (global->version == ETNAVIV_IOMMU_V1) - return etnaviv_iommuv1_context_alloc(global); + ctx = etnaviv_iommuv1_context_alloc(global); else - return etnaviv_iommuv2_context_alloc(global); + ctx = etnaviv_iommuv2_context_alloc(global); + + if (!ctx) + return NULL; + + ret = etnaviv_cmdbuf_suballoc_map(suballoc, ctx, &ctx->cmdbuf_mapping, + global->memory_base); + if (ret) { + global->ops->free(ctx); + return NULL; + } + + return ctx; } void etnaviv_iommu_restore(struct etnaviv_gpu *gpu, @@ -319,6 +337,12 @@ int etnaviv_iommu_get_suballoc_va(struct etnaviv_iommu_context *context, { mutex_lock(&context->lock); + if (mapping->use > 0) { + mapping->use++; + mutex_unlock(&context->lock); + return 0; + } + /* * For MMUv1 we don't add the suballoc region to the pagetables, as * those GPUs can only work with cmdbufs accessed through the linear @@ -340,7 +364,6 @@ int etnaviv_iommu_get_suballoc_va(struct etnaviv_iommu_context *context, mapping->iova = node->start; ret = etnaviv_context_map(context, node->start, paddr, size, ETNAVIV_PROT_READ); - if (ret < 0) { drm_mm_remove_node(node); mutex_unlock(&context->lock); @@ -363,15 +386,14 @@ void etnaviv_iommu_put_suballoc_va(struct etnaviv_iommu_context *context, { struct drm_mm_node *node = &mapping->vram_node; - if (!mapping->use) - return; - - mapping->use = 0; + mutex_lock(&context->lock); + mapping->use--; - if (context->global->version == ETNAVIV_IOMMU_V1) + if (mapping->use > 0 || context->global->version == ETNAVIV_IOMMU_V1) { + mutex_unlock(&context->lock); return; + } - mutex_lock(&context->lock); etnaviv_context_unmap(context, node->start, node->size); drm_mm_remove_node(node); mutex_unlock(&context->lock); diff --git a/drivers/gpu/drm/etnaviv/etnaviv_mmu.h b/drivers/gpu/drm/etnaviv/etnaviv_mmu.h index 4438d66db6ab..c01491a6c4d8 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_mmu.h +++ b/drivers/gpu/drm/etnaviv/etnaviv_mmu.h @@ -47,6 +47,8 @@ struct etnaviv_iommu_global { void *bad_page_cpu; dma_addr_t bad_page_dma; + u32 memory_base; + /* * This union holds members needed by either MMUv1 or MMUv2, which * can not exist at the same time. @@ -74,6 +76,9 @@ struct etnaviv_iommu_context { struct list_head mappings; struct drm_mm mm; unsigned int flush_seq; + + /* Not part of the context, but needs to have the same lifetime */ + struct etnaviv_vram_mapping cmdbuf_mapping; }; int etnaviv_iommu_global_init(struct etnaviv_gpu *gpu); @@ -98,7 +103,8 @@ size_t etnaviv_iommu_dump_size(struct etnaviv_iommu_context *ctx); void etnaviv_iommu_dump(struct etnaviv_iommu_context *ctx, void *buf); struct etnaviv_iommu_context * -etnaviv_iommu_context_init(struct etnaviv_iommu_global *global); +etnaviv_iommu_context_init(struct etnaviv_iommu_global *global, + struct etnaviv_cmdbuf_suballoc *suballoc); static inline void etnaviv_iommu_context_get(struct etnaviv_iommu_context *ctx) { kref_get(&ctx->refcount); @@ -112,4 +118,7 @@ etnaviv_iommuv1_context_alloc(struct etnaviv_iommu_global *global); struct etnaviv_iommu_context * etnaviv_iommuv2_context_alloc(struct etnaviv_iommu_global *global); +u32 etnaviv_iommuv2_get_mtlb_addr(struct etnaviv_iommu_context *context); +unsigned short etnaviv_iommuv2_get_pta_id(struct etnaviv_iommu_context *context); + #endif /* __ETNAVIV_MMU_H__ */