From patchwork Fri Oct 6 13:31:42 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 9989477 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 0F0706020F for ; Fri, 6 Oct 2017 13:45:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E2249283D9 for ; Fri, 6 Oct 2017 13:45:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D5DFB28CAC; Fri, 6 Oct 2017 13:45:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 38B88283D9 for ; Fri, 6 Oct 2017 13:45:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Yo2sQ6rRrdofnPUZQMGvmzRBUtiqlrWqgR6FrNHsH7c=; b=A1y3bCvVbyALThdZ6HRGy6BjYY RHhRVtx/FwS5zyo+m3mFfphN7HMbVJ1o47qgtlLArHPQUpa5RyN9N4OQimhWhrnIFRp7aUlgh+OVC MkQ8DYszTgjvl7KF5kzvNsnCWt/53rHADgQj6hj4PiK5K5XMDAz20HQrRchQcvT4frpITcRAAZ9nK 5ann9R8Sm05ySn+UHuGrQkqhwSIA4bA/fiogB6W0nT7ZYTuB3HQgBXiiSnTOuMVYa0BmKt4SaRXtT 0ib5a/8n3woYFAVXyB9BKds9rIApj3nnrH3f4OdCEUZWpENf0OL/WLcYoHWYiJwMVZfl1qZE6IhEN rUFkP3wA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1e0Svn-00056V-IL; Fri, 06 Oct 2017 13:44:47 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1e0Svl-00053q-Lv for linux-arm-kernel@bombadil.infradead.org; Fri, 06 Oct 2017 13:44:46 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=kqtupX9D34cPvKftjKFG0uMP3/T/9isdpJLqq/fhP+M=; b=Tk+4YP5pJ6blnjuvYTtTkbJRk +qyNmLNJ/mgWSoiQ/Dgu6FNiH6P5Wfcl/TcWDI8uQqaj8EueousTknwD4+P6KBjwP9beow5F7o2ii wYEZUl+Kpmnukw+8Aq0L19JwJggW50+UCia7o0OpLY1f988lKo3U/IqWKofe1imCkkpbnwS9XZTWj ocKPEjjMRmUmVZDQwqKAaD0Y5FO9VSATDZFHyqZAumnxU0BV2D1gvhIKTBow36Y/tljOkAqR16DUr EcBPHQc4OIk42pIY6ONDfMP4l3dQbQXUC4i+4mHjQI0v5UvDv9u40t7xlkB9YQDttZn7msTvSluHq UfdRmWNCw==; Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by casper.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1e0SoF-0007lZ-G4 for linux-arm-kernel@lists.infradead.org; Fri, 06 Oct 2017 13:37:03 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 33801168F; Fri, 6 Oct 2017 06:29:09 -0700 (PDT) Received: from e106794-lin.cambridge.arm.com (e106794-lin.cambridge.arm.com [10.1.211.72]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5A8103F578; Fri, 6 Oct 2017 06:29:04 -0700 (PDT) From: Jean-Philippe Brucker To: linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-acpi@vger.kernel.org, devicetree@vger.kernel.org, iommu@lists.linux-foundation.org Subject: [RFCv2 PATCH 15/36] iommu/arm-smmu-v3: Add second level of context descriptor table Date: Fri, 6 Oct 2017 14:31:42 +0100 Message-Id: <20171006133203.22803-16-jean-philippe.brucker@arm.com> X-Mailer: git-send-email 2.13.3 In-Reply-To: <20171006133203.22803-1-jean-philippe.brucker@arm.com> References: <20171006133203.22803-1-jean-philippe.brucker@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20171006_143659_852021_646072D7 X-CRM114-Status: GOOD ( 21.73 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, xieyisheng1@huawei.com, gabriele.paoloni@huawei.com, catalin.marinas@arm.com, will.deacon@arm.com, okaya@codeaurora.org, yi.l.liu@intel.com, lorenzo.pieralisi@arm.com, ashok.raj@intel.com, tn@semihalf.com, joro@8bytes.org, rfranz@cavium.com, lenb@kernel.org, jacob.jun.pan@linux.intel.com, alex.williamson@redhat.com, robh+dt@kernel.org, thunder.leizhen@huawei.com, bhelgaas@google.com, dwmw2@infradead.org, liubo95@huawei.com, rjw@rjwysocki.net, robdclark@gmail.com, hanjun.guo@linaro.org, sudeep.holla@arm.com, robin.murphy@arm.com, nwatters@codeaurora.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP The SMMU can support up to 20 bits of SSID. Add a second level of page tables to accommodate this. Devices that support more than 1024 SSIDs now have a table of 1024 L1 entries (8kB), pointing to tables of 1024 context descriptors (64kB), allocated on demand. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 198 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 179 insertions(+), 19 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index 37061e1cbae4..c444f9e83b91 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -239,6 +239,8 @@ #define STRTAB_STE_0_S1FMT_SHIFT 4 #define STRTAB_STE_0_S1FMT_LINEAR (0UL << STRTAB_STE_0_S1FMT_SHIFT) +#define STRTAB_STE_0_S1FMT_4K_L2 (1UL << STRTAB_STE_0_S1FMT_SHIFT) +#define STRTAB_STE_0_S1FMT_64K_L2 (2UL << STRTAB_STE_0_S1FMT_SHIFT) #define STRTAB_STE_0_S1CTXPTR_SHIFT 6 #define STRTAB_STE_0_S1CTXPTR_MASK 0x3ffffffffffUL #define STRTAB_STE_0_S1CDMAX_SHIFT 59 @@ -287,7 +289,21 @@ #define STRTAB_STE_3_S2TTB_SHIFT 4 #define STRTAB_STE_3_S2TTB_MASK 0xfffffffffffUL -/* Context descriptor (stage-1 only) */ +/* + * Context descriptor + * + * Linear: when less than 1024 SSIDs are supported + * 2lvl: at most 1024 L1 entrie, + * 1024 lazy entries per table. + */ +#define CTXDESC_SPLIT 10 +#define CTXDESC_NUM_L2_ENTRIES (1 << CTXDESC_SPLIT) + +#define CTXDESC_L1_DESC_DWORD 1 +#define CTXDESC_L1_DESC_VALID 1 +#define CTXDESC_L1_DESC_L2PTR_SHIFT 12 +#define CTXDESC_L1_DESC_L2PTR_MASK 0xfffffffffUL + #define CTXDESC_CD_DWORDS 8 #define CTXDESC_CD_0_TCR_T0SZ_SHIFT 0 #define ARM64_TCR_T0SZ_SHIFT 0 @@ -567,9 +583,24 @@ struct arm_smmu_ctx_desc { u64 mair; }; -struct arm_smmu_s1_cfg { +struct arm_smmu_cd_table { __le64 *cdptr; dma_addr_t cdptr_dma; +}; + +struct arm_smmu_s1_cfg { + bool linear; + + union { + struct arm_smmu_cd_table table; + struct { + __le64 *ptr; + dma_addr_t ptr_dma; + size_t num_entries; + + struct arm_smmu_cd_table *tables; + } l1; + }; size_t num_contexts; @@ -1000,7 +1031,8 @@ static void arm_smmu_cmdq_issue_cmd(struct arm_smmu_device *smmu, } /* Context descriptor manipulation functions */ -static void arm_smmu_sync_cd(struct arm_smmu_domain *smmu_domain, u32 ssid) +static void arm_smmu_sync_cd(struct arm_smmu_domain *smmu_domain, u32 ssid, + bool leaf) { size_t i; unsigned long flags; @@ -1010,7 +1042,7 @@ static void arm_smmu_sync_cd(struct arm_smmu_domain *smmu_domain, u32 ssid) .opcode = CMDQ_OP_CFGI_CD, .cfgi = { .ssid = ssid, - .leaf = true, + .leaf = leaf, }, }; @@ -1029,6 +1061,69 @@ static void arm_smmu_sync_cd(struct arm_smmu_domain *smmu_domain, u32 ssid) arm_smmu_cmdq_issue_cmd(smmu, &cmd); } +static int arm_smmu_alloc_cd_leaf_table(struct arm_smmu_device *smmu, + struct arm_smmu_cd_table *desc, + size_t num_entries) +{ + size_t size = num_entries * (CTXDESC_CD_DWORDS << 3); + + desc->cdptr = dmam_alloc_coherent(smmu->dev, size, &desc->cdptr_dma, + GFP_ATOMIC | __GFP_ZERO); + if (!desc->cdptr) + return -ENOMEM; + + return 0; +} + +static void arm_smmu_free_cd_leaf_table(struct arm_smmu_device *smmu, + struct arm_smmu_cd_table *desc, + size_t num_entries) +{ + size_t size = num_entries * (CTXDESC_CD_DWORDS << 3); + + dmam_free_coherent(smmu->dev, size, desc->cdptr, desc->cdptr_dma); +} + +static void arm_smmu_write_cd_l1_desc(__le64 *dst, + struct arm_smmu_cd_table *table) +{ + u64 val = (table->cdptr_dma & CTXDESC_L1_DESC_L2PTR_MASK + << CTXDESC_L1_DESC_L2PTR_SHIFT) | CTXDESC_L1_DESC_VALID; + + *dst = cpu_to_le64(val); +} + +static __u64 *arm_smmu_get_cd_ptr(struct arm_smmu_domain *smmu_domain, u32 ssid) +{ + unsigned long idx; + struct arm_smmu_cd_table *l1_desc; + struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg; + + if (cfg->linear) + return cfg->table.cdptr + ssid * CTXDESC_CD_DWORDS; + + idx = ssid >> CTXDESC_SPLIT; + if (idx >= cfg->l1.num_entries) + return NULL; + + l1_desc = &cfg->l1.tables[idx]; + if (!l1_desc->cdptr) { + __le64 *l1ptr = cfg->l1.ptr + idx * CTXDESC_L1_DESC_DWORD; + + if (arm_smmu_alloc_cd_leaf_table(smmu_domain->smmu, l1_desc, + CTXDESC_NUM_L2_ENTRIES)) + return NULL; + + arm_smmu_write_cd_l1_desc(l1ptr, l1_desc); + /* An invalid L1 entry is allowed to be cached */ + arm_smmu_sync_cd(smmu_domain, idx << CTXDESC_SPLIT, false); + } + + idx = ssid & (CTXDESC_NUM_L2_ENTRIES - 1); + + return l1_desc->cdptr + idx * CTXDESC_CD_DWORDS; +} + static u64 arm_smmu_cpu_tcr_to_cd(u64 tcr) { u64 val = 0; @@ -1052,7 +1147,7 @@ static void arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, { u64 val; bool cd_live; - __u64 *cdptr = (__u64 *)smmu_domain->s1_cfg.cdptr + ssid * CTXDESC_CD_DWORDS; + __u64 *cdptr = arm_smmu_get_cd_ptr(smmu_domain, ssid); /* * This function handles the following cases: @@ -1067,6 +1162,9 @@ static void arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, * (4) Remove a secondary CD and invalidate it. */ + if (WARN_ON(!cdptr)) + return; + val = le64_to_cpu(cdptr[0]); cd_live = !!(val & CTXDESC_CD_0_V); @@ -1074,7 +1172,7 @@ static void arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, /* (4) */ cdptr[0] = 0; if (ssid) - arm_smmu_sync_cd(smmu_domain, ssid); + arm_smmu_sync_cd(smmu_domain, ssid, true); return; } @@ -1102,7 +1200,7 @@ static void arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, * time. Ensure it observes the rest of the CD before we * enable it. */ - arm_smmu_sync_cd(smmu_domain, ssid); + arm_smmu_sync_cd(smmu_domain, ssid, true); val = arm_smmu_cpu_tcr_to_cd(cd->tcr) | #ifdef __BIG_ENDIAN @@ -1122,12 +1220,15 @@ static void arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, } if (ssid || cd_live) - arm_smmu_sync_cd(smmu_domain, ssid); + arm_smmu_sync_cd(smmu_domain, ssid, true); } static int arm_smmu_alloc_cd_tables(struct arm_smmu_domain *smmu_domain) { + int ret; int num_ssids; + size_t num_leaf_entries, size = 0; + struct arm_smmu_cd_table *leaf_table; struct arm_smmu_device *smmu = smmu_domain->smmu; struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg; @@ -1135,28 +1236,80 @@ static int arm_smmu_alloc_cd_tables(struct arm_smmu_domain *smmu_domain) return -EINVAL; num_ssids = cfg->num_contexts; + if (num_ssids <= CTXDESC_NUM_L2_ENTRIES) { + /* Fits in a single table */ + cfg->linear = true; + num_leaf_entries = num_ssids; + leaf_table = &cfg->table; + } else { + /* + * SSID[S1CDmax-1:10] indexes 1st-level table, SSID[9:0] indexes + * 2nd-level + */ + cfg->linear = false; + cfg->l1.num_entries = num_ssids / CTXDESC_NUM_L2_ENTRIES; - cfg->cdptr = dmam_alloc_coherent(smmu->dev, - num_ssids * (CTXDESC_CD_DWORDS << 3), - &cfg->cdptr_dma, - GFP_KERNEL | __GFP_ZERO); - if (!cfg->cdptr) - return -ENOMEM; + cfg->l1.tables = devm_kzalloc(smmu->dev, + sizeof(struct arm_smmu_cd_table) * + cfg->l1.num_entries, GFP_KERNEL); + if (!cfg->l1.tables) + return -ENOMEM; + + size = cfg->l1.num_entries * (CTXDESC_L1_DESC_DWORD << 3); + cfg->l1.ptr = dmam_alloc_coherent(smmu->dev, size, + &cfg->l1.ptr_dma, + GFP_KERNEL | __GFP_ZERO); + if (!cfg->l1.ptr) { + devm_kfree(smmu->dev, cfg->l1.tables); + return -ENOMEM; + } + + num_leaf_entries = CTXDESC_NUM_L2_ENTRIES; + leaf_table = cfg->l1.tables; + } + + ret = arm_smmu_alloc_cd_leaf_table(smmu, leaf_table, num_leaf_entries); + if (ret) { + if (!cfg->linear) { + dmam_free_coherent(smmu->dev, size, cfg->l1.ptr, + cfg->l1.ptr_dma); + devm_kfree(smmu->dev, cfg->l1.tables); + } + + return ret; + } + + if (!cfg->linear) + arm_smmu_write_cd_l1_desc(cfg->l1.ptr, leaf_table); return 0; } static void arm_smmu_free_cd_tables(struct arm_smmu_domain *smmu_domain) { + size_t i, size; struct arm_smmu_device *smmu = smmu_domain->smmu; struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg; if (WARN_ON(smmu_domain->stage != ARM_SMMU_DOMAIN_S1)) return; - dmam_free_coherent(smmu->dev, - cfg->num_contexts * (CTXDESC_CD_DWORDS << 3), - cfg->cdptr, cfg->cdptr_dma); + if (cfg->linear) { + arm_smmu_free_cd_leaf_table(smmu, &cfg->table, cfg->num_contexts); + } else { + for (i = 0; i < cfg->l1.num_entries; i++) { + struct arm_smmu_cd_table *desc = &cfg->l1.tables[i]; + + if (!desc->cdptr) + continue; + + arm_smmu_free_cd_leaf_table(smmu, desc, + CTXDESC_NUM_L2_ENTRIES); + } + + size = cfg->l1.num_entries * (CTXDESC_L1_DESC_DWORD << 3); + dmam_free_coherent(smmu->dev, size, cfg->l1.ptr, cfg->l1.ptr_dma); + } } /* Stream table manipulation functions */ @@ -1255,10 +1408,16 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_device *smmu, u32 sid, } if (ste->s1_cfg) { + dma_addr_t s1ctxptr; unsigned int s1cdmax = ilog2(ste->s1_cfg->num_contexts); BUG_ON(ste_live); + if (ste->s1_cfg->linear) + s1ctxptr = ste->s1_cfg->table.cdptr_dma; + else + s1ctxptr = ste->s1_cfg->l1.ptr_dma; + dst[1] = cpu_to_le64( STRTAB_STE_1_S1DSS_SSID0 | STRTAB_STE_1_S1C_CACHE_WBRA @@ -1275,11 +1434,12 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_device *smmu, u32 sid, !(smmu->features & ARM_SMMU_FEAT_STALL_FORCE)) dst[1] |= cpu_to_le64(STRTAB_STE_1_S1STALLD); - val |= (ste->s1_cfg->cdptr_dma & STRTAB_STE_0_S1CTXPTR_MASK + val |= (s1ctxptr & STRTAB_STE_0_S1CTXPTR_MASK << STRTAB_STE_0_S1CTXPTR_SHIFT) | (u64)(s1cdmax & STRTAB_STE_0_S1CDMAX_MASK) << STRTAB_STE_0_S1CDMAX_SHIFT | - STRTAB_STE_0_S1FMT_LINEAR | + (ste->s1_cfg->linear ? STRTAB_STE_0_S1FMT_LINEAR : + STRTAB_STE_0_S1FMT_64K_L2) | STRTAB_STE_0_CFG_S1_TRANS; }