From patchwork Thu Apr 30 14:34:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11520519 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A1F9215AB for ; Thu, 30 Apr 2020 14:40:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5D4D62051A for ; Thu, 30 Apr 2020 14:40:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="gfyrsM/I" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5D4D62051A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3E90C8E0011; Thu, 30 Apr 2020 10:40:35 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 348DD8E0010; Thu, 30 Apr 2020 10:40:35 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C4F38E0011; Thu, 30 Apr 2020 10:40:35 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0179.hostedemail.com [216.40.44.179]) by kanga.kvack.org (Postfix) with ESMTP id ECD468E0010 for ; Thu, 30 Apr 2020 10:40:34 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A28FC483C for ; Thu, 30 Apr 2020 14:40:34 +0000 (UTC) X-FDA: 76764782388.11.cars85_5c48cfa3dc92b X-Spam-Summary: 2,0,0,16a9841281402204,d41d8cd98f00b204,jean-philippe@linaro.org,,RULES_HIT:2:41:355:379:541:800:960:966:973:982:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1730:1747:1777:1792:2196:2199:2376:2393:2553:2559:2562:2693:2895:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4049:4120:4250:4321:4385:5007:6119:6261:6653:6742:7874:7903:8603:9040:9592:10004:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12683:12895:12986:13161:13229:13894:14096:14394:21080:21433:21444:21611:21627:21990:30054:30070:30090,0,RBL:209.85.128.68:@linaro.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: cars85_5c48cfa3dc92b X-Filterd-Recvd-Size: 9734 Received: from mail-wm1-f68.google.com (mail-wm1-f68.google.com [209.85.128.68]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Thu, 30 Apr 2020 14:40:34 +0000 (UTC) Received: by mail-wm1-f68.google.com with SMTP id v8so7649293wma.0 for ; Thu, 30 Apr 2020 07:40:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ulHx7TItF7RKrAr9OLFM2TAYktCujNY4Q1KC5Wi9Kvs=; b=gfyrsM/IQafYUcT4eaNXYK7NPZJdY8Ut6cYPYLGFYEb14CfaD7rJrcKZml2LyTtlyE hfYCDO5L3G/wvbupqT7sr+9lyB3JfAqD9ZjKNcjKChh+PCpHENWbnWaO+NtR5s/Izrjl M68QjcHeilcomoKCfnNJoV0m6g7w3USGNc4f829BTmBvUmeY+PpPpMTMpwEXPs96538J m9ZbMRPUY1l4ODiSP8FreWnH+BKSWSEbcPW95oMWg4pPK7hTPCOLG8bleFrL2hjYZl25 bavOpR/Qjw8+jhP0FyaLoSXjQeNbuR3gCWwK1hu/JAc2JPImI3NdDjVmO0Qpo/eGVbsS ZRqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ulHx7TItF7RKrAr9OLFM2TAYktCujNY4Q1KC5Wi9Kvs=; b=T23TGs0+9ukX+l6K0NXwZ8IhF+CTJ+RV3k7NXDvm0Zd1os3DyIZ3UqfFQrqQkvK/XQ 7Hc4+xuSuBjTz32u4vW8Opovv5eGXQfaXt3muUoXiA5Fb1Xc2B0txjBMke94eo2JZI6b V/u8YSJi9uVpGSkkz5S4TFxxCg6hnQxhIybGqf1iV0V7wYkOC8hTywb4fpnBpXQ9Qh1U 7EJbtAKwT7PKd5/rW907yjNyevYiweKUwB0Bz2UktpcIjq8tLWkyxr79gZpnXY0oP8Y9 +elR6leUKAh6qANgDeHyrm1ueWr3WpSW2FMQ0DdkAE6HfTtcY75vGozgbLJi/RFLT7fq XaEQ== X-Gm-Message-State: AGi0PubR/6Z1c5el00R85OlFl9FrzPYA+YhVXowcpKYV2Xce9qWn+Kep FYtbCGRYQUrkHWcN0hRXr307Kw== X-Google-Smtp-Source: APiQypJ46AjhsgwjU6+8jK682sr3bY29vZdr/rRH0411nTgNbJxhobRamLvm4JHXPcjf5QkjmKyktw== X-Received: by 2002:a7b:c0c5:: with SMTP id s5mr3308375wmh.134.1588257632747; Thu, 30 Apr 2020 07:40:32 -0700 (PDT) Received: from localhost.localdomain ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id n2sm4153286wrt.33.2020.04.30.07.40.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 07:40:32 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, christian.koenig@amd.com, felix.kuehling@amd.com, zhangfei.gao@linaro.org, jgg@ziepe.ca, xuzaibo@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v6 12/25] iommu/arm-smmu-v3: Seize private ASID Date: Thu, 30 Apr 2020 16:34:11 +0200 Message-Id: <20200430143424.2787566-13-jean-philippe@linaro.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200430143424.2787566-1-jean-philippe@linaro.org> References: <20200430143424.2787566-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The SMMU has a single ASID space, the union of shared and private ASID sets. This means that the SMMU driver competes with the arch allocator for ASIDs. Shared ASIDs are those of Linux processes, allocated by the arch, and contribute in broadcast TLB maintenance. Private ASIDs are allocated by the SMMU driver and used for "classic" map/unmap DMA. They require command-queue TLB invalidations. When we pin down an mm_context and get an ASID that is already in use by the SMMU, it belongs to a private context. We used to simply abort the bind, but this is unfair to users that would be unable to bind a few seemingly random processes. Try to allocate a new private ASID for the context, and make the old ASID shared. Introduce a new lock to prevent races when rewriting context descriptors. Unfortunately it has to be a spinlock since we take it while holding the asid lock, which will be held in non-sleepable context (freeing ASIDs from an RCU callback). Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 83 +++++++++++++++++++++++++++++-------- 1 file changed, 66 insertions(+), 17 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index fb3116045df0f..aad49d565c592 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -730,6 +730,7 @@ struct arm_smmu_option_prop { }; static DEFINE_XARRAY_ALLOC1(asid_xa); +static DEFINE_SPINLOCK(contexts_lock); static struct arm_smmu_option_prop arm_smmu_options[] = { { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" }, @@ -1534,6 +1535,17 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu, } /* Context descriptor manipulation functions */ +static void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) +{ + struct arm_smmu_cmdq_ent cmd = { + .opcode = CMDQ_OP_TLBI_NH_ASID, + .tlbi.asid = asid, + }; + + arm_smmu_cmdq_issue_cmd(smmu, &cmd); + arm_smmu_cmdq_issue_sync(smmu); +} + static void arm_smmu_sync_cd(struct arm_smmu_domain *smmu_domain, int ssid, bool leaf) { @@ -1568,7 +1580,7 @@ static int arm_smmu_alloc_cd_leaf_table(struct arm_smmu_device *smmu, size_t size = CTXDESC_L2_ENTRIES * (CTXDESC_CD_DWORDS << 3); l1_desc->l2ptr = dmam_alloc_coherent(smmu->dev, size, - &l1_desc->l2ptr_dma, GFP_KERNEL); + &l1_desc->l2ptr_dma, GFP_ATOMIC); if (!l1_desc->l2ptr) { dev_warn(smmu->dev, "failed to allocate context descriptor table\n"); @@ -1614,8 +1626,8 @@ static __le64 *arm_smmu_get_cd_ptr(struct arm_smmu_domain *smmu_domain, return l1_desc->l2ptr + idx * CTXDESC_CD_DWORDS; } -static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, - int ssid, struct arm_smmu_ctx_desc *cd) +static int __arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, + int ssid, struct arm_smmu_ctx_desc *cd) { /* * This function handles the following cases: @@ -1691,6 +1703,17 @@ static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, return 0; } +static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, + int ssid, struct arm_smmu_ctx_desc *cd) +{ + int ret; + + spin_lock(&contexts_lock); + ret = __arm_smmu_write_ctx_desc(smmu_domain, ssid, cd); + spin_unlock(&contexts_lock); + return ret; +} + static int arm_smmu_alloc_cd_tables(struct arm_smmu_domain *smmu_domain) { int ret; @@ -1794,9 +1817,18 @@ static bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd) return free; } +/* + * Try to reserve this ASID in the SMMU. If it is in use, try to steal it from + * the private entry. Careful here, we may be modifying the context tables of + * another SMMU! + */ static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid) { + int ret; + u32 new_asid; struct arm_smmu_ctx_desc *cd; + struct arm_smmu_device *smmu; + struct arm_smmu_domain *smmu_domain; cd = xa_load(&asid_xa, asid); if (!cd) @@ -1808,11 +1840,31 @@ static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid) return cd; } + smmu_domain = container_of(cd, struct arm_smmu_domain, s1_cfg.cd); + smmu = smmu_domain->smmu; + + /* + * Race with unmap: TLB invalidations will start targeting the new ASID, + * which isn't assigned yet. We'll do an invalidate-all on the old ASID + * later, so it doesn't matter. + */ + ret = __xa_alloc(&asid_xa, &new_asid, cd, + XA_LIMIT(1, 1 << smmu->asid_bits), GFP_ATOMIC); + if (ret) + return ERR_PTR(-ENOSPC); + cd->asid = new_asid; + /* - * Ouch, ASID is already in use for a private cd. - * TODO: seize it. + * Update ASID and invalidate CD in all associated masters. There will + * be some overlap between use of both ASIDs, until we invalidate the + * TLB. */ - return ERR_PTR(-EEXIST); + arm_smmu_write_ctx_desc(smmu_domain, 0, cd); + + /* Invalidate TLB entries previously associated with that context */ + arm_smmu_tlb_inv_asid(smmu, asid); + + return NULL; } __maybe_unused @@ -2402,15 +2454,6 @@ static void arm_smmu_tlb_inv_context(void *cookie) struct arm_smmu_device *smmu = smmu_domain->smmu; struct arm_smmu_cmdq_ent cmd; - if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { - cmd.opcode = CMDQ_OP_TLBI_NH_ASID; - cmd.tlbi.asid = smmu_domain->s1_cfg.cd.asid; - cmd.tlbi.vmid = 0; - } else { - cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; - cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; - } - /* * NOTE: when io-pgtable is in non-strict mode, we may get here with * PTEs previously cleared by unmaps on the current CPU not yet visible @@ -2418,8 +2461,14 @@ static void arm_smmu_tlb_inv_context(void *cookie) * insertion to guarantee those are observed before the TLBI. Do be * careful, 007. */ - arm_smmu_cmdq_issue_cmd(smmu, &cmd); - arm_smmu_cmdq_issue_sync(smmu); + if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { + arm_smmu_tlb_inv_asid(smmu, smmu_domain->s1_cfg.cd.asid); + } else { + cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; + cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; + arm_smmu_cmdq_issue_cmd(smmu, &cmd); + arm_smmu_cmdq_issue_sync(smmu); + } arm_smmu_atc_inv_domain(smmu_domain, 0, 0, 0); }