From patchwork Thu Jun 18 15:51:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612727 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 929DA92A for ; Thu, 18 Jun 2020 15:55:23 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7032720732 for ; Thu, 18 Jun 2020 15:55:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="QS38QuMs"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="JW/LxFcI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7032720732 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=bepbn9oRNJof/6five8Uleo9E9adAKqVtHIOgGhaAig=; b=QS38QuMsLjfrd8 0xI7AmiYJgJ7G8kiQjJ1thBE8gOXPPMWzx6bg73Exv1yS0+MQm2II2ub1VF9S+6iz4Pqq6P3rS8zz lOyuOeECYwHCDJt7E9S2h8ON5g5eRGKR4bXTIwODzAeVyn7Dd2j3/HZAn4xorLN78XGqGd9c/84gP OjZb28owXnbbW0F4FtxLBwiwJwmBH9/TW/1SH3re3u2WOM32m90KzNEemYrZYGTKhHE2D65xskdCZ GW+Kh2qT7eBcKOn6zCGq3N9jXsfEGSwzHYDiNpuij0rOzUQJhDPIG9+MyapWhntbnHNqFApuihnE5 qfDXDvXEufFnwe6Hxxcg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwsl-0004vH-5a; Thu, 18 Jun 2020 15:55:15 +0000 Received: from mail-ed1-x543.google.com ([2a00:1450:4864:20::543]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwqH-00028h-PA for linux-arm-kernel@lists.infradead.org; Thu, 18 Jun 2020 15:52:49 +0000 Received: by mail-ed1-x543.google.com with SMTP id g1so5256894edv.6 for ; Thu, 18 Jun 2020 08:52:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yTcZ/yCb5ZAy6PQyVCfw2AueTEndBOUdt4lrv7Qejz4=; b=JW/LxFcIwVMI0FUMp2Hp5J9IV4CbwtAZdbhqoRMEX3idf6gsNu+wnpKU+K3PCkEzT8 Z3hEEF3Lg+SzSDaBz8tcAqxQjhy71Hu/Nh7PPuiPBeASjeLBJqrfF5i7bXMkwYa+eigH 8+Y0YMlZi4ZiOEFSBA3y9zy+M2LQWPS8nlsoOCRjNS4B97KfnIBhcC8uZYIvrZkmkCXZ fy4O/KVuDHT8KFgHZkRQkJ1/cOE6bXgfZT9ug04N9mAHyTmklDntEvl1BaNCOX6gnZO9 laYmc4vN5AFc+ATfGfVf5zHp5iPGL+9By9clhNoZNcyQO2BWBFKRiJtia49h75bNu6W8 +pAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yTcZ/yCb5ZAy6PQyVCfw2AueTEndBOUdt4lrv7Qejz4=; b=XWEV/aPySV30+3zVCYOWuGFSHzR+4b1DFE3r5Evh+o7cXD3LCmSsGqxoCh0gixyvAK 3/ntzlyVmAcfUqAf/IvrR5scYo7ZeN/cgh5oKF7EOASKsc4hqHh84/1xguy0H2rZ/qqF MJy6iCS6qzDt0B3FZRuS2B6ojr+xkWQ3gJHi47D7Wvc6PE3ZvkLmGn9EjIHYy1co/qWW GBHMR+ijqt0fAAY5T06Xk2fn6zc5MWHILOZ/Xnk/aYQqoMw5GU29jvnsND2zVCzyITrZ O1n2m30X8PWR8Yx1fsgLSEmlpLUOWo0UGZ3wv/f/3U4SVv6cv1H8KVOnJe8g7Qhp4Ehy 3pWQ== X-Gm-Message-State: AOAM532cFLpnwwKW3AlQA8//sOLKmSYFfP0l6k5w1sgjjHNh4+Ghrvoz XzbLs1gDnJCqOknzuNMitg928A== X-Google-Smtp-Source: ABdhPJyr+YsmKXyyVo8ML8+oCnbztzhK2ZSjk0S597/jP9A5VGpW8hFIgvZu6EuzmCv59bIsyCybEw== X-Received: by 2002:aa7:c908:: with SMTP id b8mr4725695edt.76.1592495559755; Thu, 18 Jun 2020 08:52:39 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:39 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH v8 01/12] mm: Define pasid in mm Date: Thu, 18 Jun 2020 17:51:14 +0200 Message-Id: <20200618155125.1548969-2-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200618_085241_879257_1AB5D96D X-CRM114-Status: UNSURE ( 9.58 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:543 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fenghua.yu@intel.com, jacob.jun.pan@linux.intel.com, catalin.marinas@arm.com, joro@8bytes.org, robin.murphy@arm.com, hch@infradead.org, zhengxiang9@huawei.com, Tony Luck , Christoph Hellwig , Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, will@kernel.org, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org From: Fenghua Yu PASID is shared by all threads in a process. So the logical place to keep track of it is in the "mm". Both ARM and X86 need to use the PASID in the "mm". Suggested-by: Christoph Hellwig Signed-off-by: Fenghua Yu Reviewed-by: Tony Luck --- https://lore.kernel.org/linux-iommu/1592418233-17762-9-git-send-email-fenghua.yu@intel.com/ --- include/linux/mm_types.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 64ede5f150dc5..1ad0e54ebbbaa 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -538,6 +538,10 @@ struct mm_struct { atomic_long_t hugetlb_usage; #endif struct work_struct async_put_work; + +#ifdef CONFIG_IOMMU_SUPPORT + unsigned int pasid; +#endif } __randomize_layout; /* From patchwork Thu Jun 18 15:51:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612729 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1474792A for ; Thu, 18 Jun 2020 15:55:40 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BDB3520732 for ; Thu, 18 Jun 2020 15:55:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Oiur89z1"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="Eds4e2o9" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BDB3520732 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=j5bZwIsLtWbIYeQlqv77Zvvz/FNfy17H6HF3TTpPV1I=; b=Oiur89z1nBszhW TgHsB2q05OQLCNrbnvXilYTC4CZJ4aUIgW4/K3Z8a40tmNgOvIqe8HweTk/h0jU6hrh9Pnc6Eua0E mkj0NQ4tx3lGKOopmVMY3pMSuMypQvsaryHZIAzpJbUHaXOafuKgUW67piHHgRxi1n0EXgTXdTVXP /BFNejxk/wWOxzi1QByu3nhHX3L9x/srFAY2l/7ZXUNoSWDP+1UYpPydTk+D7Vauo5khishNoLiLu RnOA6bQPOyTdDOQFtSWR+kAxCjSJ7JXr47iRHEeYhvXVVAG7dxO9+6s5N06J7ywOCNhH1msWw2rCi HMLWY0TPq4jfdmkSKhUg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwt7-0006fz-5y; Thu, 18 Jun 2020 15:55:37 +0000 Received: from mail-ej1-x643.google.com ([2a00:1450:4864:20::643]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwqJ-00029W-0n for linux-arm-kernel@lists.infradead.org; Thu, 18 Jun 2020 15:52:50 +0000 Received: by mail-ej1-x643.google.com with SMTP id dp18so6926086ejc.8 for ; Thu, 18 Jun 2020 08:52:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gANJ9nXJh01yt9NJy6mJRUbndSE80vIrxk0QteKIq3c=; b=Eds4e2o9/aTlytifgQchMEkJmVJnHukU0ovaep4b99cT1h6zth+zUqp366TT6K7ePX HiZEqi5WIama9YvlIAens9d1oSuRcu30EdLiupH19lT8Hmm2muD9gKIAMgXLTT8oZzjY MoyoHHPnrcr8cOojysoVY47yfdgPaVRYmZmo+khdV4Ej00BOefC8o/w6f/PYTJyS6JMS IE97PCTm2L9rvU65hRt0iESOfgfc1BzLFCJZHQEDWvQTL3BUfPf06ylKZClOTGVvHCHR HxJDIw/+s/Qhsbwm5tathG8DD43vy8bx/z/kb1lxFBK35YaCkkejan7zgAl8K15t3TWZ HQ7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gANJ9nXJh01yt9NJy6mJRUbndSE80vIrxk0QteKIq3c=; b=VY30fSVziyoHNxLbKRWw+9ElPQvHS/da6KN+Bkdekb+8Z82jE0N51TVN14aUSuOTk0 jf/rTcFq7VC610HOzgyfigdTgMynYbWm6dA6i7DWfhhvzD1dxjckrEkOmyKFbON+BBHd qPVOraa4URb4lMsHYUZp3yyl7fwFgyex9Z57aWDohEabXajkyrK8HELg1vy+ed6/1RGh +fhYjXP9n4zwL55Km3zgnZq8fg4tUlmRvn+/DPUBUUNXmm3PG6Z8IppvfcYMB2XgWx7s XiV9hBdSIDtLYosQzcqDJJHMS7Yc/+e/Tp7cQrjxgqtuRYB9QsRqfeZVA8a+XWI6BALR UGFQ== X-Gm-Message-State: AOAM531Ws57k9UZfaCNJwMyU2nj55i/jcNvugj/oNL52lyJ07H0iejJZ Tf8oKjJGPqbJS6ytFdLkZbYOyA== X-Google-Smtp-Source: ABdhPJwyzOwFmlJDohdbHbSJOb408dno2oi+63HZld2Ti01+pRXVJHbN4UECF6xp5+TKBsOTjUOPvw== X-Received: by 2002:a17:906:68c5:: with SMTP id y5mr4450793ejr.436.1592495560894; Thu, 18 Jun 2020 08:52:40 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:40 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH v8 02/12] iommu/ioasid: Add ioasid references Date: Thu, 18 Jun 2020 17:51:15 +0200 Message-Id: <20200618155125.1548969-3-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200618_085243_093078_7CEBBFC0 X-CRM114-Status: GOOD ( 16.98 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:643 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fenghua.yu@intel.com, jacob.jun.pan@linux.intel.com, Jean-Philippe Brucker , catalin.marinas@arm.com, joro@8bytes.org, robin.murphy@arm.com, hch@infradead.org, zhengxiang9@huawei.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, will@kernel.org, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Let IOASID users take references to existing ioasids with ioasid_get(). ioasid_put() drops a reference and only frees the ioasid when its reference number is zero. It returns true if the ioasid was freed. For drivers that don't call ioasid_get(), ioasid_put() is the same as ioasid_free(). Reviewed-by: Lu Baolu Signed-off-by: Jean-Philippe Brucker --- include/linux/ioasid.h | 10 ++++++++-- drivers/iommu/intel/iommu.c | 4 ++-- drivers/iommu/intel/svm.c | 6 +++--- drivers/iommu/ioasid.c | 38 +++++++++++++++++++++++++++++++++---- 4 files changed, 47 insertions(+), 11 deletions(-) diff --git a/include/linux/ioasid.h b/include/linux/ioasid.h index 6f000d7a0ddcd..e9dacd4b9f6bb 100644 --- a/include/linux/ioasid.h +++ b/include/linux/ioasid.h @@ -34,7 +34,8 @@ struct ioasid_allocator_ops { #if IS_ENABLED(CONFIG_IOASID) ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, void *private); -void ioasid_free(ioasid_t ioasid); +void ioasid_get(ioasid_t ioasid); +bool ioasid_put(ioasid_t ioasid); void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, bool (*getter)(void *)); int ioasid_register_allocator(struct ioasid_allocator_ops *allocator); @@ -48,10 +49,15 @@ static inline ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, return INVALID_IOASID; } -static inline void ioasid_free(ioasid_t ioasid) +static inline void ioasid_get(ioasid_t ioasid) { } +static inline bool ioasid_put(ioasid_t ioasid) +{ + return false; +} + static inline void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, bool (*getter)(void *)) { diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 9129663a7406b..a5840b8ef14b7 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -5131,7 +5131,7 @@ static void auxiliary_unlink_device(struct dmar_domain *domain, domain->auxd_refcnt--; if (!domain->auxd_refcnt && domain->default_pasid > 0) - ioasid_free(domain->default_pasid); + ioasid_put(domain->default_pasid); } static int aux_domain_add_dev(struct dmar_domain *domain, @@ -5193,7 +5193,7 @@ static int aux_domain_add_dev(struct dmar_domain *domain, spin_unlock(&iommu->lock); spin_unlock_irqrestore(&device_domain_lock, flags); if (!domain->auxd_refcnt && domain->default_pasid > 0) - ioasid_free(domain->default_pasid); + ioasid_put(domain->default_pasid); return ret; } diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c index 6c87c807a0abb..b078a697c42e8 100644 --- a/drivers/iommu/intel/svm.c +++ b/drivers/iommu/intel/svm.c @@ -546,7 +546,7 @@ intel_svm_bind_mm(struct device *dev, int flags, struct svm_dev_ops *ops, if (mm) { ret = mmu_notifier_register(&svm->notifier, mm); if (ret) { - ioasid_free(svm->pasid); + ioasid_put(svm->pasid); kfree(svm); kfree(sdev); goto out; @@ -564,7 +564,7 @@ intel_svm_bind_mm(struct device *dev, int flags, struct svm_dev_ops *ops, if (ret) { if (mm) mmu_notifier_unregister(&svm->notifier, mm); - ioasid_free(svm->pasid); + ioasid_put(svm->pasid); kfree(svm); kfree(sdev); goto out; @@ -639,7 +639,7 @@ static int intel_svm_unbind_mm(struct device *dev, int pasid) kfree_rcu(sdev, rcu); if (list_empty(&svm->devs)) { - ioasid_free(svm->pasid); + ioasid_put(svm->pasid); if (svm->mm) mmu_notifier_unregister(&svm->notifier, svm->mm); list_del(&svm->list); diff --git a/drivers/iommu/ioasid.c b/drivers/iommu/ioasid.c index 0f8dd377aada3..50ee27bbd04ec 100644 --- a/drivers/iommu/ioasid.c +++ b/drivers/iommu/ioasid.c @@ -2,7 +2,7 @@ /* * I/O Address Space ID allocator. There is one global IOASID space, split into * subsets. Users create a subset with DECLARE_IOASID_SET, then allocate and - * free IOASIDs with ioasid_alloc and ioasid_free. + * free IOASIDs with ioasid_alloc and ioasid_put. */ #include #include @@ -15,6 +15,7 @@ struct ioasid_data { struct ioasid_set *set; void *private; struct rcu_head rcu; + refcount_t refs; }; /* @@ -314,6 +315,7 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, data->set = set; data->private = private; + refcount_set(&data->refs, 1); /* * Custom allocator needs allocator data to perform platform specific @@ -346,11 +348,34 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, EXPORT_SYMBOL_GPL(ioasid_alloc); /** - * ioasid_free - Free an IOASID + * ioasid_get - obtain a reference to the IOASID + */ +void ioasid_get(ioasid_t ioasid) +{ + struct ioasid_data *ioasid_data; + + spin_lock(&ioasid_allocator_lock); + ioasid_data = xa_load(&active_allocator->xa, ioasid); + if (ioasid_data) + refcount_inc(&ioasid_data->refs); + else + WARN_ON(1); + spin_unlock(&ioasid_allocator_lock); +} +EXPORT_SYMBOL_GPL(ioasid_get); + +/** + * ioasid_put - Release a reference to an ioasid * @ioasid: the ID to remove + * + * Put a reference to the IOASID, free it when the number of references drops to + * zero. + * + * Return: %true if the IOASID was freed, %false otherwise. */ -void ioasid_free(ioasid_t ioasid) +bool ioasid_put(ioasid_t ioasid) { + bool free = false; struct ioasid_data *ioasid_data; spin_lock(&ioasid_allocator_lock); @@ -360,6 +385,10 @@ void ioasid_free(ioasid_t ioasid) goto exit_unlock; } + free = refcount_dec_and_test(&ioasid_data->refs); + if (!free) + goto exit_unlock; + active_allocator->ops->free(ioasid, active_allocator->ops->pdata); /* Custom allocator needs additional steps to free the xa element */ if (active_allocator->flags & IOASID_ALLOCATOR_CUSTOM) { @@ -369,8 +398,9 @@ void ioasid_free(ioasid_t ioasid) exit_unlock: spin_unlock(&ioasid_allocator_lock); + return free; } -EXPORT_SYMBOL_GPL(ioasid_free); +EXPORT_SYMBOL_GPL(ioasid_put); /** * ioasid_find - Find IOASID data From patchwork Thu Jun 18 15:51:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612733 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 686C214E3 for ; Thu, 18 Jun 2020 15:56:21 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 23E1D20732 for ; Thu, 18 Jun 2020 15:56:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="COMx50Xo"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="sROLygHw" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 23E1D20732 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=T1cjozv4sE58TZ24jIUA/7aXeB9dCimnhpFhEZmXqsc=; b=COMx50XoefLo4g xJsVo0sXtgdPlVjsfwvMceAIHPZau3JTQWqE3C4YIMxd1knil/cNv+k14blZBUMy+nSXlWUyRV2VV +gvJIy27r/boRWvlGoGkCFI5tXwt77dXccDQ0YldvEA+s7JXkmUNKyNCl6deBcnsNEwT44VdsXHI0 JxEhtJIDB4FRewKlFtUhrIM4ILSWzzk0RA0Btbg1jj/qfj29LqrmfCXF2kZkQNuKtjLsAubSRy2y+ Q1E4t1hAj1S0Nzv+Z/V7pYkOxc6kMLrO4KhZbM2S5wKlWF80F3tMeVRhgTrN+nhhOZPlhMtSbQvt4 Cm5uZNd5peWQgEu+TUeQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwtn-0007Gv-47; Thu, 18 Jun 2020 15:56:19 +0000 Received: from mail-ej1-x643.google.com ([2a00:1450:4864:20::643]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwqK-0002AV-30 for linux-arm-kernel@lists.infradead.org; Thu, 18 Jun 2020 15:52:52 +0000 Received: by mail-ej1-x643.google.com with SMTP id n24so6972561ejd.0 for ; Thu, 18 Jun 2020 08:52:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RIXfyWsdCSmhkCPoxdMwgb5HoQjrgnKhMBMykTzupqQ=; b=sROLygHw732kSPKgqm1mAJj8riM2WWL2GR/3RO14UTzqzDo+ON4NJHm/mIDeC2LCl4 +RP9frVhkxIRCUhwMIcnsdOC7S0WnA2G6KfJuTe/Je028x/UMJrQov9nesMCYthhJ/lP pJ1rzJVPXLe1HLn7VCsYhkZNAAQANsfO/2H3NkbXt6L2YJO5UoWuzfUlEG0dlXco6I1Q 0c6m2NjRQFUEJEYSQQaKlU23EX57kMW88YOG6S0iyWbgDwJGGwXBOOO7jnrsWYGC/4O9 KIfikh5Y7YxEg+ObMIpjzN59DZNakYHrfmJokgjdphmsjh21ggon+8JPNKuNaBofykdO FZOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RIXfyWsdCSmhkCPoxdMwgb5HoQjrgnKhMBMykTzupqQ=; b=VpLYwfhS7PT3xEgAGnx1FFOtLmKXfvLLVfanxuKcD2rtikKBE3KBrxtX59vStXCK3b CtM0TBA74eQIsStY+NF9s+xHtQZlr+h5cz334oB9lV9L2+HvFR9KaJoOd7AkBjeYQWVL hp4FYviz/BlgZpuuQ+92F9MEK3EJzPGM4+J0pMkNRq2wCyrJN+mQDDmRflcEhMaZ9pGr zuyIJuTuUS2dCYi1aLGSAFuebgSIYz5XeSprFfJp3n3B346LywxGo6zLtyr/yANo305v 90h1AbeFehNyZaBEjIFfjaK1GL/pWJpMJPAIDGBMcFuvheSdKYye1+EpFEnSf8uhacsY iGow== X-Gm-Message-State: AOAM533MgSB6TmYCcaR4YHK7p7AQhWybTfuJveZBnfLIUiJLGXdIN/4s tmNLB7AlATf3GCJbgEpvDCyCbw== X-Google-Smtp-Source: ABdhPJxBNDhV30ThfSe9ibwBo1JFl7QA4iTqkJCsARWmrputOq3YDJfzAm2LRPiQdaBFKF1afEsQww== X-Received: by 2002:a17:906:1f4f:: with SMTP id d15mr4481967ejk.206.1592495562342; Thu, 18 Jun 2020 08:52:42 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:41 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH v8 03/12] iommu/sva: Add PASID helpers Date: Thu, 18 Jun 2020 17:51:16 +0200 Message-Id: <20200618155125.1548969-4-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200618_085244_270147_5A858B87 X-CRM114-Status: GOOD ( 17.65 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:643 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fenghua.yu@intel.com, jacob.jun.pan@linux.intel.com, Jean-Philippe Brucker , catalin.marinas@arm.com, joro@8bytes.org, robin.murphy@arm.com, hch@infradead.org, zhengxiang9@huawei.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, will@kernel.org, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Let IOMMU drivers allocate a single PASID per mm. Store the mm in the IOASID set to allow refcounting and searching mm by PASID, when handling an I/O page fault. Signed-off-by: Jean-Philippe Brucker Reviewed-by: Lu Baolu --- v7->v8: rename to IOMMU_SVA_LIB (Lu Baolu) --- drivers/iommu/Kconfig | 5 +++ drivers/iommu/Makefile | 1 + drivers/iommu/iommu-sva-lib.h | 15 +++++++ drivers/iommu/iommu-sva-lib.c | 85 +++++++++++++++++++++++++++++++++++ 4 files changed, 106 insertions(+) create mode 100644 drivers/iommu/iommu-sva-lib.h create mode 100644 drivers/iommu/iommu-sva-lib.c diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index b510f67dfa499..74a10e7a8d082 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -102,6 +102,11 @@ config IOMMU_DMA select IRQ_MSI_IOMMU select NEED_SG_DMA_LENGTH +# Shared Virtual Addressing library +config IOMMU_SVA_LIB + bool + select IOASID + config FSL_PAMU bool "Freescale IOMMU support" depends on PCI diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile index 342190196dfb0..0fe5a7f9bc69c 100644 --- a/drivers/iommu/Makefile +++ b/drivers/iommu/Makefile @@ -38,3 +38,4 @@ obj-$(CONFIG_S390_IOMMU) += s390-iommu.o obj-$(CONFIG_QCOM_IOMMU) += qcom_iommu.o obj-$(CONFIG_HYPERV_IOMMU) += hyperv-iommu.o obj-$(CONFIG_VIRTIO_IOMMU) += virtio-iommu.o +obj-$(CONFIG_IOMMU_SVA_LIB) += iommu-sva-lib.o diff --git a/drivers/iommu/iommu-sva-lib.h b/drivers/iommu/iommu-sva-lib.h new file mode 100644 index 0000000000000..b40990aef3fde --- /dev/null +++ b/drivers/iommu/iommu-sva-lib.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * SVA library for IOMMU drivers + */ +#ifndef _IOMMU_SVA_LIB_H +#define _IOMMU_SVA_LIB_H + +#include +#include + +int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max); +void iommu_sva_free_pasid(struct mm_struct *mm); +struct mm_struct *iommu_sva_find(ioasid_t pasid); + +#endif /* _IOMMU_SVA_LIB_H */ diff --git a/drivers/iommu/iommu-sva-lib.c b/drivers/iommu/iommu-sva-lib.c new file mode 100644 index 0000000000000..db7e6c104d6b0 --- /dev/null +++ b/drivers/iommu/iommu-sva-lib.c @@ -0,0 +1,85 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Helpers for IOMMU drivers implementing SVA + */ +#include +#include + +#include "iommu-sva-lib.h" + +static DEFINE_MUTEX(iommu_sva_lock); +static DECLARE_IOASID_SET(iommu_sva_pasid); + +/** + * iommu_sva_alloc_pasid - Allocate a PASID for the mm + * @mm: the mm + * @min: minimum PASID value (inclusive) + * @max: maximum PASID value (inclusive) + * + * Try to allocate a PASID for this mm, or take a reference to the existing one + * provided it fits within the [min, max] range. On success the PASID is + * available in mm->pasid, and must be released with iommu_sva_free_pasid(). + * + * Returns 0 on success and < 0 on error. + */ +int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max) +{ + int ret = 0; + ioasid_t pasid; + + if (min == INVALID_IOASID || max == INVALID_IOASID || + min == 0 || max < min) + return -EINVAL; + + mutex_lock(&iommu_sva_lock); + if (mm->pasid) { + if (mm->pasid >= min && mm->pasid <= max) + ioasid_get(mm->pasid); + else + ret = -EOVERFLOW; + } else { + pasid = ioasid_alloc(&iommu_sva_pasid, min, max, mm); + if (pasid == INVALID_IOASID) + ret = -ENOMEM; + else + mm->pasid = pasid; + } + mutex_unlock(&iommu_sva_lock); + return ret; +} +EXPORT_SYMBOL_GPL(iommu_sva_alloc_pasid); + +/** + * iommu_sva_free_pasid - Release the mm's PASID + * @mm: the mm. + * + * Drop one reference to a PASID allocated with iommu_sva_alloc_pasid() + */ +void iommu_sva_free_pasid(struct mm_struct *mm) +{ + mutex_lock(&iommu_sva_lock); + if (ioasid_put(mm->pasid)) + mm->pasid = 0; + mutex_unlock(&iommu_sva_lock); +} +EXPORT_SYMBOL_GPL(iommu_sva_free_pasid); + +/* ioasid wants a void * argument */ +static bool __mmget_not_zero(void *mm) +{ + return mmget_not_zero(mm); +} + +/** + * iommu_sva_find() - Find mm associated to the given PASID + * @pasid: Process Address Space ID assigned to the mm + * + * On success a reference to the mm is taken, and must be released with mmput(). + * + * Returns the mm corresponding to this PASID, or an error if not found. + */ +struct mm_struct *iommu_sva_find(ioasid_t pasid) +{ + return ioasid_find(&iommu_sva_pasid, pasid, __mmget_not_zero); +} +EXPORT_SYMBOL_GPL(iommu_sva_find); From patchwork Thu Jun 18 15:51:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612731 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 723C392A for ; Thu, 18 Jun 2020 15:56:07 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1B2A620732 for ; Thu, 18 Jun 2020 15:56:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="MWSY7+yg"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="qKis8+0z" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1B2A620732 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=vdDT46fugeRA8nomsH7qlqtS64AGNnHL/ZliB4nyG6o=; b=MWSY7+ygfVKPbe GIadq0PD/nm1yEQaVmgFIAWWr1vw2T8Kf+K8OEhA6ep2bt2e1/aGmo8q8bk+BAWjLDlTnmEmEkLtY VrCjkkW5z7VRAOj2fg0QZmZ9KKmaikSjTVX8keUePM/G5HcgZ21b+34SiiXyp1pyf441ncStRQ1hr m6fZVju8MaEECGiH1I7qrV6OrE+XwZmy3VlSuRWl3Q4Ao/20GBg9Hw6IltEZRykoBx8b3NBMYDWa7 YwAlKRGn5S+NwbrDkm4FJAKpBdmTuxRmIHRwESswvGQIfZwxTt8zL1TO2VSg+ir7+55ZdcYCX+LA1 l3qPRZrsCrdXcl7eWvXQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwtU-00070s-BO; Thu, 18 Jun 2020 15:56:00 +0000 Received: from mail-ej1-x644.google.com ([2a00:1450:4864:20::644]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwqL-0002Bh-Da for linux-arm-kernel@lists.infradead.org; Thu, 18 Jun 2020 15:52:54 +0000 Received: by mail-ej1-x644.google.com with SMTP id dp18so6926228ejc.8 for ; Thu, 18 Jun 2020 08:52:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tNsKKlZrMS7qavcUPxigtHJpxuII47BMB2PD7LPqYhM=; b=qKis8+0zz9FpJJzO2PcVBzxN7h/KVSGahqjTo3rC/34bBhNsoWpi6DqOiFkJFrgcXw PWN4tPDht0f3kfRBr6KB4nLmSlDlX+REPkX1b57yDdX2jzIFFYyGSZK2/cg3NmCt7Cr2 yPL7/xQjFWTj9g2yUHgLHO3sPtQu6A2NoTG9flk5EjnjRekGXMA4Xp8C3ClPIkQoKt3a 8eSPbkQPV1DxwCylidi3u8x7X+PSNNwhd2TVUoCUhvNrLmS4kwWv70lYMXihoOmXlX+6 erobYWDFdNqg3HdBh17QtQaswTrMhad1LxNT5/v8Uph4G7Olvb71xPiHyDHjvF68bx2X jvKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tNsKKlZrMS7qavcUPxigtHJpxuII47BMB2PD7LPqYhM=; b=cl6V/vN8N3hd8+bZT+Mz6CT625GoldTRDiuu70lYUkohpWH/taFn5b/FKHYy7zdgTu cFEqLqPSrHGACTP1WQGcjKLjUmWsOR5BflmzDrfnagPuiKCNSAUrT/NB6nS2jKK2l3hF T6MU0nA1r1EmWiNbR9VN4DNfU79/NAQxw9pyy2ORxMWT+OxRZVD8Ak+8WDzoDCEotXLa PjRt+xH3TUPTlIAl2tD3fUoiNM8TrZTmUZkCreGoZ9Wo6gSAnoIpIH4XRMywY/1VSyQi PlJClGxRj/c3NuT5ZW7jW5rvCn/KNGDt8S6qSmglwZhaVndvJZl90OnGch9LTOtwH0pA dN4g== X-Gm-Message-State: AOAM533EdMTAgYPOvOYPEalePEKbBOyCu3b6aPetmiJbfCGireVYrWmC mUIPR0nlwCNvrHOq+8ItKbMBXw== X-Google-Smtp-Source: ABdhPJwMTz6VrTL+YpR+VUTsW2WpdHUrvHPd0nxeIDXU3aORNj7p14GXmHBByZRwRDr/GjFSjYh2ZQ== X-Received: by 2002:a17:906:4a03:: with SMTP id w3mr4472232eju.154.1592495563380; Thu, 18 Jun 2020 08:52:43 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:42 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH v8 04/12] arm64: mm: Pin down ASIDs for sharing mm with devices Date: Thu, 18 Jun 2020 17:51:17 +0200 Message-Id: <20200618155125.1548969-5-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200618_085245_473077_042AAE53 X-CRM114-Status: GOOD ( 25.25 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:644 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fenghua.yu@intel.com, jacob.jun.pan@linux.intel.com, Jean-Philippe Brucker , catalin.marinas@arm.com, joro@8bytes.org, robin.murphy@arm.com, hch@infradead.org, zhengxiang9@huawei.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, will@kernel.org, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org To enable address space sharing with the IOMMU, introduce mm_context_get() and mm_context_put(), that pin down a context and ensure that it will keep its ASID after a rollover. Export the symbols to let the modular SMMUv3 driver use them. Pinning is necessary because a device constantly needs a valid ASID, unlike tasks that only require one when running. Without pinning, we would need to notify the IOMMU when we're about to use a new ASID for a task, and it would get complicated when a new task is assigned a shared ASID. Consider the following scenario with no ASID pinned: 1. Task t1 is running on CPUx with shared ASID (gen=1, asid=1) 2. Task t2 is scheduled on CPUx, gets ASID (1, 2) 3. Task tn is scheduled on CPUy, a rollover occurs, tn gets ASID (2, 1) We would now have to immediately generate a new ASID for t1, notify the IOMMU, and finally enable task tn. We are holding the lock during all that time, since we can't afford having another CPU trigger a rollover. The IOMMU issues invalidation commands that can take tens of milliseconds. It gets needlessly complicated. All we wanted to do was schedule task tn, that has no business with the IOMMU. By letting the IOMMU pin tasks when needed, we avoid stalling the slow path, and let the pinning fail when we're out of shareable ASIDs. After a rollover, the allocator expects at least one ASID to be available in addition to the reserved ones (one per CPU). So (NR_ASIDS - NR_CPUS - 1) is the maximum number of ASIDs that can be shared with the IOMMU. Signed-off-by: Jean-Philippe Brucker --- arch/arm64/include/asm/mmu.h | 1 + arch/arm64/include/asm/mmu_context.h | 11 +++- arch/arm64/mm/context.c | 95 +++++++++++++++++++++++++++- 3 files changed, 104 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 68140fdd89d6b..bbdd291e31d59 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -19,6 +19,7 @@ typedef struct { atomic64_t id; + unsigned long pinned; void *vdso; unsigned long flags; } mm_context_t; diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index b0bd9b55594c5..7b0e0f6cb7e87 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -177,7 +177,13 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp) #define destroy_context(mm) do { } while(0) void check_and_switch_context(struct mm_struct *mm, unsigned int cpu); -#define init_new_context(tsk,mm) ({ atomic64_set(&(mm)->context.id, 0); 0; }) +static inline int +init_new_context(struct task_struct *tsk, struct mm_struct *mm) +{ + atomic64_set(&mm->context.id, 0); + mm->context.pinned = 0; + return 0; +} #ifdef CONFIG_ARM64_SW_TTBR0_PAN static inline void update_saved_ttbr0(struct task_struct *tsk, @@ -250,6 +256,9 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next, void verify_cpu_asid_bits(void); void post_ttbr_update_workaround(void); +unsigned long mm_context_get(struct mm_struct *mm); +void mm_context_put(struct mm_struct *mm); + #endif /* !__ASSEMBLY__ */ #endif /* !__ASM_MMU_CONTEXT_H */ diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index d702d60e64dab..d0ddd413f5645 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -27,6 +27,10 @@ static DEFINE_PER_CPU(atomic64_t, active_asids); static DEFINE_PER_CPU(u64, reserved_asids); static cpumask_t tlb_flush_pending; +static unsigned long max_pinned_asids; +static unsigned long nr_pinned_asids; +static unsigned long *pinned_asid_map; + #define ASID_MASK (~GENMASK(asid_bits - 1, 0)) #define ASID_FIRST_VERSION (1UL << asid_bits) @@ -74,6 +78,9 @@ void verify_cpu_asid_bits(void) static void set_kpti_asid_bits(void) { + unsigned int k; + u8 *dst = (u8 *)asid_map; + u8 *src = (u8 *)pinned_asid_map; unsigned int len = BITS_TO_LONGS(NUM_USER_ASIDS) * sizeof(unsigned long); /* * In case of KPTI kernel/user ASIDs are allocated in @@ -81,7 +88,8 @@ static void set_kpti_asid_bits(void) * is set, then the ASID will map only userspace. Thus * mark even as reserved for kernel. */ - memset(asid_map, 0xaa, len); + for (k = 0; k < len; k++) + dst[k] = src[k] | 0xaa; } static void set_reserved_asid_bits(void) @@ -89,7 +97,7 @@ static void set_reserved_asid_bits(void) if (arm64_kernel_unmapped_at_el0()) set_kpti_asid_bits(); else - bitmap_clear(asid_map, 0, NUM_USER_ASIDS); + bitmap_copy(asid_map, pinned_asid_map, NUM_USER_ASIDS); } #define asid_gen_match(asid) \ @@ -165,6 +173,14 @@ static u64 new_context(struct mm_struct *mm) if (check_update_reserved_asid(asid, newasid)) return newasid; + /* + * If it is pinned, we can keep using it. Note that reserved + * takes priority, because even if it is also pinned, we need to + * update the generation into the reserved_asids. + */ + if (mm->context.pinned) + return newasid; + /* * We had a valid ASID in a previous life, so try to re-use * it if possible. @@ -254,6 +270,68 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) cpu_switch_mm(mm->pgd, mm); } +unsigned long mm_context_get(struct mm_struct *mm) +{ + unsigned long flags; + u64 asid; + + raw_spin_lock_irqsave(&cpu_asid_lock, flags); + + asid = atomic64_read(&mm->context.id); + + if (mm->context.pinned) { + mm->context.pinned++; + asid &= ~ASID_MASK; + goto out_unlock; + } + + if (nr_pinned_asids >= max_pinned_asids) { + asid = 0; + goto out_unlock; + } + + if (!asid_gen_match(asid)) { + /* + * We went through one or more rollover since that ASID was + * used. Ensure that it is still valid, or generate a new one. + */ + asid = new_context(mm); + atomic64_set(&mm->context.id, asid); + } + + asid &= ~ASID_MASK; + + nr_pinned_asids++; + __set_bit(asid2idx(asid), pinned_asid_map); + mm->context.pinned++; + +out_unlock: + raw_spin_unlock_irqrestore(&cpu_asid_lock, flags); + + /* Set the equivalent of USER_ASID_BIT */ + if (asid && IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0)) + asid |= 1; + + return asid; +} +EXPORT_SYMBOL_GPL(mm_context_get); + +void mm_context_put(struct mm_struct *mm) +{ + unsigned long flags; + u64 asid = atomic64_read(&mm->context.id) & ~ASID_MASK; + + raw_spin_lock_irqsave(&cpu_asid_lock, flags); + + if (--mm->context.pinned == 0) { + __clear_bit(asid2idx(asid), pinned_asid_map); + nr_pinned_asids--; + } + + raw_spin_unlock_irqrestore(&cpu_asid_lock, flags); +} +EXPORT_SYMBOL_GPL(mm_context_put); + /* Errata workaround post TTBRx_EL1 update. */ asmlinkage void post_ttbr_update_workaround(void) { @@ -303,6 +381,13 @@ static int asids_update_limit(void) WARN_ON(num_available_asids - 1 <= num_possible_cpus()); pr_info("ASID allocator initialised with %lu entries\n", num_available_asids); + + /* + * We assume that an ASID is always available after a rollover. This + * means that even if all CPUs have a reserved ASID, there still is at + * least one slot available in the asid map. + */ + max_pinned_asids = num_available_asids - num_possible_cpus() - 2; return 0; } arch_initcall(asids_update_limit); @@ -317,6 +402,12 @@ static int asids_init(void) panic("Failed to allocate bitmap for %lu ASIDs\n", NUM_USER_ASIDS); + pinned_asid_map = kcalloc(BITS_TO_LONGS(NUM_USER_ASIDS), + sizeof(*pinned_asid_map), GFP_KERNEL); + if (!pinned_asid_map) + panic("Failed to allocate pinned ASID bitmap\n"); + nr_pinned_asids = 0; + /* * We cannot call set_reserved_asid_bits() here because CPU * caps are not finalized yet, so it is safer to assume KPTI From patchwork Thu Jun 18 15:51:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612735 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C4BA314E3 for ; Thu, 18 Jun 2020 15:56:48 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A250920732 for ; Thu, 18 Jun 2020 15:56:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="U2lEI/Rm"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="h0r3TSsb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A250920732 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=gEoWNIbt8OokOqKuvHDlCdRc0zPu0QXmQWNJRMjpB3U=; b=U2lEI/RmhXLICQ Ztw1up/yA+xUSJZ7iYjQDZJiYteXEQlbPLbdM06+qeg0BmOOy66TaQROZki0KPlX9mKJ8UoqOwepd ryyemXQbyfCgr0atx8x7Ly7bs/ToRGYW9LqoQnVXS4Era42MQnsel9lYUCsLEau5pXR66/tlKFUM8 ianvtynlErMddDGNtQY3e/mxcETIhoWSZnhA6q3slmrfIOO4zyeoqjXB8j3B+ONOaWXynXxXhnMQ1 sWaYfmkgItyF+3UcQ3vyCKhxZVzI6hgWSEaTrCjGrC6BQImwYdP7FNjc0N2BKTsP3zGR4RvM7Dl1d 1JfwnCv7088YTkHRyPJw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwu8-0007bV-IC; Thu, 18 Jun 2020 15:56:40 +0000 Received: from mail-ej1-x641.google.com ([2a00:1450:4864:20::641]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwqL-0002Ch-J8 for linux-arm-kernel@lists.infradead.org; Thu, 18 Jun 2020 15:52:55 +0000 Received: by mail-ej1-x641.google.com with SMTP id p20so6897034ejd.13 for ; Thu, 18 Jun 2020 08:52:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FY0aPXr4T1XKvh6YqCT/xGUuNfXzsIgkFE1SNeWNDTU=; b=h0r3TSsbVyAd/GgpK2IGqK86/znVI1rJ0Spx39sObm1jycjjG19VLycMXWz+PqwAN1 oJI824ldExIo5cDBCcj0Ykma1Yj12YE9UxnQUpmC2qwfvxAhm44lkQ/X5pRfLXPtH9Xn 58s1XcNZnylhoa5mj1clF3V/tN61SIwOU5PuOJz7lf44dVjhinD4zYC005kKGHt8NHyB pOmYn3TiDbLE9pcftz7/UUy3qwXjdz10rktsWo+MzTpkvFAVrA3BQxS77yG/SRKHV3bN eE4pDvlRXb9dT7bdEDJdIHi9I5KBg2fxj0Ez+BMUb+E08Ehg0Oi7tTdLqUqvYvMJap3t WJPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FY0aPXr4T1XKvh6YqCT/xGUuNfXzsIgkFE1SNeWNDTU=; b=RF0VekTZyDnG8nZk6QxsRNyTt5W4CdItIHcZx3fmiAEOIbbr/Ynxq7dGQv0Ga/WeHi 7QXJ0B0jCDUEWaQKhrl9ffDZFSbTV4kcan5nytAvzGdB25ZvBg+vs6U+hwdXRdaQLFEJ 6IsjCxlCwI/zLelOXFTYDojVcP054XsD3aGTJ/IAzviuNhlyHEisfH8iC4bO/Q1mGoWv MRdTIApwcxCYZ+Xcz9R/rAwfJaW9lTXSUQB4lkBP/zlDgUzFLcHFObAPIgWQWVK98dFV TM1IEyvWNKA6fJlkbT3MOFHIcGxR9m7jP/PhxmGleWpRaiDn1XQeL62NhHoUnABetME9 ms0A== X-Gm-Message-State: AOAM531WuSU1WebO4uOgCj3wXvtA+GrKdfyVDgMlGa1COWcu3+Qh0KTf Fv7pAtQiFuPAZCRPULYvXftThlspIkblUQ== X-Google-Smtp-Source: ABdhPJxPqR2tMmpRqJZE3NxKQ3H6rUsjM+EqJN0+OqhoA1wLKcbML/g//NRUZvq8y8ascWv+p439Rg== X-Received: by 2002:a17:906:3e0c:: with SMTP id k12mr4362788eji.441.1592495564354; Thu, 18 Jun 2020 08:52:44 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:43 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH v8 05/12] iommu/io-pgtable-arm: Move some definitions to a header Date: Thu, 18 Jun 2020 17:51:18 +0200 Message-Id: <20200618155125.1548969-6-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200618_085245_672975_D6973104 X-CRM114-Status: GOOD ( 12.45 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:641 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fenghua.yu@intel.com, jacob.jun.pan@linux.intel.com, Jean-Philippe Brucker , catalin.marinas@arm.com, joro@8bytes.org, robin.murphy@arm.com, hch@infradead.org, zhengxiang9@huawei.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, will@kernel.org, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Extract some of the most generic TCR defines, so they can be reused by the page table sharing code. Acked-by: Will Deacon Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/io-pgtable-arm.h | 30 ++++++++++++++++++++++++++++++ drivers/iommu/io-pgtable-arm.c | 27 ++------------------------- MAINTAINERS | 3 +-- 3 files changed, 33 insertions(+), 27 deletions(-) create mode 100644 drivers/iommu/io-pgtable-arm.h diff --git a/drivers/iommu/io-pgtable-arm.h b/drivers/iommu/io-pgtable-arm.h new file mode 100644 index 0000000000000..ba7cfdf7afa03 --- /dev/null +++ b/drivers/iommu/io-pgtable-arm.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef IO_PGTABLE_ARM_H_ +#define IO_PGTABLE_ARM_H_ + +#define ARM_LPAE_TCR_TG0_4K 0 +#define ARM_LPAE_TCR_TG0_64K 1 +#define ARM_LPAE_TCR_TG0_16K 2 + +#define ARM_LPAE_TCR_TG1_16K 1 +#define ARM_LPAE_TCR_TG1_4K 2 +#define ARM_LPAE_TCR_TG1_64K 3 + +#define ARM_LPAE_TCR_SH_NS 0 +#define ARM_LPAE_TCR_SH_OS 2 +#define ARM_LPAE_TCR_SH_IS 3 + +#define ARM_LPAE_TCR_RGN_NC 0 +#define ARM_LPAE_TCR_RGN_WBWA 1 +#define ARM_LPAE_TCR_RGN_WT 2 +#define ARM_LPAE_TCR_RGN_WB 3 + +#define ARM_LPAE_TCR_PS_32_BIT 0x0ULL +#define ARM_LPAE_TCR_PS_36_BIT 0x1ULL +#define ARM_LPAE_TCR_PS_40_BIT 0x2ULL +#define ARM_LPAE_TCR_PS_42_BIT 0x3ULL +#define ARM_LPAE_TCR_PS_44_BIT 0x4ULL +#define ARM_LPAE_TCR_PS_48_BIT 0x5ULL +#define ARM_LPAE_TCR_PS_52_BIT 0x6ULL + +#endif /* IO_PGTABLE_ARM_H_ */ diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index 04fbd4bf0ff9f..f71a2eade04ab 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -20,6 +20,8 @@ #include +#include "io-pgtable-arm.h" + #define ARM_LPAE_MAX_ADDR_BITS 52 #define ARM_LPAE_S2_MAX_CONCAT_PAGES 16 #define ARM_LPAE_MAX_LEVELS 4 @@ -100,23 +102,6 @@ #define ARM_LPAE_PTE_MEMATTR_DEV (((arm_lpae_iopte)0x1) << 2) /* Register bits */ -#define ARM_LPAE_TCR_TG0_4K 0 -#define ARM_LPAE_TCR_TG0_64K 1 -#define ARM_LPAE_TCR_TG0_16K 2 - -#define ARM_LPAE_TCR_TG1_16K 1 -#define ARM_LPAE_TCR_TG1_4K 2 -#define ARM_LPAE_TCR_TG1_64K 3 - -#define ARM_LPAE_TCR_SH_NS 0 -#define ARM_LPAE_TCR_SH_OS 2 -#define ARM_LPAE_TCR_SH_IS 3 - -#define ARM_LPAE_TCR_RGN_NC 0 -#define ARM_LPAE_TCR_RGN_WBWA 1 -#define ARM_LPAE_TCR_RGN_WT 2 -#define ARM_LPAE_TCR_RGN_WB 3 - #define ARM_LPAE_VTCR_SL0_MASK 0x3 #define ARM_LPAE_TCR_T0SZ_SHIFT 0 @@ -124,14 +109,6 @@ #define ARM_LPAE_VTCR_PS_SHIFT 16 #define ARM_LPAE_VTCR_PS_MASK 0x7 -#define ARM_LPAE_TCR_PS_32_BIT 0x0ULL -#define ARM_LPAE_TCR_PS_36_BIT 0x1ULL -#define ARM_LPAE_TCR_PS_40_BIT 0x2ULL -#define ARM_LPAE_TCR_PS_42_BIT 0x3ULL -#define ARM_LPAE_TCR_PS_44_BIT 0x4ULL -#define ARM_LPAE_TCR_PS_48_BIT 0x5ULL -#define ARM_LPAE_TCR_PS_52_BIT 0x6ULL - #define ARM_LPAE_MAIR_ATTR_SHIFT(n) ((n) << 3) #define ARM_LPAE_MAIR_ATTR_MASK 0xff #define ARM_LPAE_MAIR_ATTR_DEVICE 0x04 diff --git a/MAINTAINERS b/MAINTAINERS index 68f21d46614c4..2e3220273e4d5 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1499,8 +1499,7 @@ L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) S: Maintained F: Documentation/devicetree/bindings/iommu/arm,smmu* F: drivers/iommu/arm-smmu* -F: drivers/iommu/io-pgtable-arm-v7s.c -F: drivers/iommu/io-pgtable-arm.c +F: drivers/iommu/io-pgtable-arm* ARM SUB-ARCHITECTURES L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) From patchwork Thu Jun 18 15:51:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612739 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 32CDC14E3 for ; Thu, 18 Jun 2020 15:56:59 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 104012075E for ; Thu, 18 Jun 2020 15:56:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="pJKwEQB1"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="CXa/IxLq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 104012075E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=RRIalcTUnXrCJJkDxJkdHbExJUa5gfQkz/aJ2ji/JgM=; b=pJKwEQB1Ifi1ay PMlSgboThLkceB2QskP5qBlQdRvk8nZgZHJFz3DvzNHWQyv5p1W6kZD/shN4diDibSSk0S9oM6Gwe UAkJgaTaTAiUurA+/Wuv1rqvqGgmZ9KNXScMvKVA+wj+w66sxoeFhzBQbBx7FgGgRCmsGUhzovNdg +mT8TnUJgsD1mAQSikBtBQxlB/rHsBOLph8EiP6Q7kSHcW2//HwuDea+5OCc7Rr1U0jCUvT8nMaBu 3cQHcmhDOS1+R8LXKgKUkhTmoS6XSXg2ohmYfiAEC/9wHTcnDFMSw9TIqt1EhjAMTOV8T84L9N3/F A0XQ1QKwSR9msN7LaFng==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwuP-0007zW-O1; Thu, 18 Jun 2020 15:56:57 +0000 Received: from mail-ej1-x644.google.com ([2a00:1450:4864:20::644]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwqM-0002Db-K4 for linux-arm-kernel@lists.infradead.org; Thu, 18 Jun 2020 15:52:55 +0000 Received: by mail-ej1-x644.google.com with SMTP id y13so6959914eju.2 for ; Thu, 18 Jun 2020 08:52:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OszwHFuvAODAZ4JcDGk8vTdgEsm3AXU3gmIMXDwpr1E=; b=CXa/IxLq2sAhhWXEwadxVdBTNRXCjnNRJ86QHDU2Mv2XE4055xNa1KydScB6wK21ZB cXy76zowZ7DH6oYph6Pl7OuFVuh4YQj0mvtW0MxteQ4UcjzMXY+xljI/4UnbdNVn8beh plIgdS+Z0wk8kS8Nt5E8eD6lwtdFgeDqCh4xStJzlWi7O9hNV7KqxuDuPglptEYvakjg 4gRgbFs6be6neKEhVJpZF7i5q6uxLLCAUoZE3762HcKHIu0WyhDcJaTICqaeIgkR/5so ELvNZAlx+cNLKnbAJ14SIMrBO5wUtNh1LjFnDsET/fpJA6CuAxpepH0+ZH331HeSTxGC PuMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OszwHFuvAODAZ4JcDGk8vTdgEsm3AXU3gmIMXDwpr1E=; b=D4fmLWGfW2AqHME/oFG41U8hPMxp46q9d8meEiL/v9ymvj9iz/0R2l18lkAK4b0678 1WjtMNzTTIQdIuUS0kP9UJj09Wyet755lZvYPcHADqKnTiVjVv4LRYsTdffgfESJIWvP RmnRx6kj82rmmYdlrsSrzAlAE6jb2Zc357QL2S4C3Kd+RnEMIOgw/986APIHr6NRr2II CBb6aghijaqpbdb8AAJR1gfzkcKBjBsbRNq5w4OmyRvPssAQovj5eDItxe0J1YEmnmw9 up4ak/8IrwwbePUGQgsHDH4G/eFTtvXchrtw/GKntItwMHHfXM8vTWUZDnNAOKY4mjbJ YDDg== X-Gm-Message-State: AOAM533dlZSAjlLXfyPht5RTy7Rpbi9ZnAXTTY7pXe76AB6heT0yWM8Y N1zKLI/6bsabLG2ZoKtlNcsygA== X-Google-Smtp-Source: ABdhPJyc8Mnmi4a1Aktae1QZY9O3zjJrhf37UfpwnWXbqDPMbp6c+LvxNDW5UP6lbOX5cvWPvekjsQ== X-Received: by 2002:a17:906:9257:: with SMTP id c23mr4487938ejx.86.1592495565343; Thu, 18 Jun 2020 08:52:45 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:44 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH v8 06/12] arm64: cpufeature: Export symbol read_sanitised_ftr_reg() Date: Thu, 18 Jun 2020 17:51:19 +0200 Message-Id: <20200618155125.1548969-7-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200618_085246_743067_24968F78 X-CRM114-Status: UNSURE ( 9.52 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:644 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fenghua.yu@intel.com, jacob.jun.pan@linux.intel.com, Jean-Philippe Brucker , catalin.marinas@arm.com, joro@8bytes.org, Suzuki K Poulose , robin.murphy@arm.com, hch@infradead.org, zhengxiang9@huawei.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, will@kernel.org, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The SMMUv3 driver would like to read the MMFR0 PARANGE field in order to share CPU page tables with devices. Allow the driver to be built as module by exporting the read_sanitized_ftr_reg() cpufeature symbol. Acked-by: Suzuki K Poulose Signed-off-by: Jean-Philippe Brucker --- arch/arm64/kernel/cpufeature.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 4ae41670c2e6b..ba1e17ad17447 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1068,6 +1068,7 @@ u64 read_sanitised_ftr_reg(u32 id) return 0; return regp->sys_val; } +EXPORT_SYMBOL_GPL(read_sanitised_ftr_reg); #define read_sysreg_case(r) \ case r: return read_sysreg_s(r) From patchwork Thu Jun 18 15:51:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612743 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6DE6614E3 for ; Thu, 18 Jun 2020 15:57:52 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4B8DD20732 for ; Thu, 18 Jun 2020 15:57:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="KtzZJa7u"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="nbPHVqQ4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4B8DD20732 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=N2EnDiM3JTv3/P3P+AxSHMkav/i+K2mdelbpWX/OKv4=; b=KtzZJa7uBcn3PQ Cv+vnZKbLgjLPVQ5MPHBYMsrpjWbO2tPy4EV3WKN81dQ5R2maGCOvLYhZaC71RQpjtuzLV+dHxIzf YLZ1iCOaXOU2HTh5pM6WskyQL31AadV2T66eJBsQbSS7KXH0qa1m6B1aR49dBfV6ST0YoRiDHvfYe gmQdP0fmsDKtZczL8skBP8IENDK7MJuSvebCdVlbk1KMxnNjWYi3YlkS1BlfvWfKa8ZGuQrdkqm2A aKbZv2Kob8f6KZIBI+8ha4dRS7vLqzXygUtp0Ljx2BAnQGb6NTx1mvtntU2WPn5jHLUR6+FPvUKL+ FfuebMjgiMeWNuvGNl+A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwv9-0000Cz-Pi; Thu, 18 Jun 2020 15:57:43 +0000 Received: from mail-ej1-x641.google.com ([2a00:1450:4864:20::641]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwqN-0002Eb-ML for linux-arm-kernel@lists.infradead.org; Thu, 18 Jun 2020 15:52:58 +0000 Received: by mail-ej1-x641.google.com with SMTP id gl26so6922014ejb.11 for ; Thu, 18 Jun 2020 08:52:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WJ6cCCV8E6DMFD7HqvFq9WRcUvuKrkpNdYaDLNUVviA=; b=nbPHVqQ4T+kuk9t8qrzVLGO2UEs5ToyO9mWMZ41PrASGRYC47YvXea1KDC9gDv25Md jtIzi3L7g542lm5vl1vNYWnl3LW7kdNQbBQKQwC2PkP5LFi6I9SJt1E2rpG/3mGzYBB2 htIOMn3gzhav+OS8Em4uEM8SHSczva7uyIuO04C1EYODgPhoZxpLfS8AM5ED8QgXim4R rY/YaGnn+bWlzEiIY8ceUIF6biDZiVmVaE6Kmg/zKCqHph9n17pkcS7fCpstKVVP8VuY l4nA+Ux69hXavZ4QiDwUcRa3y8V/VIWa3AJEl/xf41kb1fLuFalWazAHILTp/PWsqczP HkPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WJ6cCCV8E6DMFD7HqvFq9WRcUvuKrkpNdYaDLNUVviA=; b=los4qfc0Hhd1nKgSlf66hoQszRfq3xv10c2Th7btX8rfSv7S7fvsn2iKbUBSw9aWRZ TSV5C2juwlvLqlTrRKY9IbHvU9lA7A/UQSJsQvvOc3Qwg4hGsH6R5zlXfcLG2VBxiUDi L5PyozYdTAGZ0wItwLciNdFQBOEO+EMtWJ7edeibAMVKH5U37z9l/bfCR713adSGCyCD z8sc2v/NqpXgPKCArwlyScd43iEnO8rHxeWDqee+47ALeqTgJ/dMO3m/pEGTLEtgCc+G jTqYAnGMlNeYpKnsyJMLJ57HbuG1mdpZ1EHilKYwru+O8T7Zw+8i3PPU7IvvHIOtJHTx feQg== X-Gm-Message-State: AOAM5325BNOdYI3qZmUTltK1n+HzuZQwLgQo1HBpInFXmVXMwvVYwp9k ba1j5mMjE7fGjezaMNaKxN2jwg== X-Google-Smtp-Source: ABdhPJwT/k720uyCWIrCGDDjhYeelrlNaUkLfPLh0LBU3DqPpBSXfORheDk7AjOINMKzueIk2y95Ug== X-Received: by 2002:a17:907:2052:: with SMTP id pg18mr4376057ejb.513.1592495566301; Thu, 18 Jun 2020 08:52:46 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:45 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH v8 07/12] iommu/arm-smmu-v3: Share process page tables Date: Thu, 18 Jun 2020 17:51:20 +0200 Message-Id: <20200618155125.1548969-8-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200618_085247_786109_079495B5 X-CRM114-Status: GOOD ( 19.50 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:641 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fenghua.yu@intel.com, jacob.jun.pan@linux.intel.com, Jean-Philippe Brucker , catalin.marinas@arm.com, joro@8bytes.org, Suzuki K Poulose , robin.murphy@arm.com, hch@infradead.org, zhengxiang9@huawei.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, will@kernel.org, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org With Shared Virtual Addressing (SVA), we need to mirror CPU TTBR, TCR, MAIR and ASIDs in SMMU contexts. Each SMMU has a single ASID space split into two sets, shared and private. Shared ASIDs correspond to those obtained from the arch ASID allocator, and private ASIDs are used for "classic" map/unmap DMA. A possible conflict happens when trying to use a shared ASID that has already been allocated for private use by the SMMU driver. This will be addressed in a later patch by replacing the private ASID. At the moment we return -EBUSY. Each mm_struct shared with the SMMU will have a single context descriptor. Add a refcount to keep track of this. It will be protected by the global SVA lock. Acked-by: Suzuki K Poulose Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 150 +++++++++++++++++++++++++++++++++++- 1 file changed, 146 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index 937aa1af428d5..cabd942e4cbf3 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -33,6 +34,8 @@ #include +#include "io-pgtable-arm.h" + /* MMIO registers */ #define ARM_SMMU_IDR0 0x0 #define IDR0_ST_LVL GENMASK(28, 27) @@ -589,6 +592,9 @@ struct arm_smmu_ctx_desc { u64 ttbr; u64 tcr; u64 mair; + + refcount_t refs; + struct mm_struct *mm; }; struct arm_smmu_l1_ctx_desc { @@ -727,6 +733,7 @@ struct arm_smmu_option_prop { }; static DEFINE_XARRAY_ALLOC1(asid_xa); +static DEFINE_MUTEX(sva_lock); static struct arm_smmu_option_prop arm_smmu_options[] = { { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" }, @@ -1662,7 +1669,8 @@ static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, #ifdef __BIG_ENDIAN CTXDESC_CD_0_ENDI | #endif - CTXDESC_CD_0_R | CTXDESC_CD_0_A | CTXDESC_CD_0_ASET | + CTXDESC_CD_0_R | CTXDESC_CD_0_A | + (cd->mm ? 0 : CTXDESC_CD_0_ASET) | CTXDESC_CD_0_AA64 | FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) | CTXDESC_CD_0_V; @@ -1766,12 +1774,144 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_domain *smmu_domain) cdcfg->cdtab = NULL; } -static void arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd) +static void arm_smmu_init_cd(struct arm_smmu_ctx_desc *cd) { + refcount_set(&cd->refs, 1); +} + +static bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd) +{ + bool free; + struct arm_smmu_ctx_desc *old_cd; + if (!cd->asid) - return; + return false; + + free = refcount_dec_and_test(&cd->refs); + if (free) { + old_cd = xa_erase(&asid_xa, cd->asid); + WARN_ON(old_cd != cd); + } + return free; +} + +static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid) +{ + struct arm_smmu_ctx_desc *cd; - xa_erase(&asid_xa, cd->asid); + cd = xa_load(&asid_xa, asid); + if (!cd) + return NULL; + + if (cd->mm) { + /* All devices bound to this mm use the same cd struct. */ + refcount_inc(&cd->refs); + return cd; + } + + /* Ouch, ASID is already in use for a private cd. */ + return ERR_PTR(-EBUSY); +} + +__maybe_unused +static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) +{ + u16 asid; + int ret = 0; + u64 tcr, par, reg; + struct arm_smmu_ctx_desc *cd; + struct arm_smmu_ctx_desc *old_cd = NULL; + + lockdep_assert_held(&sva_lock); + + asid = mm_context_get(mm); + if (!asid) + return ERR_PTR(-ESRCH); + + cd = kzalloc(sizeof(*cd), GFP_KERNEL); + if (!cd) { + ret = -ENOMEM; + goto err_put_context; + } + + arm_smmu_init_cd(cd); + + old_cd = arm_smmu_share_asid(asid); + if (IS_ERR(old_cd)) { + ret = PTR_ERR(old_cd); + goto err_free_cd; + } else if (old_cd) { + if (WARN_ON(old_cd->mm != mm)) { + ret = -EINVAL; + goto err_free_cd; + } + kfree(cd); + mm_context_put(mm); + return old_cd; + } + + /* Fails if a private ASID has been allocated since we last checked */ + ret = xa_insert(&asid_xa, asid, cd, GFP_KERNEL); + if (ret) + goto err_free_cd; + + tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - VA_BITS) | + FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) | + FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) | + FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS) | + CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64; + + switch (PAGE_SIZE) { + case SZ_4K: + tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_4K); + break; + case SZ_16K: + tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_16K); + break; + case SZ_64K: + tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_64K); + break; + default: + WARN_ON(1); + ret = -EINVAL; + goto err_free_asid; + } + + reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1); + par = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_PARANGE_SHIFT); + tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_IPS, par); + + cd->ttbr = virt_to_phys(mm->pgd); + cd->tcr = tcr; + /* + * MAIR value is pretty much constant and global, so we can just get it + * from the current CPU register + */ + cd->mair = read_sysreg(mair_el1); + cd->asid = asid; + cd->mm = mm; + + return cd; + +err_free_asid: + arm_smmu_free_asid(cd); +err_free_cd: + kfree(cd); +err_put_context: + mm_context_put(mm); + return ERR_PTR(ret); +} + +__maybe_unused +static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd) +{ + lockdep_assert_held(&sva_lock); + + if (arm_smmu_free_asid(cd)) { + /* Unpin ASID */ + mm_context_put(cd->mm); + kfree(cd); + } } /* Stream table manipulation functions */ @@ -2481,6 +2621,8 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain, struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg; typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr; + arm_smmu_init_cd(&cfg->cd); + ret = xa_alloc(&asid_xa, &asid, &cfg->cd, XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); if (ret) From patchwork Thu Jun 18 15:51:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612745 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A5BC992A for ; Thu, 18 Jun 2020 15:58:11 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 82EBF20732 for ; Thu, 18 Jun 2020 15:58:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="YjXQkAbr"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="M9XzF/zD" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 82EBF20732 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=n0BJdJTTmMGxkAIs7mLmKsEqEsUVnvtrXsZHesPaOO8=; b=YjXQkAbrcBEPf/ X/mOP5Vt+8JJ/EOhVC9HAVZIti31CYBr/3IUzj6yZZTJ7y7JPszkIAnWV92iavJRIhYKiUQBJNlPF xcil1SJ2Tg0gSLSLxV0mmgTYUMgqhS4bvm2FxQ32f6Eb7Zsp9HoC1woN7Reqpu1HfP/mmvW1KLt2N OcwP24C3eXM9asQbWGv7rYk7xcmEyxhvk05WTnMDCbxypW0/HQ96jkpVUQS9D2Up+IvTyRmjqfwDN M4523dyxaAaJI/JgiVamU/2YuoTTx6M1z/iPIk8NXVNuSqD9akjoT8Gf70V4OOpHQszBIllne60N2 pCnHQ0Yx+yd6z0Xn0ONw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwvT-0000Vz-4u; Thu, 18 Jun 2020 15:58:03 +0000 Received: from mail-ej1-x644.google.com ([2a00:1450:4864:20::644]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwqO-0002FV-Rq for linux-arm-kernel@lists.infradead.org; Thu, 18 Jun 2020 15:52:58 +0000 Received: by mail-ej1-x644.google.com with SMTP id y13so6960018eju.2 for ; Thu, 18 Jun 2020 08:52:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Gl8/KHztk13IMZalRx1XfsgApopMYKofIndN5FPkOyc=; b=M9XzF/zDD9NAd1p8Tb01DcQs5VrpQgVrIqRMkSqX0CuDMCDkNlus8SVz7PXa0p7aG4 Ux3pLWztH/B/qiGFwSaVD7dNV3BgOvBi+WcVqdduC7tTgvezsmSgG2C5h8MrR9Rw9KXy tp7tE/DV/Het50vu5ncJZRtXgvPk4c8b9h2I5QA+pWaZxHvM3tO1ra5c0vRF4AuzPs9N +V2vQGJy5/RB7B57uhi0SPJspVMaAgH9v64B7DO0NS/BN461NbV/KNlr9LoVEY30joYc ftiIDufy0ptnDHRQNB3nm7Q9MXl+6OINX0Cp37Eht4FS9eHqKpVmu9x/NM7Q9Yx4HngK UKaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Gl8/KHztk13IMZalRx1XfsgApopMYKofIndN5FPkOyc=; b=oAKTf5iWXrGeXHKsRumjRAxxId8VPvf9EdVDCG/+MOWrV7cd8Yi/e6MKsCbWyZC/dB SLivWPKvCX5EHlOe5vW9jTEqAmeutbQv/92lxb7cuHwYxHxaqF3o6eER5VSEDl50G+ft bKGa/pCoURIz8otwbYMPtID4ijXUDv5b8oDBS5AM4NaDar2TdWQ/T1mDilYoYAgU2wVa 4dULkz6LQ+nYsREgqmglYS3oYzbU+eXU2WIdyHVAA17H3nR7C/qG1AgrgJT6X04bYppF ZbuAWbTcsB2MYT5R4O0E8wKLbX/pbFVtB2j8B6AdBF+pxKjdIDEur0XssKSAIutoDcGJ pU1w== X-Gm-Message-State: AOAM533iEAVbWo+/dix0Z3Xp/npXexN690rtW1Th5yz+KD3HEdVjwjZX mvvyLnCp8C6el2K9WQUc6Z9Cuw== X-Google-Smtp-Source: ABdhPJyO56ngTMw3kmdlxk17L2Xh8uXctk6ktH/xwwqapMb0kUPyRN+SHVC6d7bMVVWlr+RSxddU0w== X-Received: by 2002:a17:906:6890:: with SMTP id n16mr4383830ejr.553.1592495567422; Thu, 18 Jun 2020 08:52:47 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:47 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH v8 08/12] iommu/arm-smmu-v3: Seize private ASID Date: Thu, 18 Jun 2020 17:51:21 +0200 Message-Id: <20200618155125.1548969-9-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200618_085248_930226_A515001D X-CRM114-Status: GOOD ( 19.94 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:644 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fenghua.yu@intel.com, jacob.jun.pan@linux.intel.com, Jean-Philippe Brucker , catalin.marinas@arm.com, joro@8bytes.org, robin.murphy@arm.com, hch@infradead.org, zhengxiang9@huawei.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, will@kernel.org, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The SMMU has a single ASID space, the union of shared and private ASID sets. This means that the SMMU driver competes with the arch allocator for ASIDs. Shared ASIDs are those of Linux processes, allocated by the arch, and contribute in broadcast TLB maintenance. Private ASIDs are allocated by the SMMU driver and used for "classic" map/unmap DMA. They require command-queue TLB invalidations. When we pin down an mm_context and get an ASID that is already in use by the SMMU, it belongs to a private context. We used to simply abort the bind, but this is unfair to users that would be unable to bind a few seemingly random processes. Try to allocate a new private ASID for the context, and make the old ASID shared. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 101 +++++++++++++++++++++++++++++------- 1 file changed, 82 insertions(+), 19 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index cabd942e4cbf3..5506add42c9c8 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -733,6 +733,7 @@ struct arm_smmu_option_prop { }; static DEFINE_XARRAY_ALLOC1(asid_xa); +static DEFINE_MUTEX(asid_lock); static DEFINE_MUTEX(sva_lock); static struct arm_smmu_option_prop arm_smmu_options[] = { @@ -1537,6 +1538,17 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu, } /* Context descriptor manipulation functions */ +static void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) +{ + struct arm_smmu_cmdq_ent cmd = { + .opcode = CMDQ_OP_TLBI_NH_ASID, + .tlbi.asid = asid, + }; + + arm_smmu_cmdq_issue_cmd(smmu, &cmd); + arm_smmu_cmdq_issue_sync(smmu); +} + static void arm_smmu_sync_cd(struct arm_smmu_domain *smmu_domain, int ssid, bool leaf) { @@ -1795,9 +1807,18 @@ static bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd) return free; } +/* + * Try to reserve this ASID in the SMMU. If it is in use, try to steal it from + * the private entry. Careful here, we may be modifying the context tables of + * another SMMU! + */ static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid) { + int ret; + u32 new_asid; struct arm_smmu_ctx_desc *cd; + struct arm_smmu_device *smmu; + struct arm_smmu_domain *smmu_domain; cd = xa_load(&asid_xa, asid); if (!cd) @@ -1809,8 +1830,31 @@ static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid) return cd; } - /* Ouch, ASID is already in use for a private cd. */ - return ERR_PTR(-EBUSY); + smmu_domain = container_of(cd, struct arm_smmu_domain, s1_cfg.cd); + smmu = smmu_domain->smmu; + + ret = xa_alloc(&asid_xa, &new_asid, cd, + XA_LIMIT(1, 1 << smmu->asid_bits), GFP_KERNEL); + if (ret) + return ERR_PTR(-ENOSPC); + /* + * Race with unmap: TLB invalidations will start targeting the new ASID, + * which isn't assigned yet. We'll do an invalidate-all on the old ASID + * later, so it doesn't matter. + */ + cd->asid = new_asid; + + /* + * Update ASID and invalidate CD in all associated masters. There will + * be some overlap between use of both ASIDs, until we invalidate the + * TLB. + */ + arm_smmu_write_ctx_desc(smmu_domain, 0, cd); + + /* Invalidate TLB entries previously associated with that context */ + arm_smmu_tlb_inv_asid(smmu, asid); + + return NULL; } __maybe_unused @@ -1836,7 +1880,20 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) arm_smmu_init_cd(cd); + /* + * Serialize against arm_smmu_domain_finalise_s1() and + * arm_smmu_domain_free() as we might need to replace the private ASID + * from an existing CD. + */ + mutex_lock(&asid_lock); old_cd = arm_smmu_share_asid(asid); + if (!old_cd) { + ret = xa_insert(&asid_xa, asid, cd, GFP_KERNEL); + if (ret) + old_cd = ERR_PTR(ret); + } + mutex_unlock(&asid_lock); + if (IS_ERR(old_cd)) { ret = PTR_ERR(old_cd); goto err_free_cd; @@ -1850,11 +1907,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) return old_cd; } - /* Fails if a private ASID has been allocated since we last checked */ - ret = xa_insert(&asid_xa, asid, cd, GFP_KERNEL); - if (ret) - goto err_free_cd; - tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - VA_BITS) | FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) | FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) | @@ -2398,15 +2450,6 @@ static void arm_smmu_tlb_inv_context(void *cookie) struct arm_smmu_device *smmu = smmu_domain->smmu; struct arm_smmu_cmdq_ent cmd; - if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { - cmd.opcode = CMDQ_OP_TLBI_NH_ASID; - cmd.tlbi.asid = smmu_domain->s1_cfg.cd.asid; - cmd.tlbi.vmid = 0; - } else { - cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; - cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; - } - /* * NOTE: when io-pgtable is in non-strict mode, we may get here with * PTEs previously cleared by unmaps on the current CPU not yet visible @@ -2414,8 +2457,14 @@ static void arm_smmu_tlb_inv_context(void *cookie) * insertion to guarantee those are observed before the TLBI. Do be * careful, 007. */ - arm_smmu_cmdq_issue_cmd(smmu, &cmd); - arm_smmu_cmdq_issue_sync(smmu); + if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { + arm_smmu_tlb_inv_asid(smmu, smmu_domain->s1_cfg.cd.asid); + } else { + cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; + cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; + arm_smmu_cmdq_issue_cmd(smmu, &cmd); + arm_smmu_cmdq_issue_sync(smmu); + } arm_smmu_atc_inv_domain(smmu_domain, 0, 0, 0); } @@ -2599,9 +2648,15 @@ static void arm_smmu_domain_free(struct iommu_domain *domain) if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg; + /* + * Prevent arm_smmu_share_asid() from rewriting CD#0 while we're + * freeing it. + */ + mutex_lock(&asid_lock); if (cfg->cdcfg.cdtab) arm_smmu_free_cd_tables(smmu_domain); arm_smmu_free_asid(&cfg->cd); + mutex_unlock(&asid_lock); } else { struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg; if (cfg->vmid) @@ -2623,10 +2678,15 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain, arm_smmu_init_cd(&cfg->cd); + /* + * Prevent arm_smmu_share_asid() from seizing the private ASID we're + * allocating here until it is written to the CD. + */ + mutex_lock(&asid_lock); ret = xa_alloc(&asid_xa, &asid, &cfg->cd, XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); if (ret) - return ret; + goto out_unlock; cfg->s1cdmax = master->ssid_bits; @@ -2654,12 +2714,15 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain, if (ret) goto out_free_cd_tables; + mutex_unlock(&asid_lock); return 0; out_free_cd_tables: arm_smmu_free_cd_tables(smmu_domain); out_free_asid: arm_smmu_free_asid(&cfg->cd); +out_unlock: + mutex_unlock(&asid_lock); return ret; } From patchwork Thu Jun 18 15:51:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612747 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EFDAB92A for ; Thu, 18 Jun 2020 15:58:19 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C993920732 for ; Thu, 18 Jun 2020 15:58:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="iIAYB38M"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="fDbtc15F" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C993920732 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=6OQN4wwf0WmvLhWA5ybMJlgpzVXl+TkeMrBavpDsI6k=; b=iIAYB38MjPtzy/ nVPApsAxFcWMvvC8ycKNBZgs2PP0v3bvGulJlFUfGqkGVe+XybCPuzdxlOBi4NcPII3E3r7V9pTIc rz5VJIctqJMesisgXb8k3E0h22IJTsxWfzUyJ2iEEJt8dVjANmHNyn6S2lTm2zeNJoRwBxuyonLDL UmlKAhIQ52n9Ymot4I4+1nek4T3HFyHN01cZixPjUpDPLnlYZ3oJVLbQWG3vc+DWk/w++mjVspvzN 4nIU1Su9vucOoGcjCFmXA48adwnhETuYbZpxxqY4ipv9yAIMvnoJV+S2AjF6YyxGgoISX3jt8agKU esstZnbJh3OH92RmgPLQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwvi-0000oo-On; Thu, 18 Jun 2020 15:58:18 +0000 Received: from mail-ej1-x641.google.com ([2a00:1450:4864:20::641]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwqP-0002GB-O9 for linux-arm-kernel@lists.infradead.org; Thu, 18 Jun 2020 15:52:58 +0000 Received: by mail-ej1-x641.google.com with SMTP id p20so6897259ejd.13 for ; Thu, 18 Jun 2020 08:52:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MNKD6Dmbg1TaE6fvYxzmdVGqFWuaOjJYtMjLhWmnIvo=; b=fDbtc15FP1PMKFiVJC1xPK+dnPneh4gHR5Z5pyZz6iDbXrnrg7Gai4MJxSBwm5mM9v NGRaKISThUuuwgRt+TMw/RfEMNNwRvvCO988/Cysnqn3CIqtZE78iqHHATKrUvig9huz 5bDTNN+8IoMVQimN2851pvQMREQWtJ9Y/CFGobZqLvRXU5qzw9C9YZmBwdZ6YbFi5CLN 2ELl6LeunpXJFw9N3rafo00si6+tSLlbAoJPAInGGLozdmWbUReNSH6gjn+rshCdWJcQ PeYvdxW62Le2lQd7JDTWKinVvZq+ztbeM4jyLNX8bNHJB3Q3vj6RoJT4QVMe457y1ZGE erhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MNKD6Dmbg1TaE6fvYxzmdVGqFWuaOjJYtMjLhWmnIvo=; b=JnpbuK6yfWTu1QHNBCwgpDf1UQwV4TBqIxVcEp9rtIYQ+574q/vCh2SYwDeollM6Ko hVt5kzyX9zkUHQI3S6yVF0AVE25zg6Aapvm8ibVFK0tEiqlAbnJVya91DCrLhXZ+tckt tNg4jKTzToMMtMKtLLgi3V9bqXigWRjUmnlORzBuNhWqZ13sdn+dmkENFYLR+zzHSUz8 Oow7ByU+KkKo8hfuwmEDebiEd/kKeHMyVqXcVe+DHp2F9IviT4WfKZ307lPnHu/GfJ+y XmEWxlCgsDHZEuv7bcqkusX0gLknzDSiOFm/Tf5aSzu5q/OkeOnStM46PmsmJY8aPrp9 enhg== X-Gm-Message-State: AOAM530GvknenOZyNIQU6+xnWlCH/L4/wS/s17aoGpbvo6/9YU908ROA mzJPATNzy8bO/QCSC/oPA1YHEg== X-Google-Smtp-Source: ABdhPJztByorObtVQlxC6WShKEAqlvU9oXK33GgelOz6P5sV8o7rZktu08Gnv9xaL32FeH+u69sT9A== X-Received: by 2002:a17:906:4e59:: with SMTP id g25mr4478633ejw.60.1592495568467; Thu, 18 Jun 2020 08:52:48 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:48 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH v8 09/12] iommu/arm-smmu-v3: Check for SVA features Date: Thu, 18 Jun 2020 17:51:22 +0200 Message-Id: <20200618155125.1548969-10-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200618_085249_798714_B0DDC909 X-CRM114-Status: GOOD ( 13.41 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:641 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fenghua.yu@intel.com, jacob.jun.pan@linux.intel.com, Jean-Philippe Brucker , catalin.marinas@arm.com, joro@8bytes.org, Suzuki K Poulose , robin.murphy@arm.com, hch@infradead.org, zhengxiang9@huawei.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, will@kernel.org, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Aggregate all sanity-checks for sharing CPU page tables with the SMMU under a single ARM_SMMU_FEAT_SVA bit. For PCIe SVA, users also need to check FEAT_ATS and FEAT_PRI. For platform SVA, they will have to check FEAT_STALLS. Introduce ARM_SMMU_FEAT_BTM (Broadcast TLB Maintenance), but don't enable it at the moment. Since the entire VMID space is shared with the CPU, enabling DVM (by clearing SMMU_CR2.PTM) could result in over-invalidation and affect performance of stage-2 mappings. Cc: Suzuki K Poulose Signed-off-by: Jean-Philippe Brucker --- v7->v8: Use id_aa64mmfr0_parange_to_phys_shift() --- drivers/iommu/arm-smmu-v3.c | 48 +++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index 5506add42c9c8..e2d5171bfb7b9 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -654,6 +654,8 @@ struct arm_smmu_device { #define ARM_SMMU_FEAT_STALL_FORCE (1 << 13) #define ARM_SMMU_FEAT_VAX (1 << 14) #define ARM_SMMU_FEAT_RANGE_INV (1 << 15) +#define ARM_SMMU_FEAT_BTM (1 << 16) +#define ARM_SMMU_FEAT_SVA (1 << 17) u32 features; #define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0) @@ -3894,6 +3896,49 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass) return 0; } +static bool arm_smmu_supports_sva(struct arm_smmu_device *smmu) +{ + unsigned long reg, fld; + unsigned long oas; + unsigned long asid_bits; + + u32 feat_mask = ARM_SMMU_FEAT_BTM | ARM_SMMU_FEAT_COHERENCY; + + if ((smmu->features & feat_mask) != feat_mask) + return false; + + if (!(smmu->pgsize_bitmap & PAGE_SIZE)) + return false; + + /* + * Get the smallest PA size of all CPUs (sanitized by cpufeature). We're + * not even pretending to support AArch32 here. Abort if the MMU outputs + * addresses larger than what we support. + */ + reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1); + fld = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_PARANGE_SHIFT); + oas = id_aa64mmfr0_parange_to_phys_shift(fld); + if (smmu->oas < oas) + return false; + + /* We can support bigger ASIDs than the CPU, but not smaller */ + fld = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_ASID_SHIFT); + asid_bits = fld ? 16 : 8; + if (smmu->asid_bits < asid_bits) + return false; + + /* + * See max_pinned_asids in arch/arm64/mm/context.c. The following is + * generally the maximum number of bindable processes. + */ + if (IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0)) + asid_bits--; + dev_dbg(smmu->dev, "%d shared contexts\n", (1 << asid_bits) - + num_possible_cpus() - 2); + + return true; +} + static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) { u32 reg; @@ -4093,6 +4138,9 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) smmu->ias = max(smmu->ias, smmu->oas); + if (arm_smmu_supports_sva(smmu)) + smmu->features |= ARM_SMMU_FEAT_SVA; + dev_info(smmu->dev, "ias %lu-bit, oas %lu-bit (features 0x%08x)\n", smmu->ias, smmu->oas, smmu->features); return 0; From patchwork Thu Jun 18 15:51:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612741 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A759992A for ; Thu, 18 Jun 2020 15:57:28 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 850752075E for ; Thu, 18 Jun 2020 15:57:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="fG2uk4B/"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="upmKPCGB" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 850752075E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JxyOnhgwRjdpqVhaelL0P+hJI7tMJhgGBrXgvSHPz2k=; b=fG2uk4B/hZBkcs j9s3mWW5JBcWZaNjTgHC6bdewOv9nSkmOyn4EjS1r8WrJC7I48fSIpdZ0ciWDFi/szczYqkiVyClU Wzs56fblcYDNBGRFiBeDKzFvy+qveEm7PwaIiKijB3zMh9ikbE2PMgGSws46XtPX26uIzYHT8Aqw7 DCfPQrDpaFT1iCcGSsuGe0cUqh8pQH0U48oxCsUbnEGxxmECW2ukGwt1CumiUYKNyWdkOgs813ty/ 5j7XAz1w8Xh+rjiuWtEufYtRxL9kwJCipfLJAX9wHu/23+3zeHXiy75urgHPMjxEQaAggs9WUZvLV MQlu6Mdo5vQk88yDluwQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwus-0008OZ-5T; Thu, 18 Jun 2020 15:57:26 +0000 Received: from mail-ej1-x644.google.com ([2a00:1450:4864:20::644]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwqQ-0002HA-QS for linux-arm-kernel@lists.infradead.org; Thu, 18 Jun 2020 15:52:58 +0000 Received: by mail-ej1-x644.google.com with SMTP id gl26so6922174ejb.11 for ; Thu, 18 Jun 2020 08:52:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fvgnML7PsW3yuBKl7xZjxhkiu8/hUKUQrLt8sPfDx2o=; b=upmKPCGBBIWHElLSb36en66bExWNyxKTU8+A6gjfnLDCbV4TrPCbN/fl0wuPvVmkGr d8jLr65IH2jaDOYHVDrozzBMWtqseimc1dPo+ZDyg2SFM8cQgdHPycaXPFy5e2K/WwOD jwbVFdqYWQeju2yJe6gVJE2fEZ13tW4N0uLNM5zfiLAON/7LJ+VHAMXiXbr/XzEvC+4e 9dzkdOSj32WiAZUYacq0/NJtrKIAQK5MWosVVifejiJz3X9Zox6CZ3aBbn4584SnaRbh 2iy7wI/GO2l0PzURfeGYBqufuvU14VMyND1O1Yle9TjmVSoLHc1QIYduERuKv4e1yaJx DFgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fvgnML7PsW3yuBKl7xZjxhkiu8/hUKUQrLt8sPfDx2o=; b=AoRAPxw2hypWV2Ew4wnFKLtgpwN100+222JVb3gDdBfQX9r5ZFjWWe3kSfqeAeaRZP a2sVrKXKJqgjx+iZCu+UDg0lG8bwBP5QrfvIAAgsPyx3ihD8qnBKoqGr9oQ0LXMqUE3e hh4XSuPRl5js8/c5ki2ZOfAq0BvaDo8qyd0OsrLVuvedr3tX/E9or9VSiQqyvvJoK8el V581XbhdiBz/fVx0N1MuIB+Mtpiqm6yYFSSNSftHxfqX44r90bGke/aV8V3DP3oQqvBD mpHW7azqetRmBtM9w0ygJP9kXRVrtjiWl83/6eMQ7x15LeqUsm4IbarwfxL/gOHbh82x xM8w== X-Gm-Message-State: AOAM531GdDf0SaP/R8MTJtuyYm55kOcgT9/jJ/wiELecnQCFM1o1V2t9 uy4x2XWmTI/TVcIr5U5d8DbcU9W4NMBGHA== X-Google-Smtp-Source: ABdhPJwfoWVDnLWg9TdIqXpYSSCtIY2YsTCrcdjU9F3xU3f5D07bEev+3xV58gc2YYHmLVPFIbJD1g== X-Received: by 2002:a17:906:1d5b:: with SMTP id o27mr4460428ejh.344.1592495569497; Thu, 18 Jun 2020 08:52:49 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:49 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH v8 10/12] iommu/arm-smmu-v3: Add SVA device feature Date: Thu, 18 Jun 2020 17:51:23 +0200 Message-Id: <20200618155125.1548969-11-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200618_085250_951878_AC8F0DC3 X-CRM114-Status: GOOD ( 14.74 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:644 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fenghua.yu@intel.com, jacob.jun.pan@linux.intel.com, Jean-Philippe Brucker , catalin.marinas@arm.com, joro@8bytes.org, robin.murphy@arm.com, hch@infradead.org, zhengxiang9@huawei.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, will@kernel.org, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Implement the IOMMU device feature callbacks to support the SVA feature. At the moment dev_has_feat() returns false since I/O Page Faults isn't yet implemented. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 124 ++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index e2d5171bfb7b9..d357c9f77bf7f 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -698,6 +698,8 @@ struct arm_smmu_master { u32 *sids; unsigned int num_sids; bool ats_enabled; + bool sva_enabled; + struct list_head bonds; unsigned int ssid_bits; }; @@ -3000,6 +3002,19 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) master = dev_iommu_priv_get(dev); smmu = master->smmu; + /* + * Checking that SVA is disabled ensures that this device isn't bound to + * any mm, and can be safely detached from its old domain. Bonds cannot + * be removed concurrently since we're holding the group mutex. + */ + mutex_lock(&sva_lock); + if (master->sva_enabled) { + mutex_unlock(&sva_lock); + dev_err(dev, "cannot attach - SVA enabled\n"); + return -EBUSY; + } + mutex_unlock(&sva_lock); + arm_smmu_detach_dev(master); mutex_lock(&smmu_domain->init_mutex); @@ -3147,6 +3162,7 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) master->smmu = smmu; master->sids = fwspec->ids; master->num_sids = fwspec->num_ids; + INIT_LIST_HEAD(&master->bonds); dev_iommu_priv_set(dev, master); /* Check the SIDs are in range of the SMMU and our stream table */ @@ -3199,6 +3215,7 @@ static void arm_smmu_release_device(struct device *dev) return; master = dev_iommu_priv_get(dev); + WARN_ON(master->sva_enabled); arm_smmu_detach_dev(master); arm_smmu_disable_pasid(master); kfree(master); @@ -3316,6 +3333,109 @@ static void arm_smmu_get_resv_regions(struct device *dev, iommu_dma_get_resv_regions(dev, head); } +static bool arm_smmu_iopf_supported(struct arm_smmu_master *master) +{ + return false; +} + +static bool arm_smmu_dev_has_feature(struct device *dev, + enum iommu_dev_features feat) +{ + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + + if (!master) + return false; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + if (!(master->smmu->features & ARM_SMMU_FEAT_SVA)) + return false; + + /* SSID and IOPF support are mandatory for the moment */ + return master->ssid_bits && arm_smmu_iopf_supported(master); + default: + return false; + } +} + +static bool arm_smmu_dev_feature_enabled(struct device *dev, + enum iommu_dev_features feat) +{ + bool enabled = false; + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + + if (!master) + return false; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + mutex_lock(&sva_lock); + enabled = master->sva_enabled; + mutex_unlock(&sva_lock); + return enabled; + default: + return false; + } +} + +static int arm_smmu_dev_enable_sva(struct device *dev) +{ + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + + mutex_lock(&sva_lock); + master->sva_enabled = true; + mutex_unlock(&sva_lock); + + return 0; +} + +static int arm_smmu_dev_disable_sva(struct device *dev) +{ + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + + mutex_lock(&sva_lock); + if (!list_empty(&master->bonds)) { + dev_err(dev, "cannot disable SVA, device is bound\n"); + mutex_unlock(&sva_lock); + return -EBUSY; + } + master->sva_enabled = false; + mutex_unlock(&sva_lock); + + return 0; +} + +static int arm_smmu_dev_enable_feature(struct device *dev, + enum iommu_dev_features feat) +{ + if (!arm_smmu_dev_has_feature(dev, feat)) + return -ENODEV; + + if (arm_smmu_dev_feature_enabled(dev, feat)) + return -EBUSY; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + return arm_smmu_dev_enable_sva(dev); + default: + return -EINVAL; + } +} + +static int arm_smmu_dev_disable_feature(struct device *dev, + enum iommu_dev_features feat) +{ + if (!arm_smmu_dev_feature_enabled(dev, feat)) + return -EINVAL; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + return arm_smmu_dev_disable_sva(dev); + default: + return -EINVAL; + } +} + static struct iommu_ops arm_smmu_ops = { .capable = arm_smmu_capable, .domain_alloc = arm_smmu_domain_alloc, @@ -3334,6 +3454,10 @@ static struct iommu_ops arm_smmu_ops = { .of_xlate = arm_smmu_of_xlate, .get_resv_regions = arm_smmu_get_resv_regions, .put_resv_regions = generic_iommu_put_resv_regions, + .dev_has_feat = arm_smmu_dev_has_feature, + .dev_feat_enabled = arm_smmu_dev_feature_enabled, + .dev_enable_feat = arm_smmu_dev_enable_feature, + .dev_disable_feat = arm_smmu_dev_disable_feature, .pgsize_bitmap = -1UL, /* Restricted during device attach */ }; From patchwork Thu Jun 18 15:51:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612749 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C185492A for ; Thu, 18 Jun 2020 15:58:51 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7C01C20732 for ; Thu, 18 Jun 2020 15:58:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="UtAoAS/j"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="F3rq2hPx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7C01C20732 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=o3Z3Lzie/vLOKDb8zWvoFBYujDkrgzr6uwIHY1y5tkM=; b=UtAoAS/jW7zN1O f3WSoxlZzyKllmfSIfi+jKGz05T3KtpzZ1uvIdWIummWC1+8BBAvDho2EIAdaVEeDCrfOUNU/0SKi ea8h/fmbOQpDuEsZxrZzYU3TVhshyeB31uI230Z1cgUA7/HbrB9lzvAPk2L1VqatWr/oK/7x1bOAd /yAFeQ+swpo7NzUW+2cYkdo21KWA+LyM1gDB+kcO2FpADxb9EdOYL/UQ2LGA4PJv4lAgNGqVHTN26 /9p2NLqLL49wf1vAuaoL43xdIJ9KKHecDkrk56/6G9s8BB0oJrRO5capH6Fxx4ZMC73nWA7cwL9pf 3rKqHAIYZPn8XPfanV7A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlww7-000169-5W; Thu, 18 Jun 2020 15:58:43 +0000 Received: from mail-ej1-x644.google.com ([2a00:1450:4864:20::644]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwqS-0002ID-Aa for linux-arm-kernel@lists.infradead.org; Thu, 18 Jun 2020 15:53:01 +0000 Received: by mail-ej1-x644.google.com with SMTP id l27so6978458ejc.1 for ; Thu, 18 Jun 2020 08:52:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZuZSIKZP3e6lWCHyBpmCAdFcfA2fkGVm1e+K1v+Ant0=; b=F3rq2hPxOLLhcTkt0qXcmgkdti/SxA0Grlpb18yg1aqay3xp7saO7yaI5cJDrwNGU9 7YzJSsMWNHBgA8tjL/TKszprqUhJZ2rxejsrejBhQXpziLn58+b6UTqLuiAYehEKmkU9 i/Z3FzVcQRpWCmkkDlz0h0M9NhLsCLrMbDJCcyugK58ZkQzG7PYygeDXpUxg6AMOOuAT ElVKAq+V4tbKwK8OAuGOaK4LkH+lYtozfz0PzAt+DxmXYWvkE413Uj+2gdjl2jJTdKDJ YnUCvfWN1cC5XWokXC50laaYfF7W2292OEVTTaBN0HEHVpHaZErrJIg1q4mhOAiF9pHg BKIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZuZSIKZP3e6lWCHyBpmCAdFcfA2fkGVm1e+K1v+Ant0=; b=YRDoYLcgJJAX7Q4X5Cd/ptPKo8SRAJMtaESP5dsUppnEQQUd1VHRo70l9sjUYBJwb3 JJOHSQDl0rvsfLqT4bC1k/3kx2Tz7v63UdeM9Q31zB+o9kvi3+rq4JvHdpZ2vXa+hF0V I99bwP4bH5AW2bU7hq9Vkx1LbwqMzVeLaEcd5aUdIaPDiZ8107BPTdqAjwKDR0x79qZb lJFPFkCRDMYfcReGxwb//T3P0d7vaxq54nljOZWNSX0Zjk2Z4W0jHcFSK/AldikOYDSf JUSL+BR/URYIqPLN46YEV1TZLpea9ZTAhHE1uLiVFDf/c3JQDEzeoMmCCQAosvFUUwxK e6vQ== X-Gm-Message-State: AOAM532mA9Dzi4JPn6oLstQax8EVfH6iZEtEXWoj7UR8q5xwKD+IEVh9 AkV8hLGFPzeOyrtWlrPHKSUW5Q== X-Google-Smtp-Source: ABdhPJxCAIkVq2rLjUObspQLQB8nCyvTQOzGP/GfT0DoAcdczzKtcZexhFXO/NIEgpFwmTF2mlsfag== X-Received: by 2002:a17:906:241b:: with SMTP id z27mr4457689eja.267.1592495570579; Thu, 18 Jun 2020 08:52:50 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:50 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH v8 11/12] iommu/arm-smmu-v3: Implement iommu_sva_bind/unbind() Date: Thu, 18 Jun 2020 17:51:24 +0200 Message-Id: <20200618155125.1548969-12-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200618_085252_408370_9FF83BDE X-CRM114-Status: GOOD ( 22.76 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:644 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fenghua.yu@intel.com, jacob.jun.pan@linux.intel.com, Jean-Philippe Brucker , catalin.marinas@arm.com, joro@8bytes.org, robin.murphy@arm.com, hch@infradead.org, zhengxiang9@huawei.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, will@kernel.org, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The sva_bind() function allows devices to access process address spaces using a PASID (aka SSID). (1) bind() allocates or gets an existing MMU notifier tied to the (domain, mm) pair. Each mm gets one PASID. (2) Any change to the address space calls invalidate_range() which sends ATC invalidations (in a subsequent patch). (3) When the process address space dies, the release() notifier disables the CD to allow reclaiming the page tables. Since release() has to be light we do not instruct device drivers to stop DMA here, we just ignore incoming page faults from this point onwards. To avoid any event 0x0a print (C_BAD_CD) we disable translation without clearing CD.V. PCIe Translation Requests and Page Requests are silently denied. Don't clear the R bit because the S bit can't be cleared when STALL_MODEL==0b10 (forced), and clearing R without clearing S is useless. Faulting transactions will stall and will be aborted by the IOPF handler. (4) After stopping DMA, the device driver releases the bond by calling unbind(). We release the MMU notifier, free the PASID and the bond. Three structures keep track of bonds: * arm_smmu_bond: one per {device, mm} pair, the handle returned to the device driver for a bind() request. * arm_smmu_mmu_notifier: one per {domain, mm} pair, deals with ATS/TLB invalidations and clearing the context descriptor on mm exit. * arm_smmu_ctx_desc: one per mm, holds the pinned ASID and pgd. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/Kconfig | 2 + drivers/iommu/arm-smmu-v3.c | 263 +++++++++++++++++++++++++++++++++++- 2 files changed, 260 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index 74a10e7a8d082..d1ad6f63d2d42 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -437,8 +437,10 @@ config ARM_SMMU_V3 tristate "ARM Ltd. System MMU Version 3 (SMMUv3) Support" depends on ARM64 select IOMMU_API + select IOMMU_SVA_LIB select IOMMU_IO_PGTABLE_LPAE select GENERIC_MSI_IRQ_DOMAIN + select MMU_NOTIFIER help Support for implementations of the ARM System MMU architecture version 3 providing translation support to a PCIe root complex. diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index d357c9f77bf7f..af551f3c78a78 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -35,6 +36,7 @@ #include #include "io-pgtable-arm.h" +#include "iommu-sva-lib.h" /* MMIO registers */ #define ARM_SMMU_IDR0 0x0 @@ -729,8 +731,32 @@ struct arm_smmu_domain { struct list_head devices; spinlock_t devices_lock; + + struct list_head mmu_notifiers; +}; + +struct arm_smmu_mmu_notifier { + struct mmu_notifier mn; + struct arm_smmu_ctx_desc *cd; + bool cleared; + refcount_t refs; + struct list_head list; + struct arm_smmu_domain *domain; }; +#define mn_to_smmu(mn) container_of(mn, struct arm_smmu_mmu_notifier, mn) + +struct arm_smmu_bond { + struct iommu_sva sva; + struct mm_struct *mm; + struct arm_smmu_mmu_notifier *smmu_mn; + struct list_head list; + refcount_t refs; +}; + +#define sva_to_bond(handle) \ + container_of(handle, struct arm_smmu_bond, sva) + struct arm_smmu_option_prop { u32 opt; const char *prop; @@ -740,6 +766,13 @@ static DEFINE_XARRAY_ALLOC1(asid_xa); static DEFINE_MUTEX(asid_lock); static DEFINE_MUTEX(sva_lock); +/* + * When a process dies, DMA is still running but we need to clear the pgd. If we + * simply cleared the valid bit from the context descriptor, we'd get event 0x0a + * which are not recoverable. + */ +static struct arm_smmu_ctx_desc invalid_cd = { 0 }; + static struct arm_smmu_option_prop arm_smmu_options[] = { { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" }, { ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"}, @@ -1643,7 +1676,9 @@ static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, * (2) Install a secondary CD, for SID+SSID traffic. * (3) Update ASID of a CD. Atomically write the first 64 bits of the * CD, then invalidate the old entry and mappings. - * (4) Remove a secondary CD. + * (4) Quiesce the context without clearing the valid bit. Disable + * translation, and ignore any translation fault. + * (5) Remove a secondary CD. */ u64 val; bool cd_live; @@ -1660,8 +1695,10 @@ static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, val = le64_to_cpu(cdptr[0]); cd_live = !!(val & CTXDESC_CD_0_V); - if (!cd) { /* (4) */ + if (!cd) { /* (5) */ val = 0; + } else if (cd == &invalid_cd) { /* (4) */ + val |= CTXDESC_CD_0_TCR_EPD0; } else if (cd_live) { /* (3) */ val &= ~CTXDESC_CD_0_ASID; val |= FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid); @@ -1861,7 +1898,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid) return NULL; } -__maybe_unused static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) { u16 asid; @@ -1958,7 +1994,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) return ERR_PTR(ret); } -__maybe_unused static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd) { lockdep_assert_held(&sva_lock); @@ -2591,6 +2626,8 @@ static bool arm_smmu_capable(enum iommu_cap cap) } } +static struct mmu_notifier_ops arm_smmu_mmu_notifier_ops; + static struct iommu_domain *arm_smmu_domain_alloc(unsigned type) { struct arm_smmu_domain *smmu_domain; @@ -2618,6 +2655,7 @@ static struct iommu_domain *arm_smmu_domain_alloc(unsigned type) mutex_init(&smmu_domain->init_mutex); INIT_LIST_HEAD(&smmu_domain->devices); spin_lock_init(&smmu_domain->devices_lock); + INIT_LIST_HEAD(&smmu_domain->mmu_notifiers); return &smmu_domain->domain; } @@ -3114,6 +3152,207 @@ arm_smmu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova) return ops->iova_to_phys(ops, iova); } +static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) +{ + struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); + struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + + mutex_lock(&sva_lock); + if (smmu_mn->cleared) { + mutex_unlock(&sva_lock); + return; + } + + /* + * DMA may still be running. Keep the cd valid to avoid C_BAD_CD events, + * but disable translation. + */ + arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, &invalid_cd); + + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); + + smmu_mn->cleared = true; + mutex_unlock(&sva_lock); +} + +static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn) +{ + kfree(mn_to_smmu(mn)); +} + +static struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = { + .release = arm_smmu_mm_release, + .free_notifier = arm_smmu_mmu_notifier_free, +}; + +/* Allocate or get existing MMU notifier for this {domain, mm} pair */ +static struct arm_smmu_mmu_notifier * +arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain, + struct mm_struct *mm) +{ + int ret; + struct arm_smmu_ctx_desc *cd; + struct arm_smmu_mmu_notifier *smmu_mn; + + lockdep_assert_held(&sva_lock); + + list_for_each_entry(smmu_mn, &smmu_domain->mmu_notifiers, list) { + if (smmu_mn->mn.mm == mm) { + refcount_inc(&smmu_mn->refs); + return smmu_mn; + } + } + + cd = arm_smmu_alloc_shared_cd(mm); + if (IS_ERR(cd)) + return ERR_CAST(cd); + + smmu_mn = kzalloc(sizeof(*smmu_mn), GFP_KERNEL); + if (!smmu_mn) { + ret = -ENOMEM; + goto err_free_cd; + } + + refcount_set(&smmu_mn->refs, 1); + smmu_mn->cd = cd; + smmu_mn->domain = smmu_domain; + smmu_mn->mn.ops = &arm_smmu_mmu_notifier_ops; + + ret = mmu_notifier_register(&smmu_mn->mn, mm); + if (ret) { + kfree(smmu_mn); + goto err_free_cd; + } + + ret = arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, cd); + if (ret) + goto err_put_notifier; + + list_add(&smmu_mn->list, &smmu_domain->mmu_notifiers); + return smmu_mn; + +err_put_notifier: + /* Frees smmu_mn */ + mmu_notifier_put(&smmu_mn->mn); +err_free_cd: + arm_smmu_free_shared_cd(cd); + return ERR_PTR(ret); +} + +static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) +{ + struct mm_struct *mm = smmu_mn->mn.mm; + struct arm_smmu_ctx_desc *cd = smmu_mn->cd; + struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + + lockdep_assert_held(&sva_lock); + + if (!refcount_dec_and_test(&smmu_mn->refs)) + return; + + list_del(&smmu_mn->list); + arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, NULL); + + /* + * If we went through clear(), we've already invalidated, and no + * new TLB entry can have been formed. + */ + if (!smmu_mn->cleared) + arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid); + + /* Frees smmu_mn */ + mmu_notifier_put(&smmu_mn->mn); + arm_smmu_free_shared_cd(cd); +} + +static struct iommu_sva * +__arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm) +{ + int ret; + struct arm_smmu_bond *bond; + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct iommu_domain *domain = iommu_get_domain_for_dev(dev); + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + lockdep_assert_held(&sva_lock); + + if (!master || !master->sva_enabled) + return ERR_PTR(-ENODEV); + + /* If bind() was already called for this {dev, mm} pair, reuse it. */ + list_for_each_entry(bond, &master->bonds, list) { + if (bond->mm == mm) { + refcount_inc(&bond->refs); + return &bond->sva; + } + } + + bond = kzalloc(sizeof(*bond), GFP_KERNEL); + if (!bond) + return ERR_PTR(-ENOMEM); + + /* Allocate a PASID for this mm if necessary */ + ret = iommu_sva_alloc_pasid(mm, 1, (1U << master->ssid_bits) - 1); + if (ret) + goto err_free_bond; + + bond->mm = mm; + bond->sva.dev = dev; + refcount_set(&bond->refs, 1); + + bond->smmu_mn = arm_smmu_mmu_notifier_get(smmu_domain, mm); + if (IS_ERR(bond->smmu_mn)) { + ret = PTR_ERR(bond->smmu_mn); + goto err_free_pasid; + } + + list_add(&bond->list, &master->bonds); + return &bond->sva; + +err_free_pasid: + iommu_sva_free_pasid(mm); +err_free_bond: + kfree(bond); + return ERR_PTR(ret); +} + +static struct iommu_sva * +arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, void *drvdata) +{ + struct iommu_sva *handle; + struct iommu_domain *domain = iommu_get_domain_for_dev(dev); + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1) + return ERR_PTR(-EINVAL); + + mutex_lock(&sva_lock); + handle = __arm_smmu_sva_bind(dev, mm); + mutex_unlock(&sva_lock); + return handle; +} + +static void arm_smmu_sva_unbind(struct iommu_sva *handle) +{ + struct arm_smmu_bond *bond = sva_to_bond(handle); + + mutex_lock(&sva_lock); + if (refcount_dec_and_test(&bond->refs)) { + list_del(&bond->list); + arm_smmu_mmu_notifier_put(bond->smmu_mn); + iommu_sva_free_pasid(bond->mm); + kfree(bond); + } + mutex_unlock(&sva_lock); +} + +static int arm_smmu_sva_get_pasid(struct iommu_sva *handle) +{ + struct arm_smmu_bond *bond = sva_to_bond(handle); + + return bond->mm->pasid; +} + static struct platform_driver arm_smmu_driver; static @@ -3458,6 +3697,9 @@ static struct iommu_ops arm_smmu_ops = { .dev_feat_enabled = arm_smmu_dev_feature_enabled, .dev_enable_feat = arm_smmu_dev_enable_feature, .dev_disable_feat = arm_smmu_dev_disable_feature, + .sva_bind = arm_smmu_sva_bind, + .sva_unbind = arm_smmu_sva_unbind, + .sva_get_pasid = arm_smmu_sva_get_pasid, .pgsize_bitmap = -1UL, /* Restricted during device attach */ }; @@ -4520,6 +4762,16 @@ static const struct of_device_id arm_smmu_of_match[] = { }; MODULE_DEVICE_TABLE(of, arm_smmu_of_match); +static void arm_smmu_driver_unregister(struct platform_driver *drv) +{ + /* + * Wait for all notifiers free() RCU callbacks, since they are still + * using the arm_smmu_mmu_notifier_ops. + */ + mmu_notifier_synchronize(); + platform_driver_unregister(drv); +} + static struct platform_driver arm_smmu_driver = { .driver = { .name = "arm-smmu-v3", @@ -4530,7 +4782,8 @@ static struct platform_driver arm_smmu_driver = { .remove = arm_smmu_device_remove, .shutdown = arm_smmu_device_shutdown, }; -module_platform_driver(arm_smmu_driver); +module_driver(arm_smmu_driver, platform_driver_register, + arm_smmu_driver_unregister); MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations"); MODULE_AUTHOR("Will Deacon "); From patchwork Thu Jun 18 15:51:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11612751 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 94C9392A for ; Thu, 18 Jun 2020 15:59:12 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 65B4E2075E for ; Thu, 18 Jun 2020 15:59:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Y95YAGF8"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="aE81FxUo"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="DYibauvJ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 65B4E2075E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=NmBb9QC0oBb8Dk0a5LmN3RpQMFqkd66utBqzQ9+PjVk=; b=Y95YAGF876+Hxe i7Viz15Np94JXNEq9YgU4FHaGXYzn8Gx97PWSzWYxqBsqH6/SSv0BFmgnl7y9R4mUVMwaitwN27sv /fOdtjUgi8JbybcW/XiV4qXlfk/6CvF8tThxWX8Li88Lp6joXS+wwqsllXnAvzqhCBYJvoPUbLKj+ noUVv39tIGv6TKTvPWUWjCrwyXoY951CPg82o/YcRCmg1yL0ckli8zRnu+37nuFpXpnD2PIEfXN0G FBGtE4fObDT2bSZDqYGvnf1ARpDKzjX5hCz7gLsf9ln9ebVc5VJjvJBxABu3K1EgxzJ19w9qhp9OB EPUw44nPUjsL49evAN2A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwwY-0001Tc-9e; Thu, 18 Jun 2020 15:59:10 +0000 Received: from merlin.infradead.org ([205.233.59.134]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwu8-0007WD-9L for linux-arm-kernel@bombadil.infradead.org; Thu, 18 Jun 2020 15:56:40 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=K0sD9muywK0nbycZHdsqSBZ4T+dLVPNn3EX5mSicmkA=; b=aE81FxUouD2eKZh+lGWdVywcBy ERQFsxruTr2LNisxxQ05bPTBWLyRnA474r+QI47M9eVGfN+OepbK2IYf2t84vxGcmxhaMLp/vCTnd SwScqLhHyA2TOo/DHBWcM/k0FzOpzSAik+YtX76OQXN/UogcJ7TKfAgzF3IVIoF/RuAurynv3Kaw0 GnZ7kVqtG0H9Oztte/ywPyNxN0CZ54VlLUAeHy5AhT7P6rt1ZWhCNf+QkpK4ymN0YNZ34erFQfbDE TnHVcehdPb8zGL/O3fM9sqTqDw6s9F2y0KzpcUCDmR5h2mkNsJBiqrwLWCi4yhM8YZ0T/gd1TX16O YttFhLGg==; Received: from mail-ej1-x641.google.com ([2a00:1450:4864:20::641]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jlwqT-0008TH-Jz for linux-arm-kernel@lists.infradead.org; Thu, 18 Jun 2020 15:52:54 +0000 Received: by mail-ej1-x641.google.com with SMTP id p20so6897443ejd.13 for ; Thu, 18 Jun 2020 08:52:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=K0sD9muywK0nbycZHdsqSBZ4T+dLVPNn3EX5mSicmkA=; b=DYibauvJ2nimmPQCzKZsqFWxsw1cJOa83Q8lBOizOX8qC2sph7xBeWfJVDLqHcPsZT jUpy1d/5G5neFk7+aPYKN3NiWZz2OD8ek+65tg6OyKTdhesLhyiouRddaM9PTbG2zSM0 vNbXEBv/Zfu7Xpa9afY1wlNWbf7NYURm9ezvJnPuhwmG6QXxlemJYFcE9cyW4NEcWZin j2SQ+U1tQkwf6J+OWfDQA3p4lIuVm/Opa2zG6+W+u6svNiCqSklKXaS1QWT1/RMzMNp0 SPJ6ObCtmIf61+7zubUKVxAmdcChrBc908ruLceXol1cTPgQeBcB+yXy6tuW8+XaC7gk FL3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=K0sD9muywK0nbycZHdsqSBZ4T+dLVPNn3EX5mSicmkA=; b=pu8HRqhNd5Q8ZqvuKO/ID/zs1dBGDeCIuHGyjN8PDJWPr0hZSSGZu2QEgMNznjIoYy tRz2J2/mVGZvOzgGpccg294j/69gyB9wV32dqPiWUaYaon43ptupNFXG7WIuxs20eXHA B8e/LzQgCg6y6T+W+n3EQVCxBxgocuseEyG7uof/TZh6I8vS4khejxoCa8ah0QG3MJRD cVKl5qY570nX/TT5Wb3QSjCaNvYputX2tOzgYqoodrFtKhjlRpqPxneArQx1N28Rh8Dp VI/N1WMANIIuHdn8twyiORzC96JesVgK3ZIOEcfaEAGcJ4m4DMixGumXw7il5QNMIFWP Kemw== X-Gm-Message-State: AOAM530sy3BT5M/M8ROuM8kg+GVh5BkoqDB2ZyGJTJ0ey87LVTKmyEJn 2DF56YLaA8k6D8eOSv29EqUYDg== X-Google-Smtp-Source: ABdhPJyPJTA3MoAcCwZxTHDutpdNm45kp002C6I1TJUitRaZbiXuqClT8u8iuWaKBu95Oo1397Yh/g== X-Received: by 2002:a17:906:ae85:: with SMTP id md5mr4476555ejb.213.1592495571768; Thu, 18 Jun 2020 08:52:51 -0700 (PDT) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 63sm2402267edy.8.2020.06.18.08.52.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jun 2020 08:52:51 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH v8 12/12] iommu/arm-smmu-v3: Hook up ATC invalidation to mm ops Date: Thu, 18 Jun 2020 17:51:25 +0200 Message-Id: <20200618155125.1548969-13-jean-philippe@linaro.org> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200618155125.1548969-1-jean-philippe@linaro.org> References: <20200618155125.1548969-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-Spam-Note: CRM114 invocation failed X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:641 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fenghua.yu@intel.com, jacob.jun.pan@linux.intel.com, Jean-Philippe Brucker , catalin.marinas@arm.com, joro@8bytes.org, robin.murphy@arm.com, hch@infradead.org, zhengxiang9@huawei.com, Jonathan.Cameron@huawei.com, zhangfei.gao@linaro.org, will@kernel.org, xuzaibo@huawei.com, baolu.lu@linux.intel.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The invalidate_range() notifier is called for any change to the address space. Perform the required ATC invalidations. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index af551f3c78a78..972c061399fc9 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -2378,6 +2378,20 @@ arm_smmu_atc_inv_to_cmd(int ssid, unsigned long iova, size_t size, size_t inval_grain_shift = 12; unsigned long page_start, page_end; + /* + * ATS and PASID: + * + * If substream_valid is clear, the PCIe TLP is sent without a PASID + * prefix. In that case all ATC entries within the address range are + * invalidated, including those that were requested with a PASID! There + * is no way to invalidate only entries without PASID. + * + * When using STRTAB_STE_1_S1DSS_SSID0 (reserving CD 0 for non-PASID + * traffic), translation requests without PASID create ATC entries + * without PASID, which must be invalidated with substream_valid clear. + * This has the unpleasant side-effect of invalidating all PASID-tagged + * ATC entries within the address range. + */ *cmd = (struct arm_smmu_cmdq_ent) { .opcode = CMDQ_OP_ATC_INV, .substream_valid = !!ssid, @@ -2421,12 +2435,12 @@ arm_smmu_atc_inv_to_cmd(int ssid, unsigned long iova, size_t size, cmd->atc.size = log2_span; } -static int arm_smmu_atc_inv_master(struct arm_smmu_master *master) +static int arm_smmu_atc_inv_master(struct arm_smmu_master *master, int ssid) { int i; struct arm_smmu_cmdq_ent cmd; - arm_smmu_atc_inv_to_cmd(0, 0, 0, &cmd); + arm_smmu_atc_inv_to_cmd(ssid, 0, 0, &cmd); for (i = 0; i < master->num_sids; i++) { cmd.atc.sid = master->sids[i]; @@ -2955,7 +2969,7 @@ static void arm_smmu_disable_ats(struct arm_smmu_master *master) * ATC invalidation via the SMMU. */ wmb(); - arm_smmu_atc_inv_master(master); + arm_smmu_atc_inv_master(master, 0); atomic_dec(&smmu_domain->nr_ats_masters); } @@ -3152,6 +3166,16 @@ arm_smmu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova) return ops->iova_to_phys(ops, iova); } +static void arm_smmu_mm_invalidate_range(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); + + arm_smmu_atc_inv_domain(smmu_mn->domain, mm->pasid, start, + end - start + 1); +} + static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); @@ -3170,6 +3194,7 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, &invalid_cd); arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); + arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, 0, 0); smmu_mn->cleared = true; mutex_unlock(&sva_lock); @@ -3181,6 +3206,7 @@ static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn) } static struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = { + .invalidate_range = arm_smmu_mm_invalidate_range, .release = arm_smmu_mm_release, .free_notifier = arm_smmu_mmu_notifier_free, }; @@ -3257,8 +3283,10 @@ static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) * If we went through clear(), we've already invalidated, and no * new TLB entry can have been formed. */ - if (!smmu_mn->cleared) + if (!smmu_mn->cleared) { arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid); + arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, 0, 0); + } /* Frees smmu_mn */ mmu_notifier_put(&smmu_mn->mn);