From patchwork Thu Nov 12 12:55:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11900167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B2B6C2D0A3 for ; Thu, 12 Nov 2020 13:04:57 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8B0E221D7F for ; Thu, 12 Nov 2020 13:04:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="1EUhDRAq"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="PFcVzw9c" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8B0E221D7F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=+cXrsQC7qtbWPdQmIPxiNRSN75LCZBMVI2DI2J195i4=; b=1EUhDRAqYnPiu0MF8fvTqnB6I 8ug8BQQn0TJkkWBHfqynXNJJ0hCZ1I2iUdQRB5fpUbPMz8QUpEOWR6c7YmpkVgRIAHs8lkjSnAdpC KehOiSymud3HBSZuxaOqC/S3b/+QrUk38LP56VFS9KMUT7G2qjAL0ZPgHEqRfsKMv8HZYE/GMETO3 oBluSb0KXsdMW7iETCP/IuYA48iixtQrgoEZN531IHXBJY3ItubPPcEOKw0vsEFRDQEIrmfqu61S+ Ty+OHCxsy9CuLFYZPmOlP/lwPX6YhIAuQHZcR2tn8QSkplagd4Cj2vJ+w9s1lPvMHTyScDbSrnV/c Zrv9+O6Bw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCG8-0006bi-O0; Thu, 12 Nov 2020 13:03:29 +0000 Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCFx-0006WY-DY for linux-arm-kernel@lists.infradead.org; Thu, 12 Nov 2020 13:03:19 +0000 Received: by mail-wr1-x442.google.com with SMTP id 33so5908187wrl.7 for ; Thu, 12 Nov 2020 05:03:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xTMMX1Pvjrqo3g92TeWy/MVRmSEOT3mZNLGRmeCzu6U=; b=PFcVzw9c4IleobHYPZy7OVt9MV2fDkJIx8ArG94I9OiXBXpTpGTx2Joy/6NaaXrC1S lsEhim/0YR672yuxYcWchojQRZLyMo+iZTmf1HWCy5fatUgAY4fuiuxW5rnJualsqLN8 cX2ysyiSAk3vo0jUTLoLXQef+ZExMHIZ0NoQWtCMKot8fyZRM60Obc7nL7Rl0pqZkonx Xp/jrSId+YmykyRMeu1UqqizhN4hbEMbhJoLGyU1SsATG9rmIzDKZ3k+XmL6yvLocYKE XofSoudFCCtckSvPTpkHM4z3yDENT2NoeLjUYY8WCrdwsRANwBLubfZm5gs3ryp2wjyj lB1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xTMMX1Pvjrqo3g92TeWy/MVRmSEOT3mZNLGRmeCzu6U=; b=Nw7ow58kbwkWtqo1nap5LliUSD75GhzfFPfqZ+vdHiwe6NLuq0hwTmWyk3u4XDyrsn ZCEWSgquSoAYzXRnM+4EMRxIuC8tA0VHe9+f3nx7iQFflEgOHo4q0oG9NrwGpOfmbYWN BdPi431tEwSq4U9Ubp7Svipldg1l/h+zuDcUd00jeLwOVc5UqRTRwZ1LOOceh0r1G7sL Pz1UH+vwnmGTtxRQKk9TvL4n+jCculNVdKXPAKF1Okb2lrGcE0VpPjEWaBmEsioJ+c3y DU+Nwyc5JWbTtb4J6dR3NS8QKkeHm4qdeNDP8BInt/4ZvPfxQmFJPX/LUgrMm2nKq+yh NgtA== X-Gm-Message-State: AOAM532DGRTOT9s5hN9agMCybozprOjMJkRsUXLhfZN3nmwBaj5b8pzS dv4HTfRfdbGHo1xlLBhf7Ql6xA== X-Google-Smtp-Source: ABdhPJypOx1HiaTsV8rMGWkXtsNLkHrVjA73iQH1IZTsobU8wOLnaiQQzt0FC5ibavqrYHy3Jw6GpA== X-Received: by 2002:adf:fa82:: with SMTP id h2mr14646338wrr.24.1605186195453; Thu, 12 Nov 2020 05:03:15 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id m22sm6877508wrb.97.2020.11.12.05.03.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Nov 2020 05:03:14 -0800 (PST) From: Jean-Philippe Brucker To: joro@8bytes.org, will@kernel.org, lorenzo.pieralisi@arm.com, robh+dt@kernel.org Subject: [PATCH v8 1/9] iommu: Add a page fault handler Date: Thu, 12 Nov 2020 13:55:13 +0100 Message-Id: <20201112125519.3987595-2-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201112125519.3987595-1-jean-philippe@linaro.org> References: <20201112125519.3987595-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201112_080317_540100_A7F3E6FA X-CRM114-Status: GOOD ( 35.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: devicetree@vger.kernel.org, linux-acpi@vger.kernel.org, linux-pci@vger.kernel.org, guohanjun@huawei.com, rjw@rjwysocki.net, shameerali.kolothum.thodi@huawei.com, eric.auger@redhat.com, iommu@lists.linux-foundation.org, Jonathan.Cameron@huawei.com, sudeep.holla@arm.com, bhelgaas@google.com, zhangfei.gao@linaro.org, vivek.gautam@arm.com, baolu.lu@linux.intel.com, robin.murphy@arm.com, Jean-Philippe Brucker , linux-arm-kernel@lists.infradead.org, lenb@kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Some systems allow devices to handle I/O Page Faults in the core mm. For example systems implementing the PCIe PRI extension or Arm SMMU stall model. Infrastructure for reporting these recoverable page faults was added to the IOMMU core by commit 0c830e6b3282 ("iommu: Introduce device fault report API"). Add a page fault handler for host SVA. IOMMU driver can now instantiate several fault workqueues and link them to IOPF-capable devices. Drivers can choose between a single global workqueue, one per IOMMU device, one per low-level fault queue, one per domain, etc. When it receives a fault event, supposedly in an IRQ handler, the IOMMU driver reports the fault using iommu_report_device_fault(), which calls the registered handler. The page fault handler then calls the mm fault handler, and reports either success or failure with iommu_page_response(). When the handler succeeded, the IOMMU retries the access. The iopf_param pointer could be embedded into iommu_fault_param. But putting iopf_param into the iommu_param structure allows us not to care about ordering between calls to iopf_queue_add_device() and iommu_register_device_fault_handler(). Signed-off-by: Jean-Philippe Brucker --- v8: * Re-use CONFIG_IOMMU_SVA_LIB and move definitions to iommu-sva-lib.h, since this is an API internal to IOMMU drivers. * Fix typos. --- drivers/iommu/Makefile | 1 + drivers/iommu/iommu-sva-lib.h | 53 ++++ include/linux/iommu.h | 2 + drivers/iommu/io-pgfault.c | 462 ++++++++++++++++++++++++++++++++++ 4 files changed, 518 insertions(+) create mode 100644 drivers/iommu/io-pgfault.c diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile index 61bd30cd8369..60fafc23dee6 100644 --- a/drivers/iommu/Makefile +++ b/drivers/iommu/Makefile @@ -28,3 +28,4 @@ obj-$(CONFIG_S390_IOMMU) += s390-iommu.o obj-$(CONFIG_HYPERV_IOMMU) += hyperv-iommu.o obj-$(CONFIG_VIRTIO_IOMMU) += virtio-iommu.o obj-$(CONFIG_IOMMU_SVA_LIB) += iommu-sva-lib.o +obj-$(CONFIG_IOMMU_SVA_LIB) += io-pgfault.o diff --git a/drivers/iommu/iommu-sva-lib.h b/drivers/iommu/iommu-sva-lib.h index b40990aef3fd..031155010ca8 100644 --- a/drivers/iommu/iommu-sva-lib.h +++ b/drivers/iommu/iommu-sva-lib.h @@ -12,4 +12,57 @@ int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max); void iommu_sva_free_pasid(struct mm_struct *mm); struct mm_struct *iommu_sva_find(ioasid_t pasid); +/* I/O Page fault */ +struct device; +struct iommu_fault; +struct iopf_queue; + +#ifdef CONFIG_IOMMU_SVA_LIB +int iommu_queue_iopf(struct iommu_fault *fault, void *cookie); + +int iopf_queue_add_device(struct iopf_queue *queue, struct device *dev); +int iopf_queue_remove_device(struct iopf_queue *queue, + struct device *dev); +int iopf_queue_flush_dev(struct device *dev); +struct iopf_queue *iopf_queue_alloc(const char *name); +void iopf_queue_free(struct iopf_queue *queue); +int iopf_queue_discard_partial(struct iopf_queue *queue); + +#else /* CONFIG_IOMMU_SVA_LIB */ +static inline int iommu_queue_iopf(struct iommu_fault *fault, void *cookie) +{ + return -ENODEV; +} + +static inline int iopf_queue_add_device(struct iopf_queue *queue, + struct device *dev) +{ + return -ENODEV; +} + +static inline int iopf_queue_remove_device(struct iopf_queue *queue, + struct device *dev) +{ + return -ENODEV; +} + +static inline int iopf_queue_flush_dev(struct device *dev) +{ + return -ENODEV; +} + +static inline struct iopf_queue *iopf_queue_alloc(const char *name) +{ + return NULL; +} + +static inline void iopf_queue_free(struct iopf_queue *queue) +{ +} + +static inline int iopf_queue_discard_partial(struct iopf_queue *queue) +{ + return -ENODEV; +} +#endif /* CONFIG_IOMMU_SVA_LIB */ #endif /* _IOMMU_SVA_LIB_H */ diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 047480a19997..a1c78c4cdeb1 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -350,6 +350,7 @@ struct iommu_fault_param { * struct dev_iommu - Collection of per-device IOMMU data * * @fault_param: IOMMU detected device fault reporting data + * @iopf_param: I/O Page Fault queue and data * @fwspec: IOMMU fwspec data * @iommu_dev: IOMMU device this device is linked to * @priv: IOMMU Driver private data @@ -360,6 +361,7 @@ struct iommu_fault_param { struct dev_iommu { struct mutex lock; struct iommu_fault_param *fault_param; + struct iopf_device_param *iopf_param; struct iommu_fwspec *fwspec; struct iommu_device *iommu_dev; void *priv; diff --git a/drivers/iommu/io-pgfault.c b/drivers/iommu/io-pgfault.c new file mode 100644 index 000000000000..fc1d5d29ac37 --- /dev/null +++ b/drivers/iommu/io-pgfault.c @@ -0,0 +1,462 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Handle device page faults + * + * Copyright (C) 2020 ARM Ltd. + */ + +#include +#include +#include +#include +#include + +#include "iommu-sva-lib.h" + +/** + * struct iopf_queue - IO Page Fault queue + * @wq: the fault workqueue + * @devices: devices attached to this queue + * @lock: protects the device list + */ +struct iopf_queue { + struct workqueue_struct *wq; + struct list_head devices; + struct mutex lock; +}; + +/** + * struct iopf_device_param - IO Page Fault data attached to a device + * @dev: the device that owns this param + * @queue: IOPF queue + * @queue_list: index into queue->devices + * @partial: faults that are part of a Page Request Group for which the last + * request hasn't been submitted yet. + */ +struct iopf_device_param { + struct device *dev; + struct iopf_queue *queue; + struct list_head queue_list; + struct list_head partial; +}; + +struct iopf_fault { + struct iommu_fault fault; + struct list_head list; +}; + +struct iopf_group { + struct iopf_fault last_fault; + struct list_head faults; + struct work_struct work; + struct device *dev; +}; + +static int iopf_complete_group(struct device *dev, struct iopf_fault *iopf, + enum iommu_page_response_code status) +{ + struct iommu_page_response resp = { + .version = IOMMU_PAGE_RESP_VERSION_1, + .pasid = iopf->fault.prm.pasid, + .grpid = iopf->fault.prm.grpid, + .code = status, + }; + + if ((iopf->fault.prm.flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID) && + (iopf->fault.prm.flags & IOMMU_FAULT_PAGE_RESPONSE_NEEDS_PASID)) + resp.flags = IOMMU_PAGE_RESP_PASID_VALID; + + return iommu_page_response(dev, &resp); +} + +static enum iommu_page_response_code +iopf_handle_single(struct iopf_fault *iopf) +{ + vm_fault_t ret; + struct mm_struct *mm; + struct vm_area_struct *vma; + unsigned int access_flags = 0; + unsigned int fault_flags = FAULT_FLAG_REMOTE; + struct iommu_fault_page_request *prm = &iopf->fault.prm; + enum iommu_page_response_code status = IOMMU_PAGE_RESP_INVALID; + + if (!(prm->flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID)) + return status; + + mm = iommu_sva_find(prm->pasid); + if (IS_ERR_OR_NULL(mm)) + return status; + + mmap_read_lock(mm); + + vma = find_extend_vma(mm, prm->addr); + if (!vma) + /* Unmapped area */ + goto out_put_mm; + + if (prm->perm & IOMMU_FAULT_PERM_READ) + access_flags |= VM_READ; + + if (prm->perm & IOMMU_FAULT_PERM_WRITE) { + access_flags |= VM_WRITE; + fault_flags |= FAULT_FLAG_WRITE; + } + + if (prm->perm & IOMMU_FAULT_PERM_EXEC) { + access_flags |= VM_EXEC; + fault_flags |= FAULT_FLAG_INSTRUCTION; + } + + if (!(prm->perm & IOMMU_FAULT_PERM_PRIV)) + fault_flags |= FAULT_FLAG_USER; + + if (access_flags & ~vma->vm_flags) + /* Access fault */ + goto out_put_mm; + + ret = handle_mm_fault(vma, prm->addr, fault_flags, NULL); + status = ret & VM_FAULT_ERROR ? IOMMU_PAGE_RESP_INVALID : + IOMMU_PAGE_RESP_SUCCESS; + +out_put_mm: + mmap_read_unlock(mm); + mmput(mm); + + return status; +} + +static void iopf_handle_group(struct work_struct *work) +{ + struct iopf_group *group; + struct iopf_fault *iopf, *next; + enum iommu_page_response_code status = IOMMU_PAGE_RESP_SUCCESS; + + group = container_of(work, struct iopf_group, work); + + list_for_each_entry_safe(iopf, next, &group->faults, list) { + /* + * For the moment, errors are sticky: don't handle subsequent + * faults in the group if there is an error. + */ + if (status == IOMMU_PAGE_RESP_SUCCESS) + status = iopf_handle_single(iopf); + + if (!(iopf->fault.prm.flags & + IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE)) + kfree(iopf); + } + + iopf_complete_group(group->dev, &group->last_fault, status); + kfree(group); +} + +/** + * iommu_queue_iopf - IO Page Fault handler + * @fault: fault event + * @cookie: struct device, passed to iommu_register_device_fault_handler. + * + * Add a fault to the device workqueue, to be handled by mm. + * + * This module doesn't handle PCI PASID Stop Marker; IOMMU drivers must discard + * them before reporting faults. A PASID Stop Marker (LRW = 0b100) doesn't + * expect a response. It may be generated when disabling a PASID (issuing a + * PASID stop request) by some PCI devices. + * + * The PASID stop request is issued by the device driver before unbind(). Once + * it completes, no page request is generated for this PASID anymore and + * outstanding ones have been pushed to the IOMMU (as per PCIe 4.0r1.0 - 6.20.1 + * and 10.4.1.2 - Managing PASID TLP Prefix Usage). Some PCI devices will wait + * for all outstanding page requests to come back with a response before + * completing the PASID stop request. Others do not wait for page responses, and + * instead issue this Stop Marker that tells us when the PASID can be + * reallocated. + * + * It is safe to discard the Stop Marker because it is an optimization. + * a. Page requests, which are posted requests, have been flushed to the IOMMU + * when the stop request completes. + * b. The IOMMU driver flushes all fault queues on unbind() before freeing the + * PASID. + * + * So even though the Stop Marker might be issued by the device *after* the stop + * request completes, outstanding faults will have been dealt with by the time + * the PASID is freed. + * + * Return: 0 on success and <0 on error. + */ +int iommu_queue_iopf(struct iommu_fault *fault, void *cookie) +{ + int ret; + struct iopf_group *group; + struct iopf_fault *iopf, *next; + struct iopf_device_param *iopf_param; + + struct device *dev = cookie; + struct dev_iommu *param = dev->iommu; + + lockdep_assert_held(¶m->lock); + + if (fault->type != IOMMU_FAULT_PAGE_REQ) + /* Not a recoverable page fault */ + return -EOPNOTSUPP; + + /* + * As long as we're holding param->lock, the queue can't be unlinked + * from the device and therefore cannot disappear. + */ + iopf_param = param->iopf_param; + if (!iopf_param) + return -ENODEV; + + if (!(fault->prm.flags & IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE)) { + iopf = kzalloc(sizeof(*iopf), GFP_KERNEL); + if (!iopf) + return -ENOMEM; + + iopf->fault = *fault; + + /* Non-last request of a group. Postpone until the last one */ + list_add(&iopf->list, &iopf_param->partial); + + return 0; + } + + group = kzalloc(sizeof(*group), GFP_KERNEL); + if (!group) { + /* + * The caller will send a response to the hardware. But we do + * need to clean up before leaving, otherwise partial faults + * will be stuck. + */ + ret = -ENOMEM; + goto cleanup_partial; + } + + group->dev = dev; + group->last_fault.fault = *fault; + INIT_LIST_HEAD(&group->faults); + list_add(&group->last_fault.list, &group->faults); + INIT_WORK(&group->work, iopf_handle_group); + + /* See if we have partial faults for this group */ + list_for_each_entry_safe(iopf, next, &iopf_param->partial, list) { + if (iopf->fault.prm.grpid == fault->prm.grpid) + /* Insert *before* the last fault */ + list_move(&iopf->list, &group->faults); + } + + queue_work(iopf_param->queue->wq, &group->work); + return 0; + +cleanup_partial: + list_for_each_entry_safe(iopf, next, &iopf_param->partial, list) { + if (iopf->fault.prm.grpid == fault->prm.grpid) { + list_del(&iopf->list); + kfree(iopf); + } + } + return ret; +} +EXPORT_SYMBOL_GPL(iommu_queue_iopf); + +/** + * iopf_queue_flush_dev - Ensure that all queued faults have been processed + * @dev: the endpoint whose faults need to be flushed. + * + * The IOMMU driver calls this before releasing a PASID, to ensure that all + * pending faults for this PASID have been handled, and won't hit the address + * space of the next process that uses this PASID. The driver must make sure + * that no new fault is added to the queue. In particular it must flush its + * low-level queue before calling this function. + * + * Return: 0 on success and <0 on error. + */ +int iopf_queue_flush_dev(struct device *dev) +{ + int ret = 0; + struct iopf_device_param *iopf_param; + struct dev_iommu *param = dev->iommu; + + if (!param) + return -ENODEV; + + mutex_lock(¶m->lock); + iopf_param = param->iopf_param; + if (iopf_param) + flush_workqueue(iopf_param->queue->wq); + else + ret = -ENODEV; + mutex_unlock(¶m->lock); + + return ret; +} +EXPORT_SYMBOL_GPL(iopf_queue_flush_dev); + +/** + * iopf_queue_discard_partial - Remove all pending partial fault + * @queue: the queue whose partial faults need to be discarded + * + * When the hardware queue overflows, last page faults in a group may have been + * lost and the IOMMU driver calls this to discard all partial faults. The + * driver shouldn't be adding new faults to this queue concurrently. + * + * Return: 0 on success and <0 on error. + */ +int iopf_queue_discard_partial(struct iopf_queue *queue) +{ + struct iopf_fault *iopf, *next; + struct iopf_device_param *iopf_param; + + if (!queue) + return -EINVAL; + + mutex_lock(&queue->lock); + list_for_each_entry(iopf_param, &queue->devices, queue_list) { + list_for_each_entry_safe(iopf, next, &iopf_param->partial, + list) { + list_del(&iopf->list); + kfree(iopf); + } + } + mutex_unlock(&queue->lock); + return 0; +} +EXPORT_SYMBOL_GPL(iopf_queue_discard_partial); + +/** + * iopf_queue_add_device - Add producer to the fault queue + * @queue: IOPF queue + * @dev: device to add + * + * Return: 0 on success and <0 on error. + */ +int iopf_queue_add_device(struct iopf_queue *queue, struct device *dev) +{ + int ret = -EBUSY; + struct iopf_device_param *iopf_param; + struct dev_iommu *param = dev->iommu; + + if (!param) + return -ENODEV; + + iopf_param = kzalloc(sizeof(*iopf_param), GFP_KERNEL); + if (!iopf_param) + return -ENOMEM; + + INIT_LIST_HEAD(&iopf_param->partial); + iopf_param->queue = queue; + iopf_param->dev = dev; + + mutex_lock(&queue->lock); + mutex_lock(¶m->lock); + if (!param->iopf_param) { + list_add(&iopf_param->queue_list, &queue->devices); + param->iopf_param = iopf_param; + ret = 0; + } + mutex_unlock(¶m->lock); + mutex_unlock(&queue->lock); + + if (ret) + kfree(iopf_param); + + return ret; +} +EXPORT_SYMBOL_GPL(iopf_queue_add_device); + +/** + * iopf_queue_remove_device - Remove producer from fault queue + * @queue: IOPF queue + * @dev: device to remove + * + * Caller makes sure that no more faults are reported for this device. + * + * Return: 0 on success and <0 on error. + */ +int iopf_queue_remove_device(struct iopf_queue *queue, struct device *dev) +{ + int ret = 0; + struct iopf_fault *iopf, *next; + struct iopf_device_param *iopf_param; + struct dev_iommu *param = dev->iommu; + + if (!param || !queue) + return -EINVAL; + + mutex_lock(&queue->lock); + mutex_lock(¶m->lock); + iopf_param = param->iopf_param; + if (iopf_param && iopf_param->queue == queue) { + list_del(&iopf_param->queue_list); + param->iopf_param = NULL; + } else { + ret = -EINVAL; + } + mutex_unlock(¶m->lock); + mutex_unlock(&queue->lock); + if (ret) + return ret; + + /* Just in case some faults are still stuck */ + list_for_each_entry_safe(iopf, next, &iopf_param->partial, list) + kfree(iopf); + + kfree(iopf_param); + + return 0; +} +EXPORT_SYMBOL_GPL(iopf_queue_remove_device); + +/** + * iopf_queue_alloc - Allocate and initialize a fault queue + * @name: a unique string identifying the queue (for workqueue) + * + * Return: the queue on success and NULL on error. + */ +struct iopf_queue *iopf_queue_alloc(const char *name) +{ + struct iopf_queue *queue; + + queue = kzalloc(sizeof(*queue), GFP_KERNEL); + if (!queue) + return NULL; + + /* + * The WQ is unordered because the low-level handler enqueues faults by + * group. PRI requests within a group have to be ordered, but once + * that's dealt with, the high-level function can handle groups out of + * order. + */ + queue->wq = alloc_workqueue("iopf_queue/%s", WQ_UNBOUND, 0, name); + if (!queue->wq) { + kfree(queue); + return NULL; + } + + INIT_LIST_HEAD(&queue->devices); + mutex_init(&queue->lock); + + return queue; +} +EXPORT_SYMBOL_GPL(iopf_queue_alloc); + +/** + * iopf_queue_free - Free IOPF queue + * @queue: queue to free + * + * Counterpart to iopf_queue_alloc(). The driver must not be queuing faults or + * adding/removing devices on this queue anymore. + */ +void iopf_queue_free(struct iopf_queue *queue) +{ + struct iopf_device_param *iopf_param, *next; + + if (!queue) + return; + + list_for_each_entry_safe(iopf_param, next, &queue->devices, queue_list) + iopf_queue_remove_device(queue, iopf_param->dev); + + destroy_workqueue(queue->wq); + kfree(queue); +} +EXPORT_SYMBOL_GPL(iopf_queue_free); From patchwork Thu Nov 12 12:55:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11900133 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F20EBC2D0A3 for ; Thu, 12 Nov 2020 13:04:14 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 670B021D91 for ; Thu, 12 Nov 2020 13:04:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="nkzIpRtv"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="Dn59G2jx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 670B021D91 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=UK/NgWCIaDhyhtK6oMz9Jc02ON99ODi3U87sVQWZr00=; b=nkzIpRtveGOxhLvXIcc+T8EpU TgIIQZ0xpv+03IaWIlFOG+6ccBkeH8lC8Juvfy8PZZS8d7Nc7q9ZG0fGRzPuTfkswLBG/ziLD61mz 5Sv1Bx7A5gQQezZTCxJgB0jV/DfMay/warsY4HYpSpInDq1CI6k2pudDLNSKoZfeiwRP/M1qWnFl+ y3AvcRK95bXSAtNnASmBxyY1QWR+rJpeV+1lujzo+IBCn97utrlZ3muxMY3qO/msFgtQoshNL9V0k 4orGroKYpSPyw+77PRsnWbWO514AZFIHUWDOIHqe7Jx5tNc6c+zibuQ2cLOIjBjQEa3alwRYM1Yu8 CVfX+E9ww==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCGS-0006jg-B8; Thu, 12 Nov 2020 13:03:48 +0000 Received: from mail-wr1-x443.google.com ([2a00:1450:4864:20::443]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCFy-0006XH-2k for linux-arm-kernel@lists.infradead.org; Thu, 12 Nov 2020 13:03:19 +0000 Received: by mail-wr1-x443.google.com with SMTP id b6so5921482wrt.4 for ; Thu, 12 Nov 2020 05:03:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ct/A3+BqtZ1mh7NbSkQbTndKvraebCJIuStSjYZJj6w=; b=Dn59G2jxCJpXQrWrRlrml739Y0UGynYN7lH9lyw+1yaOF++vjgKKIvvlKYEGsBjX5C 7bdJR7rJAuBNA6tM+LlGlnNHHmCoaTyD4NWrpKiOjUIhCkls45V8Se7r6SFaJcMeKREy ODKuG3eMApLS+UrxG7p2Rcg/bO3sqLM8QwGCYaFJO+3NlqESiIm3vY68Cvs5nj2xrMeD dlNhxa1IyOUWrAQ7Uo1BnZ0Tj6HXA5JKm9KJDaeeMJSSnG8yPwAA3+IUcziyR/s3DN3v nlgfSNdjoxc/NeIWDE7/RF1ZTAOZSxAbW5VF+X0d1jOT8pO+zEOA1bjRrngz4uSQbQBP YIhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ct/A3+BqtZ1mh7NbSkQbTndKvraebCJIuStSjYZJj6w=; b=iXsZUiDY8oYFylgBHeZCIMuW6hBUDYksC3X5XZ9TUObNS3RVAOpJl7CmQ6JvOlkM5y MiiZDLHWCggCGuWVU2ZNfL4I2ERybuhXH2IEQaUFZCn9F7sajZuOlLm27a6ThT/HgyHc WfoYcLR+qN0PnO+UKsOCwETBvWQq/H+sgHMOlUADwQEBvcd74shJVoA1WuFp9Sg+TGLq 5GAy1/FWFyrNw71lPPBLQE4DGEAVlEiIUtl2X5kVCVyU/654JDvFitU0/fJcetG0R26c dpeczTrHmY9QaOpJqHPTJkMKA+G/ZvU919ZAz7658S0jT3Lo5b+mrpNsCPtmkD5M5dXb 8gDQ== X-Gm-Message-State: AOAM533dlC0RUrRFCx3XvRxTIScObiaKEFQZ8gehKm9O+TYfSFmaYFLr ckk65HuiZJt5HFRRNY9OmtmQxQ== X-Google-Smtp-Source: ABdhPJyxBTftn6g9SekhMYf7fGQS110g7UZOp/jgF7KzIYPddsTdLxbRGQWK9hsJJjBj65Z7/hzkFQ== X-Received: by 2002:adf:eb47:: with SMTP id u7mr35100618wrn.163.1605186197169; Thu, 12 Nov 2020 05:03:17 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id m22sm6877508wrb.97.2020.11.12.05.03.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Nov 2020 05:03:16 -0800 (PST) From: Jean-Philippe Brucker To: joro@8bytes.org, will@kernel.org, lorenzo.pieralisi@arm.com, robh+dt@kernel.org Subject: [PATCH v8 2/9] iommu/arm-smmu-v3: Maintain a SID->device structure Date: Thu, 12 Nov 2020 13:55:14 +0100 Message-Id: <20201112125519.3987595-3-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201112125519.3987595-1-jean-philippe@linaro.org> References: <20201112125519.3987595-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201112_080318_194877_77A56D68 X-CRM114-Status: GOOD ( 24.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: devicetree@vger.kernel.org, linux-acpi@vger.kernel.org, linux-pci@vger.kernel.org, guohanjun@huawei.com, rjw@rjwysocki.net, shameerali.kolothum.thodi@huawei.com, eric.auger@redhat.com, iommu@lists.linux-foundation.org, Jonathan.Cameron@huawei.com, sudeep.holla@arm.com, bhelgaas@google.com, zhangfei.gao@linaro.org, vivek.gautam@arm.com, baolu.lu@linux.intel.com, robin.murphy@arm.com, Jean-Philippe Brucker , linux-arm-kernel@lists.infradead.org, lenb@kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When handling faults from the event or PRI queue, we need to find the struct device associated to a SID. Add a rb_tree to keep track of SIDs. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 13 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 161 ++++++++++++++++---- 2 files changed, 144 insertions(+), 30 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 58868a5677b6..7bd98fdce5c3 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -648,6 +648,15 @@ struct arm_smmu_device { /* IOMMU core code handle */ struct iommu_device iommu; + + struct rb_root streams; + struct mutex streams_mutex; +}; + +struct arm_smmu_stream { + u32 id; + struct arm_smmu_master *master; + struct rb_node node; }; /* SMMU private data for each master */ @@ -656,8 +665,8 @@ struct arm_smmu_master { struct device *dev; struct arm_smmu_domain *domain; struct list_head domain_head; - u32 *sids; - unsigned int num_sids; + struct arm_smmu_stream *streams; + unsigned int num_streams; bool ats_enabled; bool sva_enabled; struct list_head bonds; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 453478c83933..d87c87136d63 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -918,8 +918,8 @@ static void arm_smmu_sync_cd(struct arm_smmu_domain *smmu_domain, spin_lock_irqsave(&smmu_domain->devices_lock, flags); list_for_each_entry(master, &smmu_domain->devices, domain_head) { - for (i = 0; i < master->num_sids; i++) { - cmd.cfgi.sid = master->sids[i]; + for (i = 0; i < master->num_streams; i++) { + cmd.cfgi.sid = master->streams[i].id; arm_smmu_cmdq_batch_add(smmu, &cmds, &cmd); } } @@ -1371,6 +1371,32 @@ static int arm_smmu_init_l2_strtab(struct arm_smmu_device *smmu, u32 sid) return 0; } +__maybe_unused +static struct arm_smmu_master * +arm_smmu_find_master(struct arm_smmu_device *smmu, u32 sid) +{ + struct rb_node *node; + struct arm_smmu_stream *stream; + struct arm_smmu_master *master = NULL; + + mutex_lock(&smmu->streams_mutex); + node = smmu->streams.rb_node; + while (node) { + stream = rb_entry(node, struct arm_smmu_stream, node); + if (stream->id < sid) { + node = node->rb_right; + } else if (stream->id > sid) { + node = node->rb_left; + } else { + master = stream->master; + break; + } + } + mutex_unlock(&smmu->streams_mutex); + + return master; +} + /* IRQ and event handlers */ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) { @@ -1604,8 +1630,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master) arm_smmu_atc_inv_to_cmd(0, 0, 0, &cmd); - for (i = 0; i < master->num_sids; i++) { - cmd.atc.sid = master->sids[i]; + for (i = 0; i < master->num_streams; i++) { + cmd.atc.sid = master->streams[i].id; arm_smmu_cmdq_issue_cmd(master->smmu, &cmd); } @@ -1648,8 +1674,8 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, if (!master->ats_enabled) continue; - for (i = 0; i < master->num_sids; i++) { - cmd.atc.sid = master->sids[i]; + for (i = 0; i < master->num_streams; i++) { + cmd.atc.sid = master->streams[i].id; arm_smmu_cmdq_batch_add(smmu_domain->smmu, &cmds, &cmd); } } @@ -2064,13 +2090,13 @@ static void arm_smmu_install_ste_for_dev(struct arm_smmu_master *master) int i, j; struct arm_smmu_device *smmu = master->smmu; - for (i = 0; i < master->num_sids; ++i) { - u32 sid = master->sids[i]; + for (i = 0; i < master->num_streams; ++i) { + u32 sid = master->streams[i].id; __le64 *step = arm_smmu_get_step_for_sid(smmu, sid); /* Bridged PCI devices may end up with duplicated IDs */ for (j = 0; j < i; j++) - if (master->sids[j] == sid) + if (master->streams[j].id == sid) break; if (j < i) continue; @@ -2339,11 +2365,101 @@ static bool arm_smmu_sid_in_range(struct arm_smmu_device *smmu, u32 sid) return sid < limit; } +static int arm_smmu_insert_master(struct arm_smmu_device *smmu, + struct arm_smmu_master *master) +{ + int i; + int ret = 0; + struct arm_smmu_stream *new_stream, *cur_stream; + struct rb_node **new_node, *parent_node = NULL; + struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(master->dev); + + master->streams = kcalloc(fwspec->num_ids, + sizeof(struct arm_smmu_stream), GFP_KERNEL); + if (!master->streams) + return -ENOMEM; + master->num_streams = fwspec->num_ids; + + mutex_lock(&smmu->streams_mutex); + for (i = 0; i < fwspec->num_ids && !ret; i++) { + u32 sid = fwspec->ids[i]; + + new_stream = &master->streams[i]; + new_stream->id = sid; + new_stream->master = master; + + /* + * Check the SIDs are in range of the SMMU and our stream table + */ + if (!arm_smmu_sid_in_range(smmu, sid)) { + ret = -ERANGE; + break; + } + + /* Ensure l2 strtab is initialised */ + if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) { + ret = arm_smmu_init_l2_strtab(smmu, sid); + if (ret) + break; + } + + /* Insert into SID tree */ + new_node = &(smmu->streams.rb_node); + while (*new_node) { + cur_stream = rb_entry(*new_node, struct arm_smmu_stream, + node); + parent_node = *new_node; + if (cur_stream->id > new_stream->id) { + new_node = &((*new_node)->rb_left); + } else if (cur_stream->id < new_stream->id) { + new_node = &((*new_node)->rb_right); + } else { + dev_warn(master->dev, + "stream %u already in tree\n", + cur_stream->id); + ret = -EINVAL; + break; + } + } + + if (!ret) { + rb_link_node(&new_stream->node, parent_node, new_node); + rb_insert_color(&new_stream->node, &smmu->streams); + } + } + + if (ret) { + for (; i > 0; i--) + rb_erase(&master->streams[i].node, &smmu->streams); + kfree(master->streams); + } + mutex_unlock(&smmu->streams_mutex); + + return ret; +} + +static void arm_smmu_remove_master(struct arm_smmu_master *master) +{ + int i; + struct arm_smmu_device *smmu = master->smmu; + struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(master->dev); + + if (!smmu || !master->streams) + return; + + mutex_lock(&smmu->streams_mutex); + for (i = 0; i < fwspec->num_ids; i++) + rb_erase(&master->streams[i].node, &smmu->streams); + mutex_unlock(&smmu->streams_mutex); + + kfree(master->streams); +} + static struct iommu_ops arm_smmu_ops; static struct iommu_device *arm_smmu_probe_device(struct device *dev) { - int i, ret; + int ret; struct arm_smmu_device *smmu; struct arm_smmu_master *master; struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); @@ -2364,27 +2480,12 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) master->dev = dev; master->smmu = smmu; - master->sids = fwspec->ids; - master->num_sids = fwspec->num_ids; INIT_LIST_HEAD(&master->bonds); dev_iommu_priv_set(dev, master); - /* Check the SIDs are in range of the SMMU and our stream table */ - for (i = 0; i < master->num_sids; i++) { - u32 sid = master->sids[i]; - - if (!arm_smmu_sid_in_range(smmu, sid)) { - ret = -ERANGE; - goto err_free_master; - } - - /* Ensure l2 strtab is initialised */ - if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) { - ret = arm_smmu_init_l2_strtab(smmu, sid); - if (ret) - goto err_free_master; - } - } + ret = arm_smmu_insert_master(smmu, master); + if (ret) + goto err_free_master; master->ssid_bits = min(smmu->ssid_bits, fwspec->num_pasid_bits); @@ -2422,6 +2523,7 @@ static void arm_smmu_release_device(struct device *dev) WARN_ON(arm_smmu_master_sva_enabled(master)); arm_smmu_detach_dev(master); arm_smmu_disable_pasid(master); + arm_smmu_remove_master(master); kfree(master); iommu_fwspec_free(dev); } @@ -2844,6 +2946,9 @@ static int arm_smmu_init_structures(struct arm_smmu_device *smmu) { int ret; + mutex_init(&smmu->streams_mutex); + smmu->streams = RB_ROOT; + ret = arm_smmu_init_queues(smmu); if (ret) return ret; From patchwork Thu Nov 12 12:55:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11900131 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D4D8C388F7 for ; Thu, 12 Nov 2020 13:04:18 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0E78921D91 for ; Thu, 12 Nov 2020 13:04:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="oumxq+UW"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="oJG4X6CI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0E78921D91 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=vfpU8qqqJdtoIXSLowI2aD/ra15lWhK9xFCHr2D3rCY=; b=oumxq+UWojVu+oie7aZl4l5rI 5iKMdUiM26tteAQ4C1CHi37WSPPFwaj0bm3NiBpHJPk457t4HI2U1hxi5Gxf8fQZRSm2sQX6ZYC5F XGzr/kulSGwW64h5jUkcz+588ORf9hpoKrlYF7fiifwpVLvt0iIeQ4Gf4wfBSr7Pztek+1/e16Ywd IMpsr0h5bLYQIDZc0XnkV4itk0d/zhXXEPeIlaNkRs0et0iS5EwYM13N0cP3TMhfxDJejKZzdB3PS NWnrSOvzW2FrHmJvSkfu7GovnJ1H/8s23l3sw/VDap8KewchMNQ/0gs1rmIbH718xaRS9HLqoXdHo AQTlHIEgQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCGW-0006l3-9I; Thu, 12 Nov 2020 13:03:52 +0000 Received: from mail-wr1-x444.google.com ([2a00:1450:4864:20::444]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCFz-0006Xp-Iu for linux-arm-kernel@lists.infradead.org; Thu, 12 Nov 2020 13:03:20 +0000 Received: by mail-wr1-x444.google.com with SMTP id b6so5921559wrt.4 for ; Thu, 12 Nov 2020 05:03:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Tid3zbIJRKyj0Bgjxj2sf+RSwZO5YPfaeHYqLr0g5iY=; b=oJG4X6CI7enblSZUp3EgWunBFmyQv6mCwtptSww+yPpxypkzQpLOkXizEPseu9wI2t 2uZpYfOj9eUqkc5Mc5tlt3K7jx+TWvIflBQqX1SEYsvTIjIegzyCiXcmEFzhPjxvoA47 6Dq4ROrha3r07LiOgqCfR01hIVv7++Ib5t5K0yXdpav8K1LR8Zat1PWjsoKfQ809OKnH 1k5v7fX1Ma0DDtnaTPNqdNu6iefW98IR6j56uIkzclVoD8+TGJgwKYfDHt4NXx+25cGn SRyX77nEr1L/nTzdOO1Yo2Myq9+BofVGSodnPeoKBdMbtEUWw45R7NEYQRJNFh3Men13 MQBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Tid3zbIJRKyj0Bgjxj2sf+RSwZO5YPfaeHYqLr0g5iY=; b=Y6kWvGXPxtmfft0tvulJSJ4eTuH9MBBAfRToC1b86FzRugaQq6/5l6uV3x0crAQKWF 5LitW+NLJ3oIMhXXzJ5FumGnLw5AAoRRrwiPhnVb/GrVMJZQ3jsXl+CWsjuiTogju7v7 3S/fySNt5GU8Acg0dJBeNnjQMklgupemQw5GAKNNMH4u0jZzik9QO6RiVPdTJNbDAp7M NBGng4zJuvByNXyxxgtpNPjimf9cfjPhaB0cWbXDE+O9T8aJnqqY/cZ9ZoGBXj0VOALM 0uW1yLJ4el0ZW8S53T0XpX1QkQY9/WlWAhLY7i0qtaPmLgz2Cbetk5p5IPJVxIpt4rLl dGwg== X-Gm-Message-State: AOAM5321kv3/6R/j6esaWxIK8CP1g/W/ahUwi0Yh0UnAg85CldqF9jqf 9xhk1YcvEbOCNFs8tdevuacoww== X-Google-Smtp-Source: ABdhPJww5aqw0Dh36h3mNuuwHJd8ZSSnPISNulq6oQQhTrIxG8XhHZa3Ym6AO5qFBMZHmbjVR9mCBw== X-Received: by 2002:adf:df88:: with SMTP id z8mr35234477wrl.113.1605186198680; Thu, 12 Nov 2020 05:03:18 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id m22sm6877508wrb.97.2020.11.12.05.03.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Nov 2020 05:03:17 -0800 (PST) From: Jean-Philippe Brucker To: joro@8bytes.org, will@kernel.org, lorenzo.pieralisi@arm.com, robh+dt@kernel.org Subject: [PATCH v8 3/9] dt-bindings: document stall property for IOMMU masters Date: Thu, 12 Nov 2020 13:55:15 +0100 Message-Id: <20201112125519.3987595-4-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201112125519.3987595-1-jean-philippe@linaro.org> References: <20201112125519.3987595-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201112_080319_762254_99CC2A0E X-CRM114-Status: GOOD ( 14.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Rob Herring , devicetree@vger.kernel.org, linux-acpi@vger.kernel.org, linux-pci@vger.kernel.org, guohanjun@huawei.com, rjw@rjwysocki.net, shameerali.kolothum.thodi@huawei.com, eric.auger@redhat.com, iommu@lists.linux-foundation.org, Jonathan.Cameron@huawei.com, sudeep.holla@arm.com, bhelgaas@google.com, zhangfei.gao@linaro.org, vivek.gautam@arm.com, baolu.lu@linux.intel.com, robin.murphy@arm.com, Jean-Philippe Brucker , linux-arm-kernel@lists.infradead.org, lenb@kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On ARM systems, some platform devices behind an IOMMU may support stall, which is the ability to recover from page faults. Let the firmware tell us when a device supports stall. Reviewed-by: Rob Herring Signed-off-by: Jean-Philippe Brucker --- .../devicetree/bindings/iommu/iommu.txt | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/Documentation/devicetree/bindings/iommu/iommu.txt b/Documentation/devicetree/bindings/iommu/iommu.txt index 3c36334e4f94..26ba9e530f13 100644 --- a/Documentation/devicetree/bindings/iommu/iommu.txt +++ b/Documentation/devicetree/bindings/iommu/iommu.txt @@ -92,6 +92,24 @@ Optional properties: tagging DMA transactions with an address space identifier. By default, this is 0, which means that the device only has one address space. +- dma-can-stall: When present, the master can wait for a transaction to + complete for an indefinite amount of time. Upon translation fault some + IOMMUs, instead of aborting the translation immediately, may first + notify the driver and keep the transaction in flight. This allows the OS + to inspect the fault and, for example, make physical pages resident + before updating the mappings and completing the transaction. Such IOMMU + accepts a limited number of simultaneous stalled transactions before + having to either put back-pressure on the master, or abort new faulting + transactions. + + Firmware has to opt-in stalling, because most buses and masters don't + support it. In particular it isn't compatible with PCI, where + transactions have to complete before a time limit. More generally it + won't work in systems and masters that haven't been designed for + stalling. For example the OS, in order to handle a stalled transaction, + may attempt to retrieve pages from secondary storage in a stalled + domain, leading to a deadlock. + Notes: ====== From patchwork Thu Nov 12 12:55:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11900165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE240C388F7 for ; Thu, 12 Nov 2020 13:05:15 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5D38121D91 for ; Thu, 12 Nov 2020 13:05:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="r41ZEOLb"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="uA63WGlR" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5D38121D91 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=vIzpNICNK9qprrk5DLEmMycGFudo5zKCkU2HN9YvL4U=; b=r41ZEOLbVU/toa5r5CwU7qmhf 9zP6D4GwfCTk4KUuQ/Vq8PKaWYTdwrcneJ8NBAKdHNFGIjfUZIt9OYxh130C9ptSOTuErirNtwcpq E/6f5OcLMXCGKvyK+w7OxpMdl/xlEFFsSdFD56Bd2W2KfojlIxgqdDwZXCWqE1bxZ91/CeOOCdXCM DxZy+xi2H8y+Ul95znme+XbawNKfpmu4Wad55olQh59kuOc70IzGYEg9iC04/VuIIS/Usni4U1ibe Lz66mVrppgAiAnd9thjTzvf7SRsbhxZDoHzkyIaTUNeOFyzCvAlPZtNdP4l8fJpR5+P3YT8a4Ooqs PHmMaeKIw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCGY-0006m5-S8; Thu, 12 Nov 2020 13:03:54 +0000 Received: from mail-wr1-x443.google.com ([2a00:1450:4864:20::443]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCG1-0006YS-0d for linux-arm-kernel@lists.infradead.org; Thu, 12 Nov 2020 13:03:22 +0000 Received: by mail-wr1-x443.google.com with SMTP id c17so5872662wrc.11 for ; Thu, 12 Nov 2020 05:03:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3ygm1AXTLu+53w4Gri7BYEJbuhvyMFezCUVbiVOA1cI=; b=uA63WGlRDXWDF9rIZV4RZkVq6oJU37wcFuSQmD8BzO159BANMh6M+qpRZeX3/9SDUX NgjC/6kMJoth33QcDclO+ynaJlZA5DobFNRQNYs96uO5BecxpNBWMkORbZcRWO5P6qD/ BJkF/jlsL/X45r34/KxlPQbUjbKshwp4elCUfaZGnESVEwpxokGBOQy9Uvj/8ap3OCIH X4XkbC3JjCUSG2bj/0UjM+bplVf6wDID7rR7qluxsv54s4WElIx8dWnQLCnODfH4+4TN 1fQtFlyLdLdG2hhM+mcaKhRK/XdUJ1cFPfj1ecKiWJDeJr3prD0gWxlUnB4H0tf5eUhK 2bmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3ygm1AXTLu+53w4Gri7BYEJbuhvyMFezCUVbiVOA1cI=; b=AFLg2If9VCUo3WENgupKAJESSMOx71+606lGm6jdMpmEgCXlTY7kE4UqziHKOiCk0q 3iBwErXGRoCcwxSck2mjkDrkAlyvu+Xpi8x0csK/AaxKSrKEDY/Mir2Dd0IOdAlcq7cE G/mesb0/SzwOJ6o/i9+38XTGT1lATFGvY9Hy/j5f3DB/f9jLaZAvph2kkSX740o1JkJY J8V+Pd4gY+tD+1kHO3ANPy6WfV7OopjwH77ska5LMsx4Sz0ZVh7naQw1RbxNRx9tVbYi cSa+E3jEMv/922J+zaUsFJq6q4CpJsuVf+JrZf5IqpwLY1FQDqvnCyNJYXCVBPXQ0xR2 o+tQ== X-Gm-Message-State: AOAM531O4SXkCkkBfywOIJkQ/9ax4t9tGgLk+NDgy1M+qjfL0CQi8Z4G FCEHyPnR4vGLGcyZqo3TNx7KWw== X-Google-Smtp-Source: ABdhPJz9bTvkN2/dsnznhn0qkGq1x4rrRxxckt9ZNtviWKF6XljTnBbloojb2RmpNHrLbr6LrtRoPQ== X-Received: by 2002:a5d:660f:: with SMTP id n15mr30481293wru.345.1605186200191; Thu, 12 Nov 2020 05:03:20 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id m22sm6877508wrb.97.2020.11.12.05.03.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Nov 2020 05:03:19 -0800 (PST) From: Jean-Philippe Brucker To: joro@8bytes.org, will@kernel.org, lorenzo.pieralisi@arm.com, robh+dt@kernel.org Subject: [PATCH v8 4/9] of/iommu: Support dma-can-stall property Date: Thu, 12 Nov 2020 13:55:16 +0100 Message-Id: <20201112125519.3987595-5-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201112125519.3987595-1-jean-philippe@linaro.org> References: <20201112125519.3987595-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201112_080321_114481_0659D1DF X-CRM114-Status: GOOD ( 14.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: devicetree@vger.kernel.org, linux-acpi@vger.kernel.org, linux-pci@vger.kernel.org, guohanjun@huawei.com, rjw@rjwysocki.net, shameerali.kolothum.thodi@huawei.com, eric.auger@redhat.com, iommu@lists.linux-foundation.org, Jonathan.Cameron@huawei.com, sudeep.holla@arm.com, bhelgaas@google.com, zhangfei.gao@linaro.org, vivek.gautam@arm.com, baolu.lu@linux.intel.com, robin.murphy@arm.com, Jean-Philippe Brucker , linux-arm-kernel@lists.infradead.org, lenb@kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Copy the dma-can-stall property into the fwspec structure. Signed-off-by: Jean-Philippe Brucker --- include/linux/iommu.h | 2 ++ drivers/iommu/of_iommu.c | 5 ++++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/include/linux/iommu.h b/include/linux/iommu.h index a1c78c4cdeb1..9076fb592c8f 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -572,6 +572,7 @@ struct iommu_group *fsl_mc_device_group(struct device *dev); * @iommu_fwnode: firmware handle for this device's IOMMU * @iommu_priv: IOMMU driver private data for this device * @num_pasid_bits: number of PASID bits supported by this device + * @can_stall: the device is allowed to stall * @num_ids: number of associated device IDs * @ids: IDs which this device may present to the IOMMU */ @@ -579,6 +580,7 @@ struct iommu_fwspec { const struct iommu_ops *ops; struct fwnode_handle *iommu_fwnode; u32 num_pasid_bits; + bool can_stall; unsigned int num_ids; u32 ids[]; }; diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c index e505b9130a1c..d6255ca823d8 100644 --- a/drivers/iommu/of_iommu.c +++ b/drivers/iommu/of_iommu.c @@ -212,9 +212,12 @@ const struct iommu_ops *of_iommu_configure(struct device *dev, err = of_iommu_configure_device(master_np, dev, id); fwspec = dev_iommu_fwspec_get(dev); - if (!err && fwspec) + if (!err && fwspec) { of_property_read_u32(master_np, "pasid-num-bits", &fwspec->num_pasid_bits); + fwspec->can_stall = of_property_read_bool(master_np, + "dma-can-stall"); + } } /* From patchwork Thu Nov 12 12:55:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11900135 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C12CFC2D0A3 for ; Thu, 12 Nov 2020 13:04:30 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 429A721D7F for ; Thu, 12 Nov 2020 13:04:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="yjR2WDEQ"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="L+UnK8b6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 429A721D7F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=B0AE13bwgKJH9q/j4tlNNib8rKiD1LumSh6Zh/mkP1A=; b=yjR2WDEQVbW9PDEhGiILF04WB CyIwItEdLGfDM0uWBJWfHeMYLTRXyFWvfFSa4IfBX4wcG53trnLp1PIqYI2bLSEf/w1TSWA9UwC1b bFe+cL/Zgz0jNXMGEN1eK2G9mqkaRgWf3PPQNcdCjAKECpdP0EO39+Q07JYtnQ6Yz4RAHHRflCbI2 /FYhpXg6jwZfxW+7Jhtw9s+R++Qn3tfOO7jlRPOZHQ/gcPSwRMxF/gXCkbo3pntYdchY62GwO3Hhj hfii3kq9DhgqBXTA8V/IanMTxpGQy6hX4LuHeivOB9d1d5+1ChqjRLciMMIFfsXRr48BzbrfYqLTQ 5/Y7cEStA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCGf-0006ne-Bd; Thu, 12 Nov 2020 13:04:01 +0000 Received: from mail-wr1-x443.google.com ([2a00:1450:4864:20::443]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCG2-0006Z0-IG for linux-arm-kernel@lists.infradead.org; Thu, 12 Nov 2020 13:03:23 +0000 Received: by mail-wr1-x443.google.com with SMTP id 23so5898580wrc.8 for ; Thu, 12 Nov 2020 05:03:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4JGD+U5wzrPrHuAbcShPgq/7SS7eVhaJGMY13GukQwc=; b=L+UnK8b6jHlxkKjKLGZ2KBHb9UiCE2bYESXAd9FIJCk+MSUtGNGNYYhyKKC6RRifPQ PgL5pNXsAPyMWO+HtY1Gz8tYhOV7PCazvQjyezuUbsHfk2M9AaPxMotgauVMZKLo3hiQ M9zO8+yMql5b+8b9cbMFbwaUXk+eUu0753EmMvk7l1HFsWM70OEcQW79vBJFUlgt+5VM KTrz/bb1p5Gfk+J3GZ0/zY9ErMbza+/lZXbPnTQ6Y0oHv+Z0tDMeL6+cldPYswOsQVrL O6uN/kZ4gFKC9yygejnVCUVwrluiILl9XVtVOe0uMkP8IGro9CIcUykOaW84nVGNiLuD 7kaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4JGD+U5wzrPrHuAbcShPgq/7SS7eVhaJGMY13GukQwc=; b=RMfhaROaZvNQLERePvCp5rvxs/+1dkLsjfYVNIPMYpUXPSUL1U/h4Ivdu0Rfz7phaI JnILXv10eEtzCk6nf4OJQ8YqUaLQCuh4oYPaqsOLqkh94pzsIlOEgXZNTLUAOZnmKxmC WDxug3X7b4axdmSZy/cv4mXXdHTvDJihcs4REAs96zLqvF+W3ka/XLC07lLXM9Ga3Dj7 d+YraWUGIuizbXTwWHRLvULpQkRWBvqVQdObAc1hwA2gEwu7ggDYRMQNyo801AaMpU11 swHFTETN8rf0+vxvNGaUu3MpHBJ55B4cxXnKMsXwhrQsHB/eYosboHV/PKUK4Zb1b7iv VbbQ== X-Gm-Message-State: AOAM533+ZRRCPiXjJP/J2qHi61QEIIt6AwKri+Dvgnc9zNzn+rzBwjas SomOMO73HIvuAcuIqu+VXXyemQ== X-Google-Smtp-Source: ABdhPJxo3CkbVLQBJt0LU+udw0kuJH0eA8lhV+b7zpBZqVmuSk9IILiYnXCbLaDmkxe9gZqnDT5qGA== X-Received: by 2002:adf:9d84:: with SMTP id p4mr24594639wre.370.1605186201700; Thu, 12 Nov 2020 05:03:21 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id m22sm6877508wrb.97.2020.11.12.05.03.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Nov 2020 05:03:20 -0800 (PST) From: Jean-Philippe Brucker To: joro@8bytes.org, will@kernel.org, lorenzo.pieralisi@arm.com, robh+dt@kernel.org Subject: [PATCH v8 5/9] ACPI/IORT: Enable stall support for platform devices Date: Thu, 12 Nov 2020 13:55:17 +0100 Message-Id: <20201112125519.3987595-6-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201112125519.3987595-1-jean-philippe@linaro.org> References: <20201112125519.3987595-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201112_080322_672068_5A1B8865 X-CRM114-Status: GOOD ( 12.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: devicetree@vger.kernel.org, linux-acpi@vger.kernel.org, linux-pci@vger.kernel.org, guohanjun@huawei.com, rjw@rjwysocki.net, shameerali.kolothum.thodi@huawei.com, eric.auger@redhat.com, iommu@lists.linux-foundation.org, Jonathan.Cameron@huawei.com, sudeep.holla@arm.com, bhelgaas@google.com, zhangfei.gao@linaro.org, vivek.gautam@arm.com, baolu.lu@linux.intel.com, robin.murphy@arm.com, Jean-Philippe Brucker , linux-arm-kernel@lists.infradead.org, lenb@kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Copy the "Stall supported" bit, that tells whether a platform device supports stall, into the fwspec struct. Signed-off-by: Jean-Philippe Brucker Acked-by: Hanjun Guo --- drivers/acpi/arm64/iort.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c index 70df1ecba7fe..3e39b2212388 100644 --- a/drivers/acpi/arm64/iort.c +++ b/drivers/acpi/arm64/iort.c @@ -968,6 +968,7 @@ static void iort_named_component_init(struct device *dev, nc = (struct acpi_iort_named_component *)node->node_data; fwspec->num_pasid_bits = FIELD_GET(ACPI_IORT_NC_PASID_BITS, nc->node_flags); + fwspec->can_stall = (nc->node_flags & ACPI_IORT_NC_STALL_SUPPORTED); } static int iort_nc_iommu_map(struct device *dev, struct acpi_iort_node *node) From patchwork Thu Nov 12 12:55:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11900169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1ED06C2D0A3 for ; Thu, 12 Nov 2020 13:05:41 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 850E421D7F for ; Thu, 12 Nov 2020 13:05:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="kd2Tv5dQ"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="xRmdctiu" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 850E421D7F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=EkZOmO+q/8uLfaIICVUlzJIbTXbDIk+5EYd+0ilQ830=; b=kd2Tv5dQqfDZX7leCt161QSHn 21IWt04EyqCQ2CK0Zxq+VgCGrKE7GudnhCpfMKUiKkwxtiaHyQsVWKLyKiNj7+z49Lb9kT/I5rFG9 gQWGmUAtpNhUCuTijrlRQErw5soO4jtXO/BNQSTTDSBFN5c71YpXqGOuF/gZ50hp98uXl5Ji9opyW 5+hpcQYhBgDHvPd8wa2XGELqi1RYHx3Mz0EWMnRTvJTOTmt1GhO72unrSY7iQAE6Bh066oi2Vu+se RJIJDsMyWJpAZORTcAhtVSrKfdfTVjN8x7WLt1UX5jjJum2XhArfr+0gVyrbnr3w27Fea7qhtdj4S 6MS8zbnTQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCGs-0006tV-NF; Thu, 12 Nov 2020 13:04:14 +0000 Received: from mail-wm1-x343.google.com ([2a00:1450:4864:20::343]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCG5-0006a5-Hk for linux-arm-kernel@lists.infradead.org; Thu, 12 Nov 2020 13:03:33 +0000 Received: by mail-wm1-x343.google.com with SMTP id 19so5216631wmf.1 for ; Thu, 12 Nov 2020 05:03:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cfEAlXk9Q/1m3LFYgSiXuEmltFiYPYDgoNOZyZUs8uU=; b=xRmdctiu68lUscLl4F5kD3aAMn/mNPne77+RMTMrwtYwTU5hY0lUgKs+DCAsBfckH5 h+jbd1Q2MJvWNTpvmGwglKXdTsy5ETmzHwulfMVXUHZR0vh+1V+D1cl4xXAmhC5bUuNf 3PnrpXxoAnAKR9uNbj7eH9MVZPZfXWyiF8Yr5Bj4Bmo8BlNgNM+jqKKxSv7jlVII/dYx 8Mm04Zag62Qd34vkI7TIxiCV4QukQy4csUJKTzUp8kGquqoLEma5QVzgRwCJl1N6s6sG 1en0i9zujF5wJHHHYWMHSAEO7BWE5l8f4u7J8uurlWNynBzacQk4+KKS/ILL/RZZU0hZ /qJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cfEAlXk9Q/1m3LFYgSiXuEmltFiYPYDgoNOZyZUs8uU=; b=BglJm2RB9KDpt7/zmvnUkx59t34BvLrMbYLZjW8xEXL80KVajmYiW4t4ZdUxIfztPj cBPYvT03SJllIYm0B8SNkywsMslWCRafF5fYSVe7uPQtaazZ5/lBUQEQPoI9l9+EEQv2 gMbEF5D8OefDP5ZctE2OgWmdUb+10W9P4Jhn6o3tD/QDNSKuw/iS3x+nxCMQvcowFKoU wscgylzTtOOzCHfwnQ+VAyBxSaUSXQ0hZKH0PhDGTHcIpR6nULU6l75LIHDN8q+O/mQQ cZ6PRIrWxMg+VNiD0oT0g+EcncgspoKaF8FgmldXBSdzM+ZLHmgqJxN71SZb6kxhocQE TtrA== X-Gm-Message-State: AOAM532n11hld5QKAzASYN7JPtH3QZiwss35IK8Q+yA3FZqvODahX7hk p32B/Kk8jQTeqf+R9FChqU1JUA== X-Google-Smtp-Source: ABdhPJyCTkmAmTs10iqu7CA7KPn82ozQDLsDMrmgaCq2Gq7UZiSspRy9iSQJIHf8wcojJHdfB8ysAQ== X-Received: by 2002:a1c:ddc4:: with SMTP id u187mr9985624wmg.55.1605186203154; Thu, 12 Nov 2020 05:03:23 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id m22sm6877508wrb.97.2020.11.12.05.03.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Nov 2020 05:03:22 -0800 (PST) From: Jean-Philippe Brucker To: joro@8bytes.org, will@kernel.org, lorenzo.pieralisi@arm.com, robh+dt@kernel.org Subject: [PATCH v8 6/9] iommu/arm-smmu-v3: Add stall support for platform devices Date: Thu, 12 Nov 2020 13:55:18 +0100 Message-Id: <20201112125519.3987595-7-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201112125519.3987595-1-jean-philippe@linaro.org> References: <20201112125519.3987595-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201112_080325_783675_C8884046 X-CRM114-Status: GOOD ( 28.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: devicetree@vger.kernel.org, linux-acpi@vger.kernel.org, linux-pci@vger.kernel.org, guohanjun@huawei.com, rjw@rjwysocki.net, shameerali.kolothum.thodi@huawei.com, eric.auger@redhat.com, iommu@lists.linux-foundation.org, Jonathan.Cameron@huawei.com, sudeep.holla@arm.com, bhelgaas@google.com, zhangfei.gao@linaro.org, vivek.gautam@arm.com, baolu.lu@linux.intel.com, robin.murphy@arm.com, Jean-Philippe Brucker , linux-arm-kernel@lists.infradead.org, lenb@kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The SMMU provides a Stall model for handling page faults in platform devices. It is similar to PCIe PRI, but doesn't require devices to have their own translation cache. Instead, faulting transactions are parked and the OS is given a chance to fix the page tables and retry the transaction. Enable stall for devices that support it (opt-in by firmware). When an event corresponds to a translation error, call the IOMMU fault handler. If the fault is recoverable, it will call us back to terminate or continue the stall. Signed-off-by: Jean-Philippe Brucker --- v8: * Extract firwmare setting into separate patch * Don't drain event queue on unbind() --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 36 ++++ .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 26 ++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 173 +++++++++++++++++- 3 files changed, 224 insertions(+), 11 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 7bd98fdce5c3..124f604ed677 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -361,6 +361,13 @@ #define CMDQ_PRI_1_GRPID GENMASK_ULL(8, 0) #define CMDQ_PRI_1_RESP GENMASK_ULL(13, 12) +#define CMDQ_RESUME_0_SID GENMASK_ULL(63, 32) +#define CMDQ_RESUME_0_RESP_TERM 0UL +#define CMDQ_RESUME_0_RESP_RETRY 1UL +#define CMDQ_RESUME_0_RESP_ABORT 2UL +#define CMDQ_RESUME_0_RESP GENMASK_ULL(13, 12) +#define CMDQ_RESUME_1_STAG GENMASK_ULL(15, 0) + #define CMDQ_SYNC_0_CS GENMASK_ULL(13, 12) #define CMDQ_SYNC_0_CS_NONE 0 #define CMDQ_SYNC_0_CS_IRQ 1 @@ -377,6 +384,25 @@ #define EVTQ_0_ID GENMASK_ULL(7, 0) +#define EVT_ID_TRANSLATION_FAULT 0x10 +#define EVT_ID_ADDR_SIZE_FAULT 0x11 +#define EVT_ID_ACCESS_FAULT 0x12 +#define EVT_ID_PERMISSION_FAULT 0x13 + +#define EVTQ_0_SSV (1UL << 11) +#define EVTQ_0_SSID GENMASK_ULL(31, 12) +#define EVTQ_0_SID GENMASK_ULL(63, 32) +#define EVTQ_1_STAG GENMASK_ULL(15, 0) +#define EVTQ_1_STALL (1UL << 31) +#define EVTQ_1_PRIV (1UL << 33) +#define EVTQ_1_EXEC (1UL << 34) +#define EVTQ_1_READ (1UL << 35) +#define EVTQ_1_S2 (1UL << 39) +#define EVTQ_1_CLASS GENMASK_ULL(41, 40) +#define EVTQ_1_TT_READ (1UL << 44) +#define EVTQ_2_ADDR GENMASK_ULL(63, 0) +#define EVTQ_3_IPA GENMASK_ULL(51, 12) + /* PRI queue */ #define PRIQ_ENT_SZ_SHIFT 4 #define PRIQ_ENT_DWORDS ((1 << PRIQ_ENT_SZ_SHIFT) >> 3) @@ -471,6 +497,13 @@ struct arm_smmu_cmdq_ent { enum pri_resp resp; } pri; + #define CMDQ_OP_RESUME 0x44 + struct { + u32 sid; + u16 stag; + u8 resp; + } resume; + #define CMDQ_OP_CMD_SYNC 0x46 struct { u64 msiaddr; @@ -529,6 +562,7 @@ struct arm_smmu_cmdq_batch { struct arm_smmu_evtq { struct arm_smmu_queue q; + struct iopf_queue *iopf; u32 max_stalls; }; @@ -668,6 +702,7 @@ struct arm_smmu_master { struct arm_smmu_stream *streams; unsigned int num_streams; bool ats_enabled; + bool stall_enabled; bool sva_enabled; struct list_head bonds; unsigned int ssid_bits; @@ -687,6 +722,7 @@ struct arm_smmu_domain { struct io_pgtable_ops *pgtbl_ops; bool non_strict; + bool stall_enabled; atomic_t nr_ats_masters; enum arm_smmu_domain_stage stage; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index f734797ea07a..64e2082ef9ed 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -435,7 +435,7 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) static bool arm_smmu_iopf_supported(struct arm_smmu_master *master) { - return false; + return master->stall_enabled; } bool arm_smmu_master_sva_supported(struct arm_smmu_master *master) @@ -459,24 +459,46 @@ bool arm_smmu_master_sva_enabled(struct arm_smmu_master *master) int arm_smmu_master_enable_sva(struct arm_smmu_master *master) { + int ret; + struct device *dev = master->dev; + + if (master->stall_enabled) { + ret = iopf_queue_add_device(master->smmu->evtq.iopf, dev); + if (ret) + return ret; + } + + ret = iommu_register_device_fault_handler(dev, iommu_queue_iopf, dev); + if (ret) + goto err_disable_iopf; + mutex_lock(&sva_lock); master->sva_enabled = true; mutex_unlock(&sva_lock); return 0; + +err_disable_iopf: + iopf_queue_remove_device(master->smmu->evtq.iopf, dev); + return ret; } int arm_smmu_master_disable_sva(struct arm_smmu_master *master) { + struct device *dev = master->dev; + mutex_lock(&sva_lock); if (!list_empty(&master->bonds)) { - dev_err(master->dev, "cannot disable SVA, device is bound\n"); + dev_err(dev, "cannot disable SVA, device is bound\n"); mutex_unlock(&sva_lock); return -EBUSY; } master->sva_enabled = false; mutex_unlock(&sva_lock); + iommu_unregister_device_fault_handler(dev); + iopf_queue_remove_device(master->smmu->evtq.iopf, dev); + return 0; } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index d87c87136d63..d412d063d3b6 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -32,6 +32,7 @@ #include #include "arm-smmu-v3.h" +#include "../../iommu-sva-lib.h" static bool disable_bypass = 1; module_param(disable_bypass, bool, 0444); @@ -324,6 +325,11 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) } cmd[1] |= FIELD_PREP(CMDQ_PRI_1_RESP, ent->pri.resp); break; + case CMDQ_OP_RESUME: + cmd[0] |= FIELD_PREP(CMDQ_RESUME_0_SID, ent->resume.sid); + cmd[0] |= FIELD_PREP(CMDQ_RESUME_0_RESP, ent->resume.resp); + cmd[1] |= FIELD_PREP(CMDQ_RESUME_1_STAG, ent->resume.stag); + break; case CMDQ_OP_CMD_SYNC: if (ent->sync.msiaddr) { cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_IRQ); @@ -887,6 +893,44 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu, return arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmds, cmds->num, true); } +static int arm_smmu_page_response(struct device *dev, + struct iommu_fault_event *unused, + struct iommu_page_response *resp) +{ + struct arm_smmu_cmdq_ent cmd = {0}; + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + int sid = master->streams[0].id; + + if (master->stall_enabled) { + cmd.opcode = CMDQ_OP_RESUME; + cmd.resume.sid = sid; + cmd.resume.stag = resp->grpid; + switch (resp->code) { + case IOMMU_PAGE_RESP_INVALID: + case IOMMU_PAGE_RESP_FAILURE: + cmd.resume.resp = CMDQ_RESUME_0_RESP_ABORT; + break; + case IOMMU_PAGE_RESP_SUCCESS: + cmd.resume.resp = CMDQ_RESUME_0_RESP_RETRY; + break; + default: + return -EINVAL; + } + } else { + return -ENODEV; + } + + arm_smmu_cmdq_issue_cmd(master->smmu, &cmd); + /* + * Don't send a SYNC, it doesn't do anything for RESUME or PRI_RESP. + * RESUME consumption guarantees that the stalled transaction will be + * terminated... at some point in the future. PRI_RESP is fire and + * forget. + */ + + return 0; +} + /* Context descriptor manipulation functions */ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) { @@ -1049,8 +1093,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) | CTXDESC_CD_0_V; - /* STALL_MODEL==0b10 && CD.S==0 is ILLEGAL */ - if (smmu->features & ARM_SMMU_FEAT_STALL_FORCE) + if (smmu_domain->stall_enabled) val |= CTXDESC_CD_0_S; } @@ -1294,7 +1337,7 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid, FIELD_PREP(STRTAB_STE_1_STRW, strw)); if (smmu->features & ARM_SMMU_FEAT_STALLS && - !(smmu->features & ARM_SMMU_FEAT_STALL_FORCE)) + !master->stall_enabled) dst[1] |= cpu_to_le64(STRTAB_STE_1_S1STALLD); val |= (s1_cfg->cdcfg.cdtab_dma & STRTAB_STE_0_S1CTXPTR_MASK) | @@ -1371,7 +1414,6 @@ static int arm_smmu_init_l2_strtab(struct arm_smmu_device *smmu, u32 sid) return 0; } -__maybe_unused static struct arm_smmu_master * arm_smmu_find_master(struct arm_smmu_device *smmu, u32 sid) { @@ -1398,9 +1440,96 @@ arm_smmu_find_master(struct arm_smmu_device *smmu, u32 sid) } /* IRQ and event handlers */ +static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64 *evt) +{ + int ret; + u32 perm = 0; + struct arm_smmu_master *master; + bool ssid_valid = evt[0] & EVTQ_0_SSV; + u8 type = FIELD_GET(EVTQ_0_ID, evt[0]); + u32 sid = FIELD_GET(EVTQ_0_SID, evt[0]); + struct iommu_fault_event fault_evt = { }; + struct iommu_fault *flt = &fault_evt.fault; + + /* Stage-2 is always pinned at the moment */ + if (evt[1] & EVTQ_1_S2) + return -EFAULT; + + master = arm_smmu_find_master(smmu, sid); + if (!master) + return -EINVAL; + + if (evt[1] & EVTQ_1_READ) + perm |= IOMMU_FAULT_PERM_READ; + else + perm |= IOMMU_FAULT_PERM_WRITE; + + if (evt[1] & EVTQ_1_EXEC) + perm |= IOMMU_FAULT_PERM_EXEC; + + if (evt[1] & EVTQ_1_PRIV) + perm |= IOMMU_FAULT_PERM_PRIV; + + if (evt[1] & EVTQ_1_STALL) { + flt->type = IOMMU_FAULT_PAGE_REQ; + flt->prm = (struct iommu_fault_page_request) { + .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE, + .grpid = FIELD_GET(EVTQ_1_STAG, evt[1]), + .perm = perm, + .addr = FIELD_GET(EVTQ_2_ADDR, evt[2]), + }; + + if (ssid_valid) { + flt->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; + flt->prm.pasid = FIELD_GET(EVTQ_0_SSID, evt[0]); + } + } else { + flt->type = IOMMU_FAULT_DMA_UNRECOV; + flt->event = (struct iommu_fault_unrecoverable) { + .flags = IOMMU_FAULT_UNRECOV_ADDR_VALID | + IOMMU_FAULT_UNRECOV_FETCH_ADDR_VALID, + .perm = perm, + .addr = FIELD_GET(EVTQ_2_ADDR, evt[2]), + .fetch_addr = FIELD_GET(EVTQ_3_IPA, evt[3]), + }; + + if (ssid_valid) { + flt->event.flags |= IOMMU_FAULT_UNRECOV_PASID_VALID; + flt->event.pasid = FIELD_GET(EVTQ_0_SSID, evt[0]); + } + + switch (type) { + case EVT_ID_TRANSLATION_FAULT: + case EVT_ID_ADDR_SIZE_FAULT: + case EVT_ID_ACCESS_FAULT: + flt->event.reason = IOMMU_FAULT_REASON_PTE_FETCH; + break; + case EVT_ID_PERMISSION_FAULT: + flt->event.reason = IOMMU_FAULT_REASON_PERMISSION; + break; + default: + /* TODO: report other unrecoverable faults. */ + return -EFAULT; + } + } + + ret = iommu_report_device_fault(master->dev, &fault_evt); + if (ret && flt->type == IOMMU_FAULT_PAGE_REQ) { + /* Nobody cared, abort the access */ + struct iommu_page_response resp = { + .pasid = flt->prm.pasid, + .grpid = flt->prm.grpid, + .code = IOMMU_PAGE_RESP_FAILURE, + }; + arm_smmu_page_response(master->dev, NULL, &resp); + } + + return ret; +} + static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) { - int i; + int i, ret; struct arm_smmu_device *smmu = dev; struct arm_smmu_queue *q = &smmu->evtq.q; struct arm_smmu_ll_queue *llq = &q->llq; @@ -1410,11 +1539,15 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) while (!queue_remove_raw(q, evt)) { u8 id = FIELD_GET(EVTQ_0_ID, evt[0]); - dev_info(smmu->dev, "event 0x%02x received:\n", id); - for (i = 0; i < ARRAY_SIZE(evt); ++i) - dev_info(smmu->dev, "\t0x%016llx\n", - (unsigned long long)evt[i]); + ret = arm_smmu_handle_evt(smmu, evt); + if (ret) { + dev_info(smmu->dev, "event 0x%02x received:\n", + id); + for (i = 0; i < ARRAY_SIZE(evt); ++i) + dev_info(smmu->dev, "\t0x%016llx\n", + (unsigned long long)evt[i]); + } } /* @@ -1927,6 +2060,8 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain, cfg->s1cdmax = master->ssid_bits; + smmu_domain->stall_enabled = master->stall_enabled; + ret = arm_smmu_alloc_cd_tables(smmu_domain); if (ret) goto out_free_asid; @@ -2270,6 +2405,12 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) smmu_domain->s1_cfg.s1cdmax, master->ssid_bits); ret = -EINVAL; goto out_unlock; + } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 && + smmu_domain->stall_enabled != master->stall_enabled) { + dev_err(dev, "cannot attach to stall-%s domain\n", + smmu_domain->stall_enabled ? "enabled" : "disabled"); + ret = -EINVAL; + goto out_unlock; } master->domain = smmu_domain; @@ -2503,6 +2644,10 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) master->ssid_bits = min_t(u8, master->ssid_bits, CTXDESC_LINEAR_CDMAX); + if ((smmu->features & ARM_SMMU_FEAT_STALLS && fwspec->can_stall) || + smmu->features & ARM_SMMU_FEAT_STALL_FORCE) + master->stall_enabled = true; + return &smmu->iommu; err_free_master: @@ -2521,6 +2666,7 @@ static void arm_smmu_release_device(struct device *dev) master = dev_iommu_priv_get(dev); WARN_ON(arm_smmu_master_sva_enabled(master)); + iopf_queue_remove_device(master->smmu->evtq.iopf, dev); arm_smmu_detach_dev(master); arm_smmu_disable_pasid(master); arm_smmu_remove_master(master); @@ -2727,6 +2873,7 @@ static struct iommu_ops arm_smmu_ops = { .sva_bind = arm_smmu_sva_bind, .sva_unbind = arm_smmu_sva_unbind, .sva_get_pasid = arm_smmu_sva_get_pasid, + .page_response = arm_smmu_page_response, .pgsize_bitmap = -1UL, /* Restricted during device attach */ }; @@ -2804,6 +2951,7 @@ static int arm_smmu_cmdq_init(struct arm_smmu_device *smmu) static int arm_smmu_init_queues(struct arm_smmu_device *smmu) { int ret; + bool sva = arm_smmu_sva_supported(smmu); /* cmdq */ ret = arm_smmu_init_one_queue(smmu, &smmu->cmdq.q, ARM_SMMU_CMDQ_PROD, @@ -2823,6 +2971,12 @@ static int arm_smmu_init_queues(struct arm_smmu_device *smmu) if (ret) return ret; + if (sva && smmu->features & ARM_SMMU_FEAT_STALLS) { + smmu->evtq.iopf = iopf_queue_alloc(dev_name(smmu->dev)); + if (!smmu->evtq.iopf) + return -ENOMEM; + } + /* priq */ if (!(smmu->features & ARM_SMMU_FEAT_PRI)) return 0; @@ -3789,6 +3943,7 @@ static int arm_smmu_device_remove(struct platform_device *pdev) iommu_device_unregister(&smmu->iommu); iommu_device_sysfs_remove(&smmu->iommu); arm_smmu_device_disable(smmu); + iopf_queue_free(smmu->evtq.iopf); return 0; } From patchwork Thu Nov 12 12:55:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11900161 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53898C2D0A3 for ; Thu, 12 Nov 2020 13:05:35 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C412421D7F for ; Thu, 12 Nov 2020 13:05:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="I37VaKoB"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="QZfgK97B" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C412421D7F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=zLH1vf4VXaHJ00nCIIdDeJvfBt8DoxorQmBNZS+wXts=; b=I37VaKoBlhs7MM+lgxJGFmA0b jadNJwbcNypTfVMtgMjdUQkJvHveQSzx665w2v2H5VDp931QtLmrbN5pw4gDljEMURQVUdE3i4+Ex a1nAQRTY/jK6E0l7J2PyXrSp0xWGGQbyyyESENiHQwZNHtdPHCjxlam7TddlRoiuM3PgIoBVsa4qX KSGcxTsBv1Yr61xtaJL8EOQ0a5w3XaiOm1sYadfSBuRa8UX3lc1DK03DOebNcarMatHjhdNrw6hqc oEX8sI5YX1qcZSMN87xDhEpyjBxJ2jaobGEy/6B9H28BhmuOWoKbhP/lAhTATRW+Cj8EWefut+myu Us9sSSpVQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCGo-0006rG-OR; Thu, 12 Nov 2020 13:04:10 +0000 Received: from mail-wm1-x343.google.com ([2a00:1450:4864:20::343]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCG5-0006aV-Hj for linux-arm-kernel@lists.infradead.org; Thu, 12 Nov 2020 13:03:29 +0000 Received: by mail-wm1-x343.google.com with SMTP id d142so5190067wmd.4 for ; Thu, 12 Nov 2020 05:03:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YNIotbOC7bka5Z9BLnFVmsCtOLB6kJ/rhFA9CsCNoBU=; b=QZfgK97B8RvBj1vgMXgdtAlJTXGErNY2MHWaYeZJYcihAKpzxznX+f3QupErxyW+y7 vl5dFcj/SwiTM7AoiBl8Ekh/VI2I+MV6RRDXOcrbDCGTkeMw66SFzS1DgJvxjcaDuKjk 2bmfttuRd5KL3r+eSvLlVqZxie8jGCaK6VIEInNDm+bzEK1xUWxTL3t2xMOYv4Bass5p gBiBnUXUjVBFvCB1BrWOR56d6TtjFa2TYTFOwxI7fDliFlDQVEIWqgTYl5sIlbDUR1ej iCxbPa+0gIBZXHXjDH2ssbGlbO7TSEgUjnx9UYgYtW9ALf/I+qK3xCqik9HnPlfBcF/J Vzrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YNIotbOC7bka5Z9BLnFVmsCtOLB6kJ/rhFA9CsCNoBU=; b=aR3JSFnRFgb3PRGUHJVG9aIZuACIj09TnqQH8wGETy61X1Fhw5kZ3zWDdUoz3Pd6Yi DdPQaephnh8l+Y/0Yox9Ddp+NLzwJMkMvbCrmu/AOnEIygj/sO11Gg2+poTLWMSFiI6k A2AfB7BNI0nA5f32WyjTvwOpFpDoX+CCVK3ZOQDnBfth3jjhJlwpo+AGqO3AvppwT1xX r1FoKkTRZDWsqxSFC226nT5lITO8KmuXR5W/UGke+mhQxK2WeZbM6ON3yoRh2yQjTXxv WRkqiQEJpC0GTU3+Xvu+t4sLawg+iAj18l5TELJmnZ4oSZyjCS9GM6Wh7SX1JEbEKHla DXpg== X-Gm-Message-State: AOAM530SGQwBdHjb4z8t0Scs+6EN1OQzI4zZyi/Feq/OYMz8gC6Pjn4S uGuXrnzUuk5nx/Na7AwJTFdSAQ== X-Google-Smtp-Source: ABdhPJwEuQSW7wZd81DIymTmLqqJtRteXgrSKTo8T+v6vTfsJF7cOILOIkbIrFkOfcCnh2vnSQXfbA== X-Received: by 2002:a1c:2586:: with SMTP id l128mr9104219wml.149.1605186204490; Thu, 12 Nov 2020 05:03:24 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id m22sm6877508wrb.97.2020.11.12.05.03.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Nov 2020 05:03:23 -0800 (PST) From: Jean-Philippe Brucker To: joro@8bytes.org, will@kernel.org, lorenzo.pieralisi@arm.com, robh+dt@kernel.org Subject: [PATCH v8 7/9] PCI/ATS: Add PRI stubs Date: Thu, 12 Nov 2020 13:55:19 +0100 Message-Id: <20201112125519.3987595-8-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201112125519.3987595-1-jean-philippe@linaro.org> References: <20201112125519.3987595-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201112_080325_740105_C0BEF38A X-CRM114-Status: GOOD ( 11.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: devicetree@vger.kernel.org, Kuppuswamy Sathyanarayanan , linux-acpi@vger.kernel.org, linux-pci@vger.kernel.org, guohanjun@huawei.com, rjw@rjwysocki.net, shameerali.kolothum.thodi@huawei.com, eric.auger@redhat.com, iommu@lists.linux-foundation.org, Jonathan.Cameron@huawei.com, sudeep.holla@arm.com, bhelgaas@google.com, zhangfei.gao@linaro.org, vivek.gautam@arm.com, baolu.lu@linux.intel.com, robin.murphy@arm.com, Jean-Philippe Brucker , linux-arm-kernel@lists.infradead.org, lenb@kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The SMMUv3 driver, which can be built without CONFIG_PCI, will soon gain support for PRI. Partially revert commit c6e9aefbf9db ("PCI/ATS: Remove unused PRI and PASID stubs") to re-introduce the PRI stubs, and avoid adding more #ifdefs to the SMMU driver. Acked-by: Bjorn Helgaas Reviewed-by: Kuppuswamy Sathyanarayanan Signed-off-by: Jean-Philippe Brucker --- include/linux/pci-ats.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/include/linux/pci-ats.h b/include/linux/pci-ats.h index df54cd5b15db..ccfca09fd232 100644 --- a/include/linux/pci-ats.h +++ b/include/linux/pci-ats.h @@ -30,6 +30,13 @@ int pci_reset_pri(struct pci_dev *pdev); int pci_prg_resp_pasid_required(struct pci_dev *pdev); bool pci_pri_supported(struct pci_dev *pdev); #else +static inline int pci_enable_pri(struct pci_dev *pdev, u32 reqs) +{ return -ENODEV; } +static inline void pci_disable_pri(struct pci_dev *pdev) { } +static inline int pci_reset_pri(struct pci_dev *pdev) +{ return -ENODEV; } +static inline int pci_prg_resp_pasid_required(struct pci_dev *pdev) +{ return 0; } static inline bool pci_pri_supported(struct pci_dev *pdev) { return false; } #endif /* CONFIG_PCI_PRI */ From patchwork Thu Nov 12 12:55:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11900159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07F88C2D0A3 for ; Thu, 12 Nov 2020 13:04:46 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7570521D7F for ; Thu, 12 Nov 2020 13:04:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="nUM8Esq7"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="AJNFqP/Z" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7570521D7F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=BxiN9UGpcA3fWwFQDbs92uUqfVsEmhJzi4oQU+89hso=; b=nUM8Esq7YkN4HbYEPuf6JQlGK C04yLnF1f2IEejWx4o1v+Ggp5kp4psUkfszpjM2u3n0B09jh84LKcI2gjPebrqZLZFETTAoqCFj6Z +0Bxpv1tVkgYUKHJfCYYlHTzwhIf/EFG7s+yhGKesloIMxS4dJY+dHF4NQKXlcmA1sHWzLLczy8TH IZhFHYkWRpWo/36cksRZPsTwYjfpmI7Mrtts5Sok6LD3Rp7HGc59zlbCr/CDAebVtBGTKNPXwclty KnJvOawj+NhYuhF8+u1fBrwGToGWxxlSnmjEpnqH/EdMWdniDxztFOdurp3J3UDZcPrmv7wlSDfMI vkRI2cWgg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCGx-0006vW-3p; Thu, 12 Nov 2020 13:04:19 +0000 Received: from mail-wm1-x343.google.com ([2a00:1450:4864:20::343]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCG8-0006br-81 for linux-arm-kernel@lists.infradead.org; Thu, 12 Nov 2020 13:03:34 +0000 Received: by mail-wm1-x343.google.com with SMTP id s13so5431647wmh.4 for ; Thu, 12 Nov 2020 05:03:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qVbQfn+wW1FJTP9+IaoEjGKkOhoG1CH1YVgrdlNJ6Sw=; b=AJNFqP/ZKhk6objxUUtP+6tvikca2WaZ6jTYQcqn0yvFmq93hZ3lVQHYyK7f33pXgE LMPNyvQtw49zSsYAQyGirgOmAjeKMdajdiJVAvrB0xvcpLKnNUh1Z8ZszvxyGNv+yFWO jRyaMPOhhzNbE3/mz7+lXJ3IhlCs6H3xrg7kN1+ZuHmW+evNoDn1JYTAFeNTCQVfBP6C UDXzfEZDz9yBRbmmCKA8RPN03ITbdvYu314gLJxyyi+H6Vek4EQpQ+DnwUjWmEWZQ4P7 ecl6SKlD8ae4yC6bXIaFUH6JgCSMUZFZyPJSzMPNHa1vLYEGIy0ojNDIzjqoS0ABNNt4 JmSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qVbQfn+wW1FJTP9+IaoEjGKkOhoG1CH1YVgrdlNJ6Sw=; b=M0SqYa4AbJYvcbPGVWmO4ttE5AbHkRqBK9f9T4XhiZGPPCF3sqxLlllOu2gy5Ymzvo rGMRo4oc8CS0ZPB9HqTplAzUgrA/NmMBnqzZhJeoL/+IB4ASMWyEkaAvpjX/5yd9mxv3 rZ1uPWBR7JVXvkXgWgDzzl+1M7jQ19DEMb4vaqC8A1m+55mcvc/bwG46JNbd+gUywr3m XasNMbBBMOxf5TrRxPuAUxGlgFpAtt4BeOEO8C9aZIMHzRHJqM4zPrRiCMTZaRubBcdH 1pXo0gEb6lL9xEn34OlLVWLJPpw3Zs7WgipcuSiiGJcv4uELuiQiiwTncrjvtB0ussW0 FwgQ== X-Gm-Message-State: AOAM530v3phEhf+gzoBFJWJPSrRYqcxVoCh3G5tB7ne/4o1RDT+sehTe vdo1j9rSsM2P1HdBligWNVc1KQ== X-Google-Smtp-Source: ABdhPJxsDYTMbvBnnj6uOORdu3x6BbabxcW+EJNp1tzj8KBQp5jtE8PRmp5+1detfGdDqBG6DaNgNA== X-Received: by 2002:a7b:c05a:: with SMTP id u26mr10063043wmc.159.1605186207268; Thu, 12 Nov 2020 05:03:27 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id m22sm6877508wrb.97.2020.11.12.05.03.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Nov 2020 05:03:26 -0800 (PST) From: Jean-Philippe Brucker To: joro@8bytes.org, will@kernel.org, lorenzo.pieralisi@arm.com, robh+dt@kernel.org Subject: [PATCH v8 8/9] PCI/ATS: Export PRI functions Date: Thu, 12 Nov 2020 13:55:20 +0100 Message-Id: <20201112125519.3987595-9-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201112125519.3987595-1-jean-philippe@linaro.org> References: <20201112125519.3987595-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201112_080329_536310_83BC4D94 X-CRM114-Status: GOOD ( 11.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: devicetree@vger.kernel.org, Kuppuswamy Sathyanarayanan , linux-acpi@vger.kernel.org, linux-pci@vger.kernel.org, guohanjun@huawei.com, rjw@rjwysocki.net, shameerali.kolothum.thodi@huawei.com, eric.auger@redhat.com, iommu@lists.linux-foundation.org, Jonathan.Cameron@huawei.com, sudeep.holla@arm.com, bhelgaas@google.com, zhangfei.gao@linaro.org, vivek.gautam@arm.com, baolu.lu@linux.intel.com, robin.murphy@arm.com, Jean-Philippe Brucker , linux-arm-kernel@lists.infradead.org, lenb@kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The SMMUv3 driver uses pci_{enable,disable}_pri() and related functions. Export those functions to allow the driver to be built as a module. Acked-by: Bjorn Helgaas Reviewed-by: Kuppuswamy Sathyanarayanan Signed-off-by: Jean-Philippe Brucker --- drivers/pci/ats.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c index 46bc7f31fb4d..e36d601015d9 100644 --- a/drivers/pci/ats.c +++ b/drivers/pci/ats.c @@ -191,6 +191,7 @@ void pci_pri_init(struct pci_dev *pdev) if (status & PCI_PRI_STATUS_PASID) pdev->pasid_required = 1; } +EXPORT_SYMBOL_GPL(pci_pri_init); /** * pci_enable_pri - Enable PRI capability @@ -238,6 +239,7 @@ int pci_enable_pri(struct pci_dev *pdev, u32 reqs) return 0; } +EXPORT_SYMBOL_GPL(pci_enable_pri); /** * pci_disable_pri - Disable PRI capability @@ -317,6 +319,7 @@ int pci_reset_pri(struct pci_dev *pdev) return 0; } +EXPORT_SYMBOL_GPL(pci_reset_pri); /** * pci_prg_resp_pasid_required - Return PRG Response PASID Required bit @@ -332,6 +335,7 @@ int pci_prg_resp_pasid_required(struct pci_dev *pdev) return pdev->pasid_required; } +EXPORT_SYMBOL_GPL(pci_prg_resp_pasid_required); /** * pci_pri_supported - Check if PRI is supported. From patchwork Thu Nov 12 12:55:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 11900163 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AE34C388F7 for ; Thu, 12 Nov 2020 13:04:54 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9B36021D7F for ; Thu, 12 Nov 2020 13:04:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Gwk7w/AC"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="QzsCjHvg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9B36021D7F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=/jtTysVOW5HsYIdqJ3BspYderyJbipxWUhG/y13fRNA=; b=Gwk7w/ACNTanEVFqU4avInE9k sSsSoPjEqwKTAVGMZuZUrW+t4JsN/buNnDb5egcZlB70QDX+SwxHK185WNifbl0/id3+cZCeLmnDV JHMpJ+BRzeSOum5S0LK7AGBpeC6Y2wKR+E0hJdFK0hzt9q/g7M3GtuYAmHlNiWZYZ4+SbPns/3MWR 5RhWCY5Y95lqeVzyNUw6dEUATqFT5pxjQGg51Bzr61CXmyo+xrissXx9ZNDJeQyFP7iqK5fg+91aQ NLjQuvLtMT3qnVHZ8PA6mC90M7BE9CxNPNetvEI76c9NyGaA+GneaOd7yLfaks52b1ps+aEBfYQuE UtLCOj3eQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCH0-0006xi-Jk; Thu, 12 Nov 2020 13:04:22 +0000 Received: from mail-wr1-x443.google.com ([2a00:1450:4864:20::443]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdCGA-0006cF-Nu for linux-arm-kernel@lists.infradead.org; Thu, 12 Nov 2020 13:03:42 +0000 Received: by mail-wr1-x443.google.com with SMTP id r17so5946471wrw.1 for ; Thu, 12 Nov 2020 05:03:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KzHG2PweYWkDaJQol3GXcq+4FrqBl3EkRXm7+VNwUmc=; b=QzsCjHvgwTaDtXJ9+Q3O62Yd7pbjZCU3265TBqqZMjaMxMwjrdRp4jHXltzEEU07rs 8mmfQkm8fEnqYqM0E0REckDPxMoROlKoFtzVSydnxin5V/rRTpPx8c5+TuZBGtaTYEuO f4LcmtfroCNh/UOj+hBVBMGKXKu9b7RngKKy6U2RaRfTApWKsBcZ42osFB8fh4Pp9Nqs W67apIz096ig0sdMkpZ2/bsUrG70tIHfMCGQW8OLPOyP3hEhSDWYdVJKLEoDu4+aCb4V Ds1lnR59Y3qMAHjkDFnLi9V4r9sGgD5xjVu1aE0drSG6GBMw+oN9u15GPn+X/9k+IF7F km9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KzHG2PweYWkDaJQol3GXcq+4FrqBl3EkRXm7+VNwUmc=; b=S5xJpsNWee9A+qhLGmr+J1lOE8X1Zk/49xQTBdrlc1h7t1hv106OIBV7odAMwnRi7q JFeOMBgagfqFLz1j1eHjpCBtFCW+rYm8OfNTOMnG1WPpEUfgdnJG0q31CJRsmhqXVynX ipcSjwfWq7sZ9V9isZ1CW4tYp+ZHiss/TYrheVTtg2UiA8xlzjKPW+XKdltPrJ/h4gCL GXtN3558zLFcwd2jnGTb1En/e1VYi+6J6Sel7kkVeHDYAP3D6A4lz7fSWRm0lXh0rKJA ZAoYnPUb86zMCb+B0qUhuY+OLwjxK+XkFfDmbUces8qYdhdkfCAu9GRpJNLy3riQ+zMY W5Pg== X-Gm-Message-State: AOAM5304+Kf9Gwc2hgs92lo6cMNo8SZU1d7WbwizXhqIvSVVijF8wm5m SLN9QYaduzFM+cwMQVaV5VQrrQ== X-Google-Smtp-Source: ABdhPJxuUemJNG+7LLzequD96xCA6dBRHFX8NgjY+pEi0PeawvJzITH5k3FZUtnf92WjB1bi7h2w8A== X-Received: by 2002:adf:f041:: with SMTP id t1mr12188827wro.139.1605186208754; Thu, 12 Nov 2020 05:03:28 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id m22sm6877508wrb.97.2020.11.12.05.03.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Nov 2020 05:03:28 -0800 (PST) From: Jean-Philippe Brucker To: joro@8bytes.org, will@kernel.org, lorenzo.pieralisi@arm.com, robh+dt@kernel.org Subject: [PATCH v8 9/9] iommu/arm-smmu-v3: Add support for PRI Date: Thu, 12 Nov 2020 13:55:21 +0100 Message-Id: <20201112125519.3987595-10-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201112125519.3987595-1-jean-philippe@linaro.org> References: <20201112125519.3987595-1-jean-philippe@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201112_080331_015067_83A09C2A X-CRM114-Status: GOOD ( 27.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: devicetree@vger.kernel.org, linux-acpi@vger.kernel.org, linux-pci@vger.kernel.org, guohanjun@huawei.com, rjw@rjwysocki.net, shameerali.kolothum.thodi@huawei.com, eric.auger@redhat.com, iommu@lists.linux-foundation.org, Jonathan.Cameron@huawei.com, sudeep.holla@arm.com, bhelgaas@google.com, zhangfei.gao@linaro.org, vivek.gautam@arm.com, baolu.lu@linux.intel.com, robin.murphy@arm.com, Jean-Philippe Brucker , linux-arm-kernel@lists.infradead.org, lenb@kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For PCI devices that support it, enable the PRI capability and handle PRI Page Requests with the generic fault handler. It is enabled on demand by iommu_sva_device_init(). Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 20 +- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 28 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 275 +++++++++++++++--- 3 files changed, 272 insertions(+), 51 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 124f604ed677..7c2d31133148 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -230,6 +230,7 @@ #define STRTAB_STE_1_S1COR GENMASK_ULL(5, 4) #define STRTAB_STE_1_S1CSH GENMASK_ULL(7, 6) +#define STRTAB_STE_1_PPAR (1UL << 18) #define STRTAB_STE_1_S1STALLD (1UL << 27) #define STRTAB_STE_1_EATS GENMASK_ULL(29, 28) @@ -360,6 +361,9 @@ #define CMDQ_PRI_0_SID GENMASK_ULL(63, 32) #define CMDQ_PRI_1_GRPID GENMASK_ULL(8, 0) #define CMDQ_PRI_1_RESP GENMASK_ULL(13, 12) +#define CMDQ_PRI_1_RESP_FAILURE 0UL +#define CMDQ_PRI_1_RESP_INVALID 1UL +#define CMDQ_PRI_1_RESP_SUCCESS 2UL #define CMDQ_RESUME_0_SID GENMASK_ULL(63, 32) #define CMDQ_RESUME_0_RESP_TERM 0UL @@ -427,12 +431,6 @@ #define MSI_IOVA_BASE 0x8000000 #define MSI_IOVA_LENGTH 0x100000 -enum pri_resp { - PRI_RESP_DENY = 0, - PRI_RESP_FAIL = 1, - PRI_RESP_SUCC = 2, -}; - struct arm_smmu_cmdq_ent { /* Common fields */ u8 opcode; @@ -494,7 +492,7 @@ struct arm_smmu_cmdq_ent { u32 sid; u32 ssid; u16 grpid; - enum pri_resp resp; + u8 resp; } pri; #define CMDQ_OP_RESUME 0x44 @@ -568,6 +566,9 @@ struct arm_smmu_evtq { struct arm_smmu_priq { struct arm_smmu_queue q; + struct iopf_queue *iopf; + u64 batch; + wait_queue_head_t wq; }; /* High-level stream table and context descriptor structures */ @@ -703,6 +704,8 @@ struct arm_smmu_master { unsigned int num_streams; bool ats_enabled; bool stall_enabled; + bool pri_supported; + bool prg_resp_needs_ssid; bool sva_enabled; struct list_head bonds; unsigned int ssid_bits; @@ -754,6 +757,9 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd); int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, unsigned long iova, size_t size); +int arm_smmu_enable_pri(struct arm_smmu_master *master); +void arm_smmu_disable_pri(struct arm_smmu_master *master); +int arm_smmu_flush_priq(struct arm_smmu_device *smmu); #ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported(struct arm_smmu_device *smmu); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 64e2082ef9ed..1fdfd40f70fd 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -370,6 +370,19 @@ arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, void *drvdata) void arm_smmu_sva_unbind(struct iommu_sva *handle) { struct arm_smmu_bond *bond = sva_to_bond(handle); + struct arm_smmu_master *master = dev_iommu_priv_get(handle->dev); + + /* + * For stall, the event queue does not need to be flushed since the + * device driver ensured all transaction are complete. For PRI however, + * although the device driver has stopped all DMA for this PASID, it may + * have left Page Requests in flight (if using the Stop Marker Message + * to stop PASID). Complete them. + */ + if (master->pri_supported) { + arm_smmu_flush_priq(master->smmu); + iopf_queue_flush_dev(handle->dev); + } mutex_lock(&sva_lock); if (refcount_dec_and_test(&bond->refs)) { @@ -435,7 +448,7 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) static bool arm_smmu_iopf_supported(struct arm_smmu_master *master) { - return master->stall_enabled; + return master->stall_enabled || master->pri_supported; } bool arm_smmu_master_sva_supported(struct arm_smmu_master *master) @@ -466,6 +479,15 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master) ret = iopf_queue_add_device(master->smmu->evtq.iopf, dev); if (ret) return ret; + } else if (master->pri_supported) { + ret = iopf_queue_add_device(master->smmu->priq.iopf, dev); + if (ret) + return ret; + + if (arm_smmu_enable_pri(master)) { + iopf_queue_remove_device(master->smmu->priq.iopf, dev); + return ret; + } } ret = iommu_register_device_fault_handler(dev, iommu_queue_iopf, dev); @@ -479,6 +501,8 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master) return 0; err_disable_iopf: + arm_smmu_disable_pri(master); + iopf_queue_remove_device(master->smmu->priq.iopf, dev); iopf_queue_remove_device(master->smmu->evtq.iopf, dev); return ret; } @@ -497,6 +521,8 @@ int arm_smmu_master_disable_sva(struct arm_smmu_master *master) mutex_unlock(&sva_lock); iommu_unregister_device_fault_handler(dev); + arm_smmu_disable_pri(master); + iopf_queue_remove_device(master->smmu->priq.iopf, dev); iopf_queue_remove_device(master->smmu->evtq.iopf, dev); return 0; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index d412d063d3b6..b4d58f3a77bc 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -315,14 +315,6 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) cmd[0] |= FIELD_PREP(CMDQ_PRI_0_SSID, ent->pri.ssid); cmd[0] |= FIELD_PREP(CMDQ_PRI_0_SID, ent->pri.sid); cmd[1] |= FIELD_PREP(CMDQ_PRI_1_GRPID, ent->pri.grpid); - switch (ent->pri.resp) { - case PRI_RESP_DENY: - case PRI_RESP_FAIL: - case PRI_RESP_SUCC: - break; - default: - return -EINVAL; - } cmd[1] |= FIELD_PREP(CMDQ_PRI_1_RESP, ent->pri.resp); break; case CMDQ_OP_RESUME: @@ -894,11 +886,12 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu, } static int arm_smmu_page_response(struct device *dev, - struct iommu_fault_event *unused, + struct iommu_fault_event *evt, struct iommu_page_response *resp) { struct arm_smmu_cmdq_ent cmd = {0}; struct arm_smmu_master *master = dev_iommu_priv_get(dev); + bool pasid_valid = resp->flags & IOMMU_PAGE_RESP_PASID_VALID; int sid = master->streams[0].id; if (master->stall_enabled) { @@ -916,6 +909,28 @@ static int arm_smmu_page_response(struct device *dev, default: return -EINVAL; } + } else if (master->pri_supported) { + bool needs_pasid = (evt->fault.prm.flags & + IOMMU_FAULT_PAGE_RESPONSE_NEEDS_PASID); + + cmd.opcode = CMDQ_OP_PRI_RESP; + cmd.substream_valid = needs_pasid && pasid_valid; + cmd.pri.sid = sid; + cmd.pri.ssid = resp->pasid; + cmd.pri.grpid = resp->grpid; + switch (resp->code) { + case IOMMU_PAGE_RESP_FAILURE: + cmd.pri.resp = CMDQ_PRI_1_RESP_FAILURE; + break; + case IOMMU_PAGE_RESP_INVALID: + cmd.pri.resp = CMDQ_PRI_1_RESP_INVALID; + break; + case IOMMU_PAGE_RESP_SUCCESS: + cmd.pri.resp = CMDQ_PRI_1_RESP_SUCCESS; + break; + default: + return -EINVAL; + } } else { return -ENODEV; } @@ -1336,6 +1351,9 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid, FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_ISH) | FIELD_PREP(STRTAB_STE_1_STRW, strw)); + if (master->prg_resp_needs_ssid) + dst[1] |= cpu_to_le64(STRTAB_STE_1_PPAR); + if (smmu->features & ARM_SMMU_FEAT_STALLS && !master->stall_enabled) dst[1] |= cpu_to_le64(STRTAB_STE_1_S1STALLD); @@ -1566,64 +1584,155 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) static void arm_smmu_handle_ppr(struct arm_smmu_device *smmu, u64 *evt) { - u32 sid, ssid; - u16 grpid; - bool ssv, last; - - sid = FIELD_GET(PRIQ_0_SID, evt[0]); - ssv = FIELD_GET(PRIQ_0_SSID_V, evt[0]); - ssid = ssv ? FIELD_GET(PRIQ_0_SSID, evt[0]) : 0; - last = FIELD_GET(PRIQ_0_PRG_LAST, evt[0]); - grpid = FIELD_GET(PRIQ_1_PRG_IDX, evt[1]); - - dev_info(smmu->dev, "unexpected PRI request received:\n"); - dev_info(smmu->dev, - "\tsid 0x%08x.0x%05x: [%u%s] %sprivileged %s%s%s access at iova 0x%016llx\n", - sid, ssid, grpid, last ? "L" : "", - evt[0] & PRIQ_0_PERM_PRIV ? "" : "un", - evt[0] & PRIQ_0_PERM_READ ? "R" : "", - evt[0] & PRIQ_0_PERM_WRITE ? "W" : "", - evt[0] & PRIQ_0_PERM_EXEC ? "X" : "", - evt[1] & PRIQ_1_ADDR_MASK); - - if (last) { - struct arm_smmu_cmdq_ent cmd = { - .opcode = CMDQ_OP_PRI_RESP, - .substream_valid = ssv, - .pri = { - .sid = sid, - .ssid = ssid, - .grpid = grpid, - .resp = PRI_RESP_DENY, - }, + bool pasid_valid, last; + struct arm_smmu_master *master; + u32 sid = FIELD_PREP(PRIQ_0_SID, evt[0]); + struct iommu_fault_event fault_evt = { + .fault.type = IOMMU_FAULT_PAGE_REQ, + .fault.prm = { + .grpid = FIELD_GET(PRIQ_1_PRG_IDX, evt[1]), + .addr = evt[1] & PRIQ_1_ADDR_MASK, + }, + }; + struct iommu_fault_page_request *pr = &fault_evt.fault.prm; + + pasid_valid = evt[0] & PRIQ_0_SSID_V; + last = evt[0] & PRIQ_0_PRG_LAST; + + /* Discard Stop PASID marker, it isn't used */ + if (!(evt[0] & (PRIQ_0_PERM_READ | PRIQ_0_PERM_WRITE)) && last) + return; + + if (last) + pr->flags |= IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE; + if (pasid_valid) { + pr->flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; + pr->pasid = FIELD_GET(PRIQ_0_SSID, evt[0]); + } + if (evt[0] & PRIQ_0_PERM_READ) + pr->perm |= IOMMU_FAULT_PERM_READ; + if (evt[0] & PRIQ_0_PERM_WRITE) + pr->perm |= IOMMU_FAULT_PERM_WRITE; + if (evt[0] & PRIQ_0_PERM_EXEC) + pr->perm |= IOMMU_FAULT_PERM_EXEC; + if (evt[0] & PRIQ_0_PERM_PRIV) + pr->perm |= IOMMU_FAULT_PERM_PRIV; + + master = arm_smmu_find_master(smmu, sid); + if (WARN_ON(!master)) + return; + + if (pasid_valid && master->prg_resp_needs_ssid) + pr->flags |= IOMMU_FAULT_PAGE_RESPONSE_NEEDS_PASID; + + if (iommu_report_device_fault(master->dev, &fault_evt)) { + /* + * No handler registered, so subsequent faults won't produce + * better results. Try to disable PRI. + */ + struct iommu_page_response resp = { + .flags = pasid_valid ? + IOMMU_PAGE_RESP_PASID_VALID : 0, + .pasid = pr->pasid, + .grpid = pr->grpid, + .code = IOMMU_PAGE_RESP_FAILURE, }; - arm_smmu_cmdq_issue_cmd(smmu, &cmd); + dev_warn(master->dev, + "PPR 0x%x:0x%llx 0x%x: nobody cared, disabling PRI\n", + pasid_valid ? pr->pasid : 0, pr->addr, pr->perm); + if (last) + arm_smmu_page_response(master->dev, NULL, &resp); } } static irqreturn_t arm_smmu_priq_thread(int irq, void *dev) { + int num_handled = 0; + bool overflow = false; struct arm_smmu_device *smmu = dev; - struct arm_smmu_queue *q = &smmu->priq.q; + struct arm_smmu_priq *priq = &smmu->priq; + struct arm_smmu_queue *q = &priq->q; struct arm_smmu_ll_queue *llq = &q->llq; + size_t queue_size = 1 << llq->max_n_shift; u64 evt[PRIQ_ENT_DWORDS]; + spin_lock(&priq->wq.lock); do { - while (!queue_remove_raw(q, evt)) + while (!queue_remove_raw(q, evt)) { + spin_unlock(&priq->wq.lock); arm_smmu_handle_ppr(smmu, evt); + spin_lock(&priq->wq.lock); + if (++num_handled == queue_size) { + priq->batch++; + wake_up_all_locked(&priq->wq); + num_handled = 0; + } + } - if (queue_sync_prod_in(q) == -EOVERFLOW) + if (queue_sync_prod_in(q) == -EOVERFLOW) { dev_err(smmu->dev, "PRIQ overflow detected -- requests lost\n"); + overflow = true; + } } while (!queue_empty(llq)); /* Sync our overflow flag, as we believe we're up to speed */ llq->cons = Q_OVF(llq->prod) | Q_WRP(llq, llq->cons) | Q_IDX(llq, llq->cons); queue_sync_cons_out(q); + + wake_up_all_locked(&priq->wq); + spin_unlock(&priq->wq.lock); + + /* + * On overflow, the SMMU might have discarded the last PPR in a group. + * There is no way to know more about it, so we have to discard all + * partial faults already queued. + */ + if (overflow) + iopf_queue_discard_partial(priq->iopf); + return IRQ_HANDLED; } +/* + * arm_smmu_flush_priq - wait until all events currently in the queue have been + * consumed. + * + * When unbinding a PASID, ensure there aren't any pending page requests for + * that PASID in the queue. + * + * Wait either that the queue becomes empty or, if new events are continually + * added the queue, that the event queue thread has handled a full batch (where + * one batch corresponds to the queue size). For that we take the batch number + * when entering flush() and wait for the event queue thread to increment it + * twice. Note that we don't handle overflows on q->batch. If it occurs, just + * wait for the queue to become empty. + */ +int arm_smmu_flush_priq(struct arm_smmu_device *smmu) +{ + int ret; + u64 batch; + bool overflow = false; + struct arm_smmu_priq *priq = &smmu->priq; + struct arm_smmu_queue *q = &priq->q; + + spin_lock(&priq->wq.lock); + if (queue_sync_prod_in(q) == -EOVERFLOW) { + dev_err(smmu->dev, "priq overflow detected -- requests lost\n"); + overflow = true; + } + + batch = priq->batch; + ret = wait_event_interruptible_locked(priq->wq, queue_empty(&q->llq) || + priq->batch >= batch + 2); + spin_unlock(&priq->wq.lock); + + if (overflow) + iopf_queue_discard_partial(priq->iopf); + return ret; +} + static int arm_smmu_device_disable(struct arm_smmu_device *smmu); static irqreturn_t arm_smmu_gerror_handler(int irq, void *dev) @@ -2336,6 +2445,73 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master) pci_disable_pasid(pdev); } +static int arm_smmu_init_pri(struct arm_smmu_master *master) +{ + struct pci_dev *pdev; + + if (!dev_is_pci(master->dev)) + return -EINVAL; + + if (!(master->smmu->features & ARM_SMMU_FEAT_PRI)) + return 0; + + pdev = to_pci_dev(master->dev); + if (!pci_pri_supported(pdev)) + return 0; + + /* If the device supports PASID and PRI, set STE.PPAR */ + if (master->ssid_bits) + master->prg_resp_needs_ssid = pci_prg_resp_pasid_required(pdev); + + master->pri_supported = true; + return 0; +} + +int arm_smmu_enable_pri(struct arm_smmu_master *master) +{ + int ret; + struct pci_dev *pdev; + /* + * TODO: find a good inflight PPR number. According to the SMMU spec we + * should divide the PRI queue by the number of PRI-capable devices, but + * it's impossible to know about future (probed late or hotplugged) + * devices. So we might miss some PPRs due to queue overflow. + */ + size_t max_inflight_pprs = 16; + + if (!master->pri_supported || !master->ats_enabled) + return -ENODEV; + + pdev = to_pci_dev(master->dev); + + ret = pci_reset_pri(pdev); + if (ret) + return ret; + + ret = pci_enable_pri(pdev, max_inflight_pprs); + if (ret) { + dev_err(master->dev, "cannot enable PRI: %d\n", ret); + return ret; + } + + return 0; +} + +void arm_smmu_disable_pri(struct arm_smmu_master *master) +{ + struct pci_dev *pdev; + + if (!dev_is_pci(master->dev)) + return; + + pdev = to_pci_dev(master->dev); + + if (!pdev->pri_enabled) + return; + + pci_disable_pri(pdev); +} + static void arm_smmu_detach_dev(struct arm_smmu_master *master) { unsigned long flags; @@ -2648,6 +2824,8 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) smmu->features & ARM_SMMU_FEAT_STALL_FORCE) master->stall_enabled = true; + arm_smmu_init_pri(master); + return &smmu->iommu; err_free_master: @@ -2666,6 +2844,7 @@ static void arm_smmu_release_device(struct device *dev) master = dev_iommu_priv_get(dev); WARN_ON(arm_smmu_master_sva_enabled(master)); + iopf_queue_remove_device(master->smmu->priq.iopf, dev); iopf_queue_remove_device(master->smmu->evtq.iopf, dev); arm_smmu_detach_dev(master); arm_smmu_disable_pasid(master); @@ -2981,6 +3160,15 @@ static int arm_smmu_init_queues(struct arm_smmu_device *smmu) if (!(smmu->features & ARM_SMMU_FEAT_PRI)) return 0; + if (sva) { + smmu->priq.iopf = iopf_queue_alloc(dev_name(smmu->dev)); + if (!smmu->priq.iopf) + return -ENOMEM; + } + + init_waitqueue_head(&smmu->priq.wq); + smmu->priq.batch = 0; + return arm_smmu_init_one_queue(smmu, &smmu->priq.q, ARM_SMMU_PRIQ_PROD, ARM_SMMU_PRIQ_CONS, PRIQ_ENT_DWORDS, "priq"); @@ -3944,6 +4132,7 @@ static int arm_smmu_device_remove(struct platform_device *pdev) iommu_device_sysfs_remove(&smmu->iommu); arm_smmu_device_disable(smmu); iopf_queue_free(smmu->evtq.iopf); + iopf_queue_free(smmu->priq.iopf); return 0; }