From patchwork Mon Nov 27 07:32:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergiy Kibrik X-Patchwork-Id: 13469252 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D3D79C4167B for ; Mon, 27 Nov 2023 07:34:42 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.641645.1000389 (Exim 4.92) (envelope-from ) id 1r7W8T-0007xh-11; Mon, 27 Nov 2023 07:34:29 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 641645.1000389; Mon, 27 Nov 2023 07:34:28 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1r7W8S-0007xa-TG; Mon, 27 Nov 2023 07:34:28 +0000 Received: by outflank-mailman (input) for mailman id 641645; Mon, 27 Nov 2023 07:34:27 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1r7W8R-0007xT-Cu for xen-devel@lists.xenproject.org; Mon, 27 Nov 2023 07:34:27 +0000 Received: from pb-smtp21.pobox.com (pb-smtp21.pobox.com [173.228.157.53]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 63b25de3-8cf7-11ee-98e2-6d05b1d4d9a1; Mon, 27 Nov 2023 08:34:26 +0100 (CET) Received: from pb-smtp21.pobox.com (unknown [127.0.0.1]) by pb-smtp21.pobox.com (Postfix) with ESMTP id 62F701A588; Mon, 27 Nov 2023 02:34:24 -0500 (EST) (envelope-from sakib@darkstar.site) Received: from pb-smtp21.sea.icgroup.com (unknown [127.0.0.1]) by pb-smtp21.pobox.com (Postfix) with ESMTP id 5A3391A586; Mon, 27 Nov 2023 02:34:24 -0500 (EST) (envelope-from sakib@darkstar.site) Received: from localhost (unknown [188.163.75.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by pb-smtp21.pobox.com (Postfix) with ESMTPSA id E7D621A585; Mon, 27 Nov 2023 02:34:20 -0500 (EST) (envelope-from sakib@darkstar.site) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 63b25de3-8cf7-11ee-98e2-6d05b1d4d9a1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=pobox.com; h=from:to:cc :subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; s=sasl; bh=F3YC9QsVnSSuRXfmGqVQj/is4 CeJzEOPdZWm5MoPMAk=; b=ePUv65SniNxzAG8cD18UU4fzCVBq0E3UPLwcXQp3D eWpixnPBs9YvRVtDVM1APsisMh1AB8cOeWDSRsbYa0bGHOY50DSUhaymCAzscVlw gk+U6f0yeQzQUdDAgvwi4AcUPqjER4N2DH08Jwog8vf2ABnoMhbWcwGC1LeBk+7C mg= From: Sergiy Kibrik To: xen-devel@lists.xenproject.org, Julien Grall Cc: Oleksandr Tyshchenko , Stefano Stabellini , Volodymyr Babchuk , Stewart Hildebrand , Juergen Gross , Wei Liu , Anthony PERARD , Sergiy Kibrik Subject: [RFC PATCH v1 4/5] libxl/arm: Reuse generic PCI-IOMMU bindings for virtio-pci devices Date: Mon, 27 Nov 2023 09:32:20 +0200 Message-Id: <20231127073221.4046427-1-Sergiy_Kibrik@epam.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231127072754.4045254-1-Sergiy_Kibrik@epam.com> References: <20231127072754.4045254-1-Sergiy_Kibrik@epam.com> MIME-Version: 1.0 X-Pobox-Relay-ID: 6124FFEC-8CF7-11EE-B321-A19503B9AAD1-90055647!pb-smtp21.pobox.com From: Oleksandr Tyshchenko Use the same "xen-grant-dma" device concept for the PCI devices behind device-tree based PCI Host controller, but with one modification. Unlike for platform devices, we cannot use generic IOMMU bindings (iommus property), as we need to support more flexible configuration. The problem is that PCI devices under the single PCI Host controller may have the backends running in different Xen domains and thus have different endpoints ID (backend domains ID). Reuse generic PCI-IOMMU bindings (iommu-map/iommu-map-mask properties) which allows us to describe relationship between PCI devices and backend domains ID properly. Linux guest is already able to deal with generic PCI-IOMMU bindings (see Linux drivers/xen/grant-dma-ops.c for details). According to Linux: - Documentation/devicetree/bindings/pci/pci-iommu.txt - Documentation/devicetree/bindings/iommu/xen,grant-dma.yaml Signed-off-by: Oleksandr Tyshchenko Signed-off-by: Sergiy Kibrik --- tools/libs/light/libxl_arm.c | 64 ++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c index 4789ee08d2..c1e39d688a 100644 --- a/tools/libs/light/libxl_arm.c +++ b/tools/libs/light/libxl_arm.c @@ -1030,6 +1030,7 @@ static int make_vpci_node(libxl__gc *gc, void *fdt, } #define PCI_IRQ_MAP_MIN_STRIDE 8 +#define PCI_IOMMU_MAP_STRIDE 4 static int create_virtio_pci_irq_map(libxl__gc *gc, void *fdt, libxl_virtio_pci_host *host) @@ -1099,6 +1100,65 @@ static int create_virtio_pci_irq_map(libxl__gc *gc, void *fdt, return 0; } +/* XXX Consider reusing libxl__realloc() to avoid an extra loop */ +static int create_virtio_pci_iommu_map(libxl__gc *gc, void *fdt, + libxl_virtio_pci_host *host, + libxl_domain_config *d_config) +{ + uint32_t *full_iommu_map, *iommu_map; + unsigned int i, len, ntranslated = 0; + int res; + + for (i = 0; i < d_config->num_virtios; i++) { + libxl_device_virtio *virtio = &d_config->virtios[i]; + + if (libxl_defbool_val(virtio->grant_usage) && + virtio->transport == LIBXL_VIRTIO_TRANSPORT_PCI && + virtio->u.pci.host_id == host->id) { + ntranslated++; + } + } + + if (!ntranslated) + return 0; + + len = ntranslated * sizeof(uint32_t) * PCI_IOMMU_MAP_STRIDE; + full_iommu_map = libxl__malloc(gc, len); + iommu_map = full_iommu_map; + + /* See Linux Documentation/devicetree/bindings/pci/pci-iommu.txt */ + for (i = 0; i < d_config->num_virtios; i++) { + libxl_device_virtio *virtio = &d_config->virtios[i]; + + if (libxl_defbool_val(virtio->grant_usage) && + virtio->transport == LIBXL_VIRTIO_TRANSPORT_PCI && + virtio->u.pci.host_id == host->id) { + uint16_t bdf = (virtio->u.pci.bus << 8) | + (virtio->u.pci.dev << 3) | virtio->u.pci.func; + unsigned int j = 0; + + /* rid_base (1 cell) */ + iommu_map[j++] = cpu_to_fdt32(bdf); + + /* iommu_phandle (1 cell) */ + iommu_map[j++] = cpu_to_fdt32(GUEST_PHANDLE_IOMMU); + + /* iommu_base (1 cell) */ + iommu_map[j++] = cpu_to_fdt32(virtio->backend_domid); + + /* length (1 cell) */ + iommu_map[j++] = cpu_to_fdt32(1 << 3); + + iommu_map += PCI_IOMMU_MAP_STRIDE; + } + } + + res = fdt_property(fdt, "iommu-map", full_iommu_map, len); + if (res) return res; + + return 0; +} + /* TODO Consider reusing make_vpci_node() */ static int make_virtio_pci_node(libxl__gc *gc, void *fdt, libxl_virtio_pci_host *host, @@ -1147,6 +1207,10 @@ static int make_virtio_pci_node(libxl__gc *gc, void *fdt, res = create_virtio_pci_irq_map(gc, fdt, host); if (res) return res; + /* xen,grant-dma bindings */ + res = create_virtio_pci_iommu_map(gc, fdt, host, d_config); + if (res) return res; + res = fdt_end_node(fdt); if (res) return res;