From patchwork Mon Feb 3 17:30:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Cameron X-Patchwork-Id: 13957940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0C00FC02197 for ; Mon, 3 Feb 2025 17:31:36 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tf0Hv-00088S-OM; Mon, 03 Feb 2025 12:31:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tf0HZ-00080G-NW; Mon, 03 Feb 2025 12:31:01 -0500 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tf0HX-0005UB-5L; Mon, 03 Feb 2025 12:30:49 -0500 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4YmtmH40d6z6K97v; Tue, 4 Feb 2025 01:29:51 +0800 (CST) Received: from frapeml500008.china.huawei.com (unknown [7.182.85.71]) by mail.maildlp.com (Postfix) with ESMTPS id D18A01400CA; Tue, 4 Feb 2025 01:30:41 +0800 (CST) Received: from SecurePC-101-06.china.huawei.com (10.122.19.247) by frapeml500008.china.huawei.com (7.182.85.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Mon, 3 Feb 2025 18:30:41 +0100 To: , Fan Ni , Peter Maydell CC: , , , , Yuquan Wang , Itaru Kitayama , =?utf-8?q?Philippe_Mathieu-Daud?= =?utf-8?q?=C3=A9?= Subject: [RFC PATCH v12 qemu 0/2] arm/virt: CXL support via pxb_cxl Date: Mon, 3 Feb 2025 17:30:38 +0000 Message-ID: <20250203173040.145763-1-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-Originating-IP: [10.122.19.247] X-ClientProxiedBy: lhrpeml500012.china.huawei.com (7.191.174.4) To frapeml500008.china.huawei.com (7.182.85.71) Received-SPF: pass client-ip=185.176.79.56; envelope-from=jonathan.cameron@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Jonathan Cameron X-Patchwork-Original-From: Jonathan Cameron via From: Jonathan Cameron Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Back in 2022, this series stalled on the absence of a solution to device tree support for PCI Expander Bridges (PXB) and we ended up only having x86 support upstream. I've been carrying the arm64 support out of tree since then, with occasional nasty surprises (e.g. UNIMP + DT issue seen a few weeks ago) and a fair number of fiddly rebases. gitlab.com/jic23/qemu cxl- A recent discussion with Peter Maydell indicated that there are various other ACPI only features now, so in general he might be more relaxed about DT support being necessary. The upcoming vSMMUv3 support would run into this problem as well. I presented the background to the PXB issue at Linaro connect 2022. In short the issue is that PXBs steal MMIO space from the main PCI root bridge. The challenge is knowing how much to steal. On ACPI platforms, we can rely on EDK2 to perform an enumeration and configuration of the PCI topology and QEMU can update the ACPI tables after EDK2 has done this when it can simply read the space used by the root ports. On device tree, there is no entity to figure out that enumeration so we don't know how to size the stolen region. Three approaches were discussed: 1) Enumerating in QEMU. Horribly complex and the last thing we want is a 3rd enumeration implementation that ends up out of sync with EDK2 and the kernel (there are frequent issues because of how those existing implementations differ. 2) Figure out how to enumerate in kernel. I never put a huge amount of work into this, but it seemed likely to involve a nasty dance with similar very specific code to that EDK2 is carrying and would very challenging to upstream (given the lack of clarity on real use cases for PXBs and DT). 3) Hack it based on the control we have which is bus numbers. No one liked this but it worked :) The other little wrinkle would be the need to define full bindings for CXL on DT + implement a fairly complex kernel stack as equivalent in ACPI involves a static table, CEDT, new runtime queries via _DSM and a description of various components. Doable, but so far there is no interest on physical platforms. Worth noting that for now, the QEMU CXL emulation is all about testing and developing the OS stack, not about virtualization (performance is terrible except in some very contrived situations!) Back to posting as an RFC because there was some discussion of approach to modelling the devices that may need a bit of redesign. The discussion kind of died out on the back of DT issue and I doubt anyone can remember the details. https://lore.kernel.org/qemu-devel/20220616141950.23374-1-Jonathan.Cameron@huawei.com/ There is only a very simple test in here, because my intent is not to duplicate what we have on x86, but just to do a smoke test that everything is hooked up. In general we need much more comprehensive end to end CXL tests but that requires a reaonsably stable guest software stack. A few people have expressed interest in working on that, but we aren't there yet. Note that this series has a very different use case to that in the proposed SBSA-ref support: https://lore.kernel.org/qemu-devel/20250117034343.26356-1-wangyuquan1236@phytium.com.cn/ SBSA-ref is a good choice if you want a relatively simple mostly fixed configuration. That works well with the limited host system discoverability etc as EDK2 can be build against a known configuration. My interest with this support in arm/virt is support host software stack development (we have a wide range of contributors, most of whom are working on emulation + the kernel support). I care about the weird corners. As such I need to be able to bring up variable numbers of host bridges, multiple CXL Fixed Memory Windows with varying characteristics (interleave etc), complex NUMA topologies with wierd performance characteristics etc. We can do that on x86 upstream today, or my gitlab tree. Note that we need arm support for some arch specific features in the near future (cache flushing). Doing kernel development with this need for flexibility on SBSA-ref is not currently practical. SBSA-ref CXL support is an excellent thing, just not much use to me for this work. Thanks, Jonathan Jonathan Cameron (2): hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances pxb-cxl qtest/cxl: Add aarch64 virt test for CXL include/hw/arm/virt.h | 4 +++ hw/arm/virt-acpi-build.c | 34 +++++++++++++++++++++++ hw/arm/virt.c | 54 ++++++++++++++++++++++++++++++++++++ tests/qtest/cxl-test.c | 59 +++++++++++++++++++++++++++++++--------- tests/qtest/meson.build | 1 + 5 files changed, 139 insertions(+), 13 deletions(-)