From patchwork Tue Jun 25 11:27:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jonathan Cameron X-Patchwork-Id: 11015357 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CA03A14E5 for ; Tue, 25 Jun 2019 11:34:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BAE7F289C6 for ; Tue, 25 Jun 2019 11:34:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AD4F228B7B; Tue, 25 Jun 2019 11:34:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 934A3289C6 for ; Tue, 25 Jun 2019 11:34:02 +0000 (UTC) Received: from localhost ([::1]:59026 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfji5-0004US-Tz for patchwork-qemu-devel@patchwork.kernel.org; Tue, 25 Jun 2019 07:34:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53746) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hfjdF-00074o-Lv for qemu-devel@nongnu.org; Tue, 25 Jun 2019 07:29:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hfjdE-0004F1-05 for qemu-devel@nongnu.org; Tue, 25 Jun 2019 07:29:01 -0400 Received: from szxga04-in.huawei.com ([45.249.212.190]:2178 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hfjd5-0003id-DU; Tue, 25 Jun 2019 07:28:55 -0400 Received: from DGGEMS406-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 12F7E70A72843ED135E0; Tue, 25 Jun 2019 19:28:38 +0800 (CST) Received: from lhrphicprd00229.huawei.com (10.123.41.22) by DGGEMS406-HUB.china.huawei.com (10.3.19.206) with Microsoft SMTP Server id 14.3.439.0; Tue, 25 Jun 2019 19:28:29 +0800 From: Jonathan Cameron To: QEMU Developers Date: Tue, 25 Jun 2019 19:27:45 +0800 Message-ID: <20190625112752.83188-1-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-Originating-IP: [10.123.41.22] X-CFilter-Loop: Reflected X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 45.249.212.190 Subject: [Qemu-devel] [RFC PATCH 0/7] qemu: CCIX pcie config space emulation X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , jcm@redhat.com, linuxarm@huawei.com, Auger Eric , qemu-arm , Jonathan Cameron Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP CCIX topologies are 'layered' on top of PCIe tree topologies. This is done primarily by allowing a single CCIX device to appear as multiple disjoint nodes in the PCIe tree. This layering is described via extensive PCIe DVSEC extended capabilities in PCIe config space across all the functions that are present in the device (some placement rules apply). The extremely flexible nature of allowed topologies makes the development of generic firmware and OS software difficult if we rely on actual hardware setups, due to the large test set that is necessary. To enable the ongoing work on EDK2 and within the Linux kernel and userspace, this series provides the bare bones of what is necessary to be able to test 'configuration' of a CCIX topology. Note that no actual CCIX data flow is being emulated within this patchset, merely a substantial subset of the interface that allows the topologies to be configured. Testing has to rely on verification based on the result rather than true emulation of the coherency protocol. (that is a very different form of emulation for which other tools are perhaps better suited). For information on how to do the coherency protocol configuration, see the forthcoming CCIX SW guide, in conjunction with the CCIX 1.0 Base Specification. An example of a 2x2 mesh with a spur to the host can be run with: qemu-system-aarch64 -M virt -m 1024 -cpu cortex-a53 \ ... -device ioh3420,id=root_port1 \ -device ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev1",bus=root_port1,addr=0.0,multifunction="on",id=up0,port_id=0 \ -device ccix-downstream-port,ccix_device="ccix_dev1",bus=up0,slot=0,chassis=2,id=bus_top,port_id=1 \ -device ccix-downstream-port,ccix_device="ccix_dev1",bus=up0,slot=1,chassis=2,id=bus_left,port_id=2 \ -device ccix-ep,primaryport=false,home_agents=1,request_agents=1,ccix_device="ccix_dev1",bus=root_port1,addr=0.1,multifunction="on" \ -device ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev2",bus=bus_top,addr=0.0,multifunction="on",id=up1,port_id=0 \ -device ccix-downstream-port,ccix_device="ccix_dev2",bus=up1,slot=0,chassis=3,id=bus_right,port_id=1 \ -device ccix-ep,primaryport=false,request_agents=2,ccix_device="ccix_dev2",bus=bus_top,addr=0.1,multifunction="on" \ -device ccix-upstream-port,num_links=4,primaryport=true,rsam_entries=4,ccix_device="ccix_dev3",bus=bus_left,addr=0.0,multifunction="on",id=up2,port_id=0 \ -device ccix-downstream-port,ccix_device="ccix_dev3",bus=up2,slot=0,chassis=4,id=bus_bottom,port_id=1,multifunciton="on" \ -device ccix-ep,primaryport=false,request_agents=2,ccix_device="ccix_dev3",bus=bus_left,addr=0.1,multifunction="on" \ -device ccix-ep,primaryport=true,request_agents=2,ccix_device="ccix_dev4",bus=bus_right,addr=0.0,port_id=0 \ -device ccix-ep,primaryport=true,request_agents=2,ccix_device="ccix_dev4",bus=bus_bottom,addr=0.0,port_id=1 I'm not going to try drawing all the detail (it's bad enough trying to draw these in inkscape, but in a very much simplifed fashion, this generates. ----------------- | Host | | | --- root_port1-- | | v ----------------- --------------- | ccix_dev1 | -------> | ccix_dev2 | ----------------- --------------- | | V V ----------------- --------------- | ccix_dev3 | -------> | ccix_dev4 | ----------------- --------------- $lspci -t -[0000:00]-+-00.0 +-01.0-[01-08]--+-00.0-[02-08]--+-00.0-[03-05]--+-00.0-[04-05]----00.0-[05]----00.0 | | | \-00.1 | | \-01.0-[06-08]--+-00.0-[07-08]----00.0-[08]----00.0 | | \-00.1 | \-00.1 RFC questions: 1. The nature of CCIX devices is that we need to extend normal PCI devices, slots, and ports. I could not find an elegant way of doing this without lots of code replication. The current solution just exposes some internal functions from xio3130 port implementations. Is there a better way to do this? 2. The association of the different PCIDevices to a given CCIX device is currently somewhat of a hack. Can anyone suggest a cleaner solution for this? I can improved the current implementation, but don't really like that we basically search for all the parts whenever a cross device implementation is needed. 3. Is emulation via a large number of PCIe devices like this a good approach or is there a nicer way to handle it? Given we need to be able to 'spread' the CCIX device configuration across multiple PCIe functions anyway perhaps this is the most sensible approach. 4. I've cut and paste a 100+ lines of code from each of the xio3130_* modules as we are also implemening PCIE up and downstream ports and as this is a emulation only device, we might as well use the same register set. There are various possible ways to avoid this: * Add a library with the shared code in it. * Add an xio3130_upstream.h header and similar to allow the CCIX port modules to call those functions directly. * Don't worry about the replication in the interests of keeping the code structure simple. 5. I'm not that familiar with qemu 'style' yet, so pointers on structures to use etc most welcome. Note that not all elements of CCIX are supported by the current implementation, for example slave agents and error records are missing. These will follow either in later revisions or as follow patches. We also have no actual accelerator functions in the current design and hence no mapping to RAs. Only a subset of configuration constraints are currently implemented. This will want tightenning up in future. As we don't have any actual chunks of the spec in here so I haven't added the statement from the trademark grant that follows. Thanks, Jonathan This patch is being distributed by the CCIX Consortium, Inc. (CCIX) to you and other parties that are paticipating (the "participants") in qemu with the understanding that the participants will use CCIX's name and trademark only when this patch is used in association with qemu. CCIX is also distributing this patch to these participants with the understanding that if any portion of the CCIX specification will be used or referenced in qemu, the participants will not modify the cited portion of the CCIX specification and will give CCIX propery copyright attribution by including the following copyright notice with the cited part of the CCIX specification: "© 2019 CCIX CONSORTIUM, INC. ALL RIGHTS RESERVED." Jonathan Cameron (7): Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy. pci: Add Huawei vendor ID and Huawei Emulated CCIX Device IDs. pci: CCIX config space emulation library. pci-bridge: CCIX capable PCIE/CCIX switch upstream port. pci-bridge: CCIX capable PCIE/CCIX switch downstream port misc: CCIX endpoint function Temp: Add to ARM64 makefiles for testing default-configs/arm-softmmu.mak | 3 +- hw/misc/Kconfig | 5 + hw/misc/Makefile.objs | 1 + hw/misc/ccix-ep.c | 112 ++ hw/pci-bridge/Kconfig | 5 + hw/pci-bridge/Makefile.objs | 1 + hw/pci-bridge/ccix_downstream.c | 222 ++++ hw/pci-bridge/ccix_upstream.c | 197 ++++ hw/pci/Kconfig | 3 + hw/pci/Makefile.objs | 1 + hw/pci/ccix_lib.c | 1299 +++++++++++++++++++++ include/hw/misc/ccix.h | 28 + include/hw/pci/pci_ids.h | 7 + include/standard-headers/linux/pci_regs.h | 3 +- 14 files changed, 1885 insertions(+), 2 deletions(-) create mode 100644 hw/misc/ccix-ep.c create mode 100644 hw/pci-bridge/ccix_downstream.c create mode 100644 hw/pci-bridge/ccix_upstream.c create mode 100644 hw/pci/ccix_lib.c create mode 100644 include/hw/misc/ccix.h