From patchwork Wed Feb 2 14:09:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jonathan Cameron X-Patchwork-Id: 12733005 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F199FC433F5 for ; Wed, 2 Feb 2022 14:10:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240122AbiBBOKj (ORCPT ); Wed, 2 Feb 2022 09:10:39 -0500 Received: from frasgout.his.huawei.com ([185.176.79.56]:4617 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231950AbiBBOKi (ORCPT ); Wed, 2 Feb 2022 09:10:38 -0500 Received: from fraeml701-chm.china.huawei.com (unknown [172.18.147.206]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Jpk9H5rm0z67xDf; Wed, 2 Feb 2022 22:05:55 +0800 (CST) Received: from lhreml710-chm.china.huawei.com (10.201.108.61) by fraeml701-chm.china.huawei.com (10.206.15.50) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2308.21; Wed, 2 Feb 2022 15:10:36 +0100 Received: from SecurePC-101-06.china.huawei.com (10.122.247.231) by lhreml710-chm.china.huawei.com (10.201.108.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.21; Wed, 2 Feb 2022 14:10:36 +0000 From: Jonathan Cameron To: , =?utf-8?q?Alex_Benn=C3=A9e?= , Marcel Apfelbaum , "Michael S . Tsirkin" , Igor Mammedov CC: , Ben Widawsky , "Peter Maydell" , , "Shameerali Kolothum Thodi" , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Saransh Gupta1 , Shreyas Shah , Chris Browy , Samarth Saxena , "Dan Williams" Subject: [PATCH v5 00/43] CXl 2.0 emulation Support Date: Wed, 2 Feb 2022 14:09:54 +0000 Message-ID: <20220202141037.17352-1-Jonathan.Cameron@huawei.com> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 X-Originating-IP: [10.122.247.231] X-ClientProxiedBy: lhreml743-chm.china.huawei.com (10.201.108.193) To lhreml710-chm.china.huawei.com (10.201.108.61) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Changes since v4: https://lore.kernel.org/linux-cxl/20220124171705.10432-1-Jonathan.Cameron@huawei.com/ Note documentation patch that Alex requested to follow. I don't want to delay getting this out as Alex mentioned possibly having time to continue reviewing in latter part of this week. Issues identified by CI / Alex Bennée - Stubs added for hw/cxl/cxl-host and hw/acpi/cxl plus related meson changes to use them as necessary. - Drop uid from cxl-test (result of last minute change in v4 that was not carried through to the test) - Fix naming clash with field name ERROR which on some arches is defined and results in the string being replaced with 0 in some of the register field related defines. Call it ERR instead. - Fix type issue around mr->size by using 64 bit acessor functions. - Add a new patch to exclude pxb-cxl from device-crash-test in similar fashion to pxb. CI tests now passing with exception of checkpatch which has what I think is a false positive and build-oss-fuzz which keeps timing out. https://gitlab.com/jic23/qemu/-/pipelines/460109208 There were a few tweaks to patch descriptions after I pushed that out (I missed a few RB from Alex). Other changes (mostly from Alex's review) - Change component register handling to now report UNIMP and return 0 for 8 byte registers as we currently don't implement any of them. Note that this means we need a kernel fix: https://lore.kernel.org/linux-cxl/20220201153437.2873-1-Jonathan.Cameron@huawei.com/ - Drop majority of the macros used in defining mailbox handlers in favour of written out code. - Use REG64 where appropriate. This was introduced whilst this set has been underdevelopment so I missed it. - Clarify some register access options wrt to CXL 2.0 Errata F4. - Change timestamp to qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) - Use typed enums to enforce types of function arguements. - Default to cxl being off in machine_class_init() removing need to set it to off in machines where there is no support as yet. - Add Alex's RB where given. Looking in particular for: * Review of the PCI interactions * x86 and ARM machine interactions (particularly the memory maps) * Review of the interleaving approach - is the basic idea acceptable? * Review of the command line interface. * CXL related review welcome but much of that got reviewed in earlier versions and hasn't changed substantially. Big TODOs: * Interleave boundary issues. I haven't yet solved this but didn't want to futher delay the review of the rest of the series. * Volatile memory devices (easy but it's more code so left for now). * Switch support. Linux kernel support is under review currently, so there is now something to test against. * Hotplug? May not need much but it's not tested yet! * More tests and tighter verification that values written to hardware are actually valid - stuff that real hardware would check. * Testing, testing and more testing. I have been running a basic set of ARM and x86 tests on this, but there is always room for more tests and greater automation. * CFMWS flags as requested by Ben. Why do we want QEMU emulation of CXL? As Ben stated in V3, QEMU support has been critical to getting OS software written given lack of availability of hardware supporting the latest CXL features (coupled with very high demand for support being ready in a timely fashion). What has become clear since Ben's v3 is that situation is a continuous one. Whilst we can't talk about them yet, CXL 3.0 features and OS support have been prototyped on top of this support and a lot of the ongoing kernel work is being tested against these patches. The kernel CXL mocking code allows some forms of testing, but QEMU provides a more versatile and exensible platform. Other features on the qemu-list that build on these include PCI-DOE /CDAT support from the Avery Design team further showing how this code is useful. Whilst not directly related this is also the test platform for work on PCI IDE/CMA + related DMTF SPDM as CXL both utilizes and extends those technologies and is likely to be an early adopter. Refs: CMA Kernel: https://lore.kernel.org/all/20210804161839.3492053-1-Jonathan.Cameron@huawei.com/ CMA Qemu: https://lore.kernel.org/qemu-devel/1624665723-5169-1-git-send-email-cbrowy@avery-design.com/ DOE Qemu: https://lore.kernel.org/qemu-devel/1623329999-15662-1-git-send-email-cbrowy@avery-design.com/ As can be seen there is non trivial interaction with other areas of Qemu, particularly PCI and keeping this set up to date is proving a burden we'd rather do without :) Ben mentioned a few other good reasons in v3: https://lore.kernel.org/qemu-devel/20210202005948.241655-1-ben.widawsky@intel.com/ The evolution of this series perhaps leave it in a less than entirely obvious order and that may get tidied up in future postings. I'm also open to this being considered in bite sized chunks. What we have here is about what you need for it to be useful for testing currently kernel code. Note the kernel code is moving fast so since v4, some features have been introduced we don't yet support in QEMU (e.g. use of the PCIe serial number extended capability). All comments welcome. qemu-system-aarch64 -M virt,gic-version=3,cxl=on \ -m 4g,maxmem=8G,slots=8 \ ... -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \ -object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M \ -object memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M \ -object memory-backend-file,id=cxl-mem4,share=on,mem-path=/tmp/cxltest4.raw,size=256M \ -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \ -object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M \ -object memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M \ -object memory-backend-file,id=cxl-lsa4,share=on,mem-path=/tmp/lsa4.raw,size=256M \ -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ -device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0,size=256M \ -device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \ -device cxl-type3,bus=root_port14,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1,size=256M \ -device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \ -device cxl-type3,bus=root_port15,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2,size=256M \ -device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \ -device cxl-type3,bus=root_port16,memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3,size=256M \ -cxl-fixed-memory-window targets=cxl.1,size=4G,interleave-granularity=8k \ -cxl-fixed-memory-window targets=cxl.1,targets=cxl.2,size=4G,interleave-granularity=8k First CFMWS suitable for up to 2 way interleave, the second for 4 way (2 way at host level and 2 way at the host bridge). targets= , multiple entries if range is disjoint. With the v5.17-rc1 + patch series listed below. cd /sys/bus/cxl/devices/ region=$(cat decoder0.1/create_region) echo $region > decoder0.1/create_region ls -lh //Note the order of devices and adjust the following to make sure they //are in order across the 4 root ports. Easy to do in a tool, but //not easy to paste in a cover letter. cd region0.1\:0 echo 4 > interleave_ways echo mem2 > target0 echo mem3 > target1 echo mem0 > target2 echo mem1 > target3 echo $((1024<<20)) > size echo 4096 > interleave_granularity echo region0.1:0 > /sys/bus/cxl/drivers/cxl_region/bind Tested with devmem2 and files with known content. Kernel tree is mainline + (I based on 5.17-rc1) [PATCH] cxl/regs: Fix size of CXL Capabilty Header Register https://lore.kernel.org/linux-cxl/20220201182934.jjvavjsf4h7oqngv@intel.com/T/#t [PATCH v3 00/40] CXL.mem Topology Discovery and Hotplug Support https://lore.kernel.org/linux-cxl/164298411792.3018233.7493009997525360044.stgit@dwillia2-desk3.amr.corp.intel.com/ Note that series has a lot of v4/v5 patches are replies but b4 does a good job of pulling out the latest. [PATCH 0/2] cxl/port: Robustness fixes for decoder enumeration https://lore.kernel.org/linux-cxl/164317463887.3438644.4087819721493502301.stgit@dwillia2-desk3.amr.corp.intel.com/ [PATCH 0/4] Unify meaning of interleave attributes https://lore.kernel.org/linux-cxl/20220127212911.127741-1-ben.widawsky@intel.com/ [PATCH v3 00/14] CXL Region driver https://lore.kernel.org/linux-cxl/20220128002707.391076-1-ben.widawsky@intel.com/ What follows is a first attempt at explaining how all these components fit together. I'll write up some formal documentation shortly. Memory Address Map for CXL elements. Note where exactly these regions appear is Arch and platform dependent. Base somewhere far up in the Host PA map.