From patchwork Fri Mar 15 09:36:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?J=C3=B8rgen_Hansen?= X-Patchwork-Id: 13593204 Received: from esa3.hgst.iphmx.com (esa3.hgst.iphmx.com [216.71.153.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F372717C66 for ; Fri, 15 Mar 2024 09:36:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=216.71.153.141 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710495389; cv=fail; b=KoTUlxJxZGigVknWyf7JAXcom3a+5cr/ShQ7RbGlt/LGRqwHpLK4DM6BK3YAuJ/SKBA10BKTbbv8j2AbmBsO3VzDrsij8sClUYFqgKWU2kfDTBWHskz5s0U+DReNQtsLG6Zloouh/MwmR70XndoPg7a/qNzN7DYU7ldoEWo2/qI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710495389; c=relaxed/simple; bh=/gV9w7K+UxXbMJgFMjMQxOAsPyVcPJGHbxV8Wp1W8XY=; h=From:To:CC:Subject:Date:Message-ID:Content-Type:MIME-Version; b=dVMEX5ZjF09vFMnEPINn3BOLsz6ZLCe/36XmYz7rwGm5smYtuf2iBo8Fi2dw+FTHD7Z+/GALo5b+Lbl/za/VeZe5Ctyite5pXuR4AGIjQ2aCcjC2ZlDFkxxoldSo5Te3yhEBOHgnZdHFWcs9pkrebKWBhEkna8O+11a7BYmRS8A= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=wdc.com; spf=pass smtp.mailfrom=wdc.com; dkim=pass (2048-bit key) header.d=wdc.com header.i=@wdc.com header.b=Ru8YsYuC; dkim=pass (1024-bit key) header.d=sharedspace.onmicrosoft.com header.i=@sharedspace.onmicrosoft.com header.b=LBeaxjkN; arc=fail smtp.client-ip=216.71.153.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=wdc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=wdc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=wdc.com header.i=@wdc.com header.b="Ru8YsYuC"; dkim=pass (1024-bit key) header.d=sharedspace.onmicrosoft.com header.i=@sharedspace.onmicrosoft.com header.b="LBeaxjkN" DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1710495387; x=1742031387; h=from:to:cc:subject:date:message-id:content-id: content-transfer-encoding:mime-version; bh=/gV9w7K+UxXbMJgFMjMQxOAsPyVcPJGHbxV8Wp1W8XY=; b=Ru8YsYuCUlXE0wV3vWG3MtZNZzhfsxR+WJeoqLOdRrcqbcPGVG6K5pue bm8xSWFU+AOyefC7UIJjzIozMbADcWmj336opNpKeoTpPQ0e0mb7M+kc1 du0crlq9Ai/IuPxpoIgOrBNLi6rfSW9ECwd4lBZ2xUlKWgxsxX3W8Xnxt fVoCGpr+3hJshMp45Qs6wTwERpfLUokId85BIPlKGIxycV0cv+C0aew8S 0Wt2ra1hBhjxcedAo28Gt34QzFudzaLydzGDJNgQVfvX4hof1abOBKmeA xtYugpA/TMe0C6rmV0v5ghuTcYR/PRLWUGbxEdCAjJXjxSDl5qzMbnmS6 A==; X-CSE-ConnectionGUID: L5mkrpWsT1q1pG7MPsrRwQ== X-CSE-MsgGUID: wAJpL+3HSvep4hNJHvcYOw== X-IronPort-AV: E=Sophos;i="6.07,128,1708358400"; d="scan'208";a="11511677" Received: from mail-dm6nam12lp2169.outbound.protection.outlook.com (HELO NAM12-DM6-obe.outbound.protection.outlook.com) ([104.47.59.169]) by ob1.hgst.iphmx.com with ESMTP; 15 Mar 2024 17:36:20 +0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LsLYBwFvcdZOpTRMi31nxB5wBmQ8l8Fm3zaJ+MHXn10hD9e+c045c5aUB0QEpkbJGrMdGLqi88IH9v7RTnxYqTZP93sRApWOTy4PKj3y6KcWe4qNCNad5EQV86wI1xOyKTmuAAx+nOFsDicaxAsfhUbWTnsToQ7mjJ9NsGt2yYm6KBuMJ9zs4SLN1RRHhGTX5I3G5yb3UE19GikO4ERPotV1sCAkVwUfsGKhXAOLBJKXZa36mVrDuAYzInkdNmxKIRSvqHEjL8/CjBwD98uPKHlUrin/aC9tbnNjAXD2dWwxqMMLrUhgsTGgJwB+vHQLsKqVIKsxvyeWkP1Q6LZVWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/gV9w7K+UxXbMJgFMjMQxOAsPyVcPJGHbxV8Wp1W8XY=; b=GbxULhgnaehHr4z7Mbf19LgtWPW3VRggD1TQxBczE8n7DPCH4yPTg7fcFjtDDk6gdlFfQtrjykx3sCgb9BYVfp4GkXzJrIB2EZwcYkkhA6hnWuys7ILignsflutumOzIqPLD79WC6US95+8HYUZR6oo0y6PUfg0t+RyO5PlxvzMgPtsxIK37CoVgJJVsB+rWtLN/LVx5KLVvWST6FNnxbJUi1os9Ioi7rPWyaZr0g+ppGJqRDdmezqNHpgcgw8uuihHYrKpo3HNijhkis3sFkQKB7vI0rwuLOpLRNVsWQOd4OKkVZ9pYl+diVkiUoW+GJOPF3jc+EhlLWZTuFkblCg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=wdc.com; dmarc=pass action=none header.from=wdc.com; dkim=pass header.d=wdc.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sharedspace.onmicrosoft.com; s=selector2-sharedspace-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/gV9w7K+UxXbMJgFMjMQxOAsPyVcPJGHbxV8Wp1W8XY=; b=LBeaxjkNNaQvUUitBsYufuVEhA2GBVzyZKqzhtSMO0HvgYBND3QdG/ncAss3v1FcYmCzd/r8yhBSwE5vv335pJDj8b+s6eXp4HvMNkqKU1sjxjFjd4l8ESkGwSHoDAJp0/63HF68u2khlDLHbUoExTk9HdGtXiDhjk1A6Y2uUaU= Received: from BYAPR04MB5431.namprd04.prod.outlook.com (2603:10b6:a03:ce::16) by PH0PR04MB7526.namprd04.prod.outlook.com (2603:10b6:510:58::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7386.22; Fri, 15 Mar 2024 09:36:17 +0000 Received: from BYAPR04MB5431.namprd04.prod.outlook.com ([fe80::2ab2:43a3:658b:b8c9]) by BYAPR04MB5431.namprd04.prod.outlook.com ([fe80::2ab2:43a3:658b:b8c9%7]) with mapi id 15.20.7386.021; Fri, 15 Mar 2024 09:36:16 +0000 From: =?utf-8?q?J=C3=B8rgen_Hansen?= To: "qemu-devel@nongnu.org" , "peter.maydell@linaro.org" , "richard.henderson@linaro.org" , Jonathan Cameron CC: "linux-cxl@vger.kernel.org" Subject: Another CXL/MMIO tcg tlb corner case Thread-Topic: Another CXL/MMIO tcg tlb corner case Thread-Index: AQHadrw6KncL+X/w106el4Z9yvqctA== Date: Fri, 15 Mar 2024 09:36:16 +0000 Message-ID: <33748BB7-E617-4661-BDE3-5D29780FC9FF@wdc.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=wdc.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: BYAPR04MB5431:EE_|PH0PR04MB7526:EE_ x-ms-office365-filtering-correlation-id: 23242598-8316-48d1-0193-08dc44d35cfc wdcipoutbound: EOP-TRUE x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: HMYNOezN/4++ezSt/tsyJRPU/HQ6Soknqyzh4g4/rMNYYDdYHM3mN8wHg6JrmzJstjAjTP9LthkJJusxOPJUkaDTDVOgBXbk/EVh6fgpbkX+y3uDMjI+waZ6ibHZGoTj5jgQiZeunOEnDRgSSDVlB4RRdueJFjritsQxLMDTQqlvVUfhICwUjOPwZPp9fMLv54BTXMOBrTKNCLd8Pt2G7+8Goy3WBHSY6pIHSzNYwhCqcSoPsnRPrm79a1l4P2HPzt4X4WtmSrNiOU9z5i34jfyBFOsr+6J2YNqCi+2L0/xYjeshbGwBHNmpMkRFJilfhaufI/+aWAXW7y6kgYn/QyiGQ4ymbJqizg34IAb2phqyaRr1cqxzZOJVYlXpStAdK9KdTfxsjogtfm4hF6kma8QorRignHeVPA3ZDS21usN1ty37cwqhzcyhgJYsqIjqqUgi4Nfi6u1UoIND/1NXVWYOEXSDVqLe/SpWknfz5nBjTEiH8EhMRVcXUCLIrEQR40ML4ZF67pFPqTt1T+lHIZZrgEEaHB3Gp9LxY75HkxpCTbGwIIgMNMq7+EKce3oOou2i05OJC6ZtK/oRksqx2jnM3sxgL0+z5sCEm+ejx/gQ5xuJafWy2ZXJ17vqADpY1ftUYvSsGn1e9ryclERQUHWCXd1qAjEy8ftHxoTWSPk= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR04MB5431.namprd04.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366007)(1800799015)(376005)(38070700009);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?q?W6Imx1QQ88gLfJTnM/2IZ8/xUZ9V?= =?utf-8?q?PUMiMhscCdScHtfS46Z/zh05lb7kUkZqZ3/UiwNh5D5CBBqeP/gNfTUs/PmloTOrW?= =?utf-8?q?ixu8foovB/DBzqxPRo+547lnid66bYuEmYYTeAhXSqf714Cj39WJnhw8M6MmMWFv+?= =?utf-8?q?Tof73jqYOK4j2DQnSZK0PSMAc21zG/z4osQeGUGKsyNJEnL/zmDf+yrIFeSSky6wv?= =?utf-8?q?TdWGi4i4t2IMDf81GfRWdqB+f2FZptSdTUIlu42Mrbxvi3tTk0GVU0nRfl/ZCR995?= =?utf-8?q?rRu4FDTsMSwI1j1poYOpzGPwNuPr01ojW/2PAzFsHOIejc6MqsaI5BSZk2xeWhIxV?= =?utf-8?q?4i6tFJXkkM+qF7lEoNcxzFgYnBo4sN7m0dcjOsRfSEQEgip2ToY35GZbY8YjTKMJs?= =?utf-8?q?K1bQ2Po/K00z1iYJzIanIH1rJBu1zldkPFi7vh3WwOstxHDV6ChCZJ9cnyixn/ijA?= =?utf-8?q?DAn3qNECu1kOAg/ymDe796GUQaP41yhwFmmjui2BzYgGnVXWKzRYfEeo32QtIQG8U?= =?utf-8?q?7maEKW1sGVBN8skd+sydS8SNbpI5p4yItM4eJ8W5dWod0Mk3FhH0/rrov9SAd+WOs?= =?utf-8?q?3JLWAQeHPKF3/7TdYBqOexWUrvAHFkzFoRfcOJUL/4FQBnQGcQMXGpjvxlB4VMRCd?= =?utf-8?q?HMz2rXuyKGPjRwqrQVDyTtIzfeGAD3LXERYixl7j0uVKyYmnuV6NJzwCc6iBrehx3?= =?utf-8?q?dNGUKgopz75D83uFwTFOAvAA8gAwCJ1zVtUDQeuyE+lOxjHWKspgn2LV7GqSKY+gV?= =?utf-8?q?P5YS1QrEuH4lDzmrfVF7wMxNo/LDz1QHCL/Uew5VMxr8dJqStR5s0CTM3zv6kmGIG?= =?utf-8?q?hQ/rVPoXK9WF5dPKkhgOjpM9goi5MdEvmRoLt5t+PVxmQwn7Hlp0IxePfw37dam6n?= =?utf-8?q?RgBNyrNaSCzcdXb8xe4/hlaGKou2LVhPuKB/Clrz/9p76OGpqJKH95UD+lzj86WyU?= =?utf-8?q?sDBOonOeOCG7agvkXyKvbIqnh1QMkPqaKZTiBI0gBm7RPGzKLpzahPXke2o4ec82x?= =?utf-8?q?R4q2lyCCziOH9cPQpC3K4BazkJ2z5l1EzW8V9fTfIbwHValgjelaeyRrE2EmXY3Ph?= =?utf-8?q?en2m+WSbDS42CTgVWkxHI2X2tJH2aSy6Pl7fUvodA5+oleYTuly6a1v7bjokYqpgF?= =?utf-8?q?jF33cdxnf55ZrIjPuqzILNeJ3HDnQZm5xgaiHrUw5IEjGg7XFW07zFdnAJ5gSdBrA?= =?utf-8?q?bVk/z/BtIvJoydBuUZPeY5Ze52c7FYOIQQtbS3aoBLPWFMVaza60wOVFIomWzF8DF?= =?utf-8?q?a+GCbWbFyQdqRccn/qEZdi68GMhbAA8debIDO2ReZD/+mDvNrBmTu+ftIjNHX8cNN?= =?utf-8?q?fmwJYgHWFrVrdijbBMx/PZPp/2jdc34A6iS2iEStL+wQxpzJTDBIqCRkb0y+RVTvu?= =?utf-8?q?ZsjIcHgK0OZLbkX+s8DnJQaIe4sDq/w9ShJ9ZYniKSlpxsxEzk/QfC1sRJvwF8gYj?= =?utf-8?q?QunrZlVmMxwh3Qs9PvTmlztXvbGQ3k12qUtI+0glFbVxGpcRLyWpoxbrVb6JICA3B?= =?utf-8?q?B5I6UkCxpyJMwuAaBIdkpCUqUopXIqM1TA=3D=3D?= Content-ID: Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: X0Tf4KZfcm3MWRGwKYEt0hL6BLajXL7kXe4qDnqJCcy1VwjkgMz0zb7s3yKiVLtVMawQc9byt5wq75uzJBRa45tA5IO9kd/vQX7SX3t2XaIN20JnwGmU60GbClxDfL1T4K8G/SyFjlLjuISfrZpTU+XPMzwVzLu1WsgghPoRvOwEv+YACh/TvnLD7tcAIsXZmufgwEAg2f+i17vh7ybW0ZVRvM87xDl/qqxdgLYOHXOkt+PKsz9NsoZFCqqoc3sek5ktoAnH6mjpLOtHso324AASQtvcP6o7oBjpjphWWnZDgZH98CqVz8Q2nYm0jwSbPpDbpxlEyE8V091/JSq7Y5sPEqMaqunoZeU9h4U+Chn/XalNOkxE5pe2vja8msco+ayHlSLJwKfbGq6ku5oFxkwkXS+jmOZhInd9igT/7Kltf7H8aN3Zpu0s7H6v9K5aS+BkkwwGEH/E0B/cLdfKk2ZgzG979pMRE/+HUXwSZqnC5mF/BPggua5BWJBqMOV3l0YwoM0juNVnJ8/uaTtNp7ltrnIGpk/ubjTQckrWAnAtaCHpczDJk/V3Ujhnu/T89QY1NI9AdWtkYhY0OGlleiKBWmVZO4lwiSqlO6seKx/6jNH6VVDdMHJ9UPHAoGGM X-OriginatorOrg: wdc.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR04MB5431.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 23242598-8316-48d1-0193-08dc44d35cfc X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Mar 2024 09:36:16.7945 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: dszx7PcktgoGGkI14ihiICVH/dvLN1AOObZt5eGpLgrT0fjBpbf6GmQPzZORyS6FklMrlBRv9StzWEvYrRZOzw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR04MB7526 Hi, While doing some testing using numactl-based interleaving of application memory across regular memory and CXL-based memory using QEMU with tcg, I ran into an issue similar to what we saw a while back - link to old issue: https://lore.kernel.org/qemu-devel/CAFEAcA_a_AyQ=Epz3_+CheAT8Crsk9mOu894wbNW_FywamkZiw@mail.gmail.com/#t. When running: numactl --interleave 0,1 ./cachebench … I hit the following: numactl --interleave 0,1 ./cachebench --json_test_config ../test_configs/hit_ratio/graph_cache_follower_assocs/config.json qemu: fatal: cpu_io_recompile: could not find TB for pc=0x7fffc3926dd4 RAX=00007f65df55ba18 RBX=00007f65df55ba60 RCX=00007f65df221620 RDX=0000000000000000 RSI=00000000011c0260 RDI=00007f65df55ba60 RBP=00007ffdb4b4b280 RSP=00007ffdb4b4b1d0 R8 =00000000011c02c0 R9 =00007f65debf6b20 R10=00000000011bf5d0 R11=00007f65deb7d300 R12=00007ffdb4b4b260 R13=00007ffdb4b4b200 R14=00007ffdb4b4b220 R15=00000000011bf5a0 RIP=00007f65df18affc RFL=00000246 [---Z-P-] CPL=3 II=0 A20=1 SMM=0 HLT=0 ES =0000 0000000000000000 00000000 00000000 CS =0033 0000000000000000 ffffffff 00affb00 DPL=3 CS64 [-RA] SS =002b 0000000000000000 ffffffff 00cff300 DPL=3 DS [-WA] DS =0000 0000000000000000 00000000 00000000 FS =0000 00007f65de2f64c0 00000000 00000000 GS =0000 0000000000000000 00000000 00000000 LDT=0000 0000000000000000 00000000 00008200 DPL=0 LDT TR =0040 fffffe6c37990000 00004087 00008900 DPL=0 TSS64-avl GDT= fffffe6c3798e000 0000007f IDT= fffffe0000000000 00000fff CR0=80050033 CR2=00007f65df1b3eb0 CR3=0000000152a1e000 CR4=00350ef0 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 CCS=0000000000000000 CCD=0000000000000001 CCO=CLR EFER=0000000000000d01 FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80 FPR0=0000000000000000 0000 FPR1=0000000000000000 0000 FPR2=0000000000000000 0000 FPR3=0000000000000000 0000 FPR4=0000000000000000 0000 FPR5=0000000000000000 0000 FPR6=0000000000000000 0000 FPR7=0000000000000000 0000 YMM00=0000000000000000 0000000000000000 00007f65df2233e0 00007f65df221620 YMM01=0000000000000000 0000000000000000 0000000000000000 43e0000000000000 YMM02=0000000000000000 0000000000000000 0000000000000000 0000000000000000 YMM03=0000000000000000 0000000000000000 0000000000000000 0000000000000000 YMM04=0000000000000000 0000000000000000 0000000000000000 3ff0000000000000 YMM05=0000000000000000 0000000000000000 0000000000000000 00007f65df2233e0 YMM06=0000000000000000 0000000000000000 0000000000000000 00007f65df2233b0 YMM07=0000000000000000 0000000000000000 62694c6568636143 2f65636170736b72 YMM08=0000000000000000 0000000000000000 6d622070656d7320 327876612031696d YMM09=0000000000000000 0000000000000000 0000000000000004 0000000000000004 YMM10=0000000000000000 0000000000000000 0000000000000002 0000000000000002 YMM11=0000000000000000 0000000000000000 0000000000000010 0000000000000010 YMM12=0000000000000000 0000000000000000 0000000000ff00fb 0000000000fe00fa YMM13=0000000000000000 0000000000000000 0000000000000000 00ff00fd00fb00f9 YMM14=0000000000000000 0000000000000000 0000000000000000 0000000000000000 YMM15=0000000000000000 0000000000000000 0000000000000000 0000000000000000 The backtrace is (using Jonathans cxl-2024-03-05 branch): (gdb) bt #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737297516096) at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=6, threadid=140737297516096) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=140737297516096, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3 0x00007ffff7642476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #4 0x00007ffff76287f3 in __GI_abort () at ./stdlib/abort.c:79 #5 0x0000555555c5a9df in cpu_abort (cpu=cpu@entry=0x5555578c19c0, fmt=fmt@entry=0x55555605d100 "cpu_io_recompile: could not find TB for pc=%p") at ../cpu-target.c:371 #6 0x0000555555caa065 in cpu_io_recompile (cpu=cpu@entry=0x5555578c19c0, retaddr=140736474541524) at ../accel/tcg/translate-all.c:610 #7 0x0000555555cacee7 in io_prepare (retaddr=140736474541524, addr=140075515361944, attrs=..., xlat=, cpu=0x5555578c19c0, out_offset=) at ../accel/tcg/cputlb.c:1336 #8 do_st_mmio_leN (cpu=0x5555578c19c0, full=0x7ffd1a1554d0, val_le=140075515361816, addr=140075515361944, size=8, mmu_idx=3, ra=140736474541524) at ../accel/tcg/cputlb.c:2591 #9 0x0000555555cb179d in do_st_8 (ra=, memop=, mmu_idx=, val=140075515361816, p=, cpu=) at ../accel/tcg/cputlb.c:2784 #10 do_st8_mmu (cpu=0x5555578c19c0, addr=39050, val=140075515361816, oi=6, ra=140736474541524) at ../accel/tcg/cputlb.c:2862 #11 0x00007fffc3926e15 in code_gen_buffer () #12 0x0000555555ca0e5b in cpu_tb_exec (cpu=cpu@entry=0x5555578c19c0, itb=itb@entry=0x7fffc3926cc0 , tb_exit=tb_exit@entry=0x7ffff49ff6d8) at ../accel/tcg/cpu-exec.c:449 #13 0x0000555555ca13ac in cpu_loop_exec_tb (tb_exit=0x7ffff49ff6d8, last_tb=, pc=, tb=0x7fffc3926cc0 , cpu=0x5555578c19c0) at ../accel/tcg/cpu-exec.c:904 #14 cpu_exec_loop (cpu=cpu@entry=0x5555578c19c0, sc=sc@entry=0x7ffff49ff770) at ../accel/tcg/cpu-exec.c:1019 #15 0x0000555555ca1bb1 in cpu_exec_setjmp (cpu=cpu@entry=0x5555578c19c0, sc=sc@entry=0x7ffff49ff770) at ../accel/tcg/cpu-exec.c:1036 #16 0x0000555555ca2388 in cpu_exec (cpu=cpu@entry=0x5555578c19c0) at ../accel/tcg/cpu-exec.c:1062 #17 0x0000555555cc65c4 in tcg_cpu_exec (cpu=cpu@entry=0x5555578c19c0) at ../accel/tcg/tcg-accel-ops.c:76 #18 0x0000555555cc671f in mttcg_cpu_thread_fn (arg=arg@entry=0x5555578c19c0) at ../accel/tcg/tcg-accel-ops-mttcg.c:95 #19 0x0000555555e61261 in qemu_thread_start (args=) at ../util/qemu-thread-posix.c:541 #20 0x00007ffff7694ac3 in start_thread (arg=) at ./nptl/pthread_create.c:442 #21 0x00007ffff7726850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 Looking at the tb being executed, it looks like it is a single instruction tb, so with my _very_ limited understanding of tcg, it shouldn’t be necessary to do the IO recompile: (gdb) up 13 #13 0x0000555555ca13ac in cpu_loop_exec_tb (tb_exit=0x7ffff49ff6d8, last_tb=, pc=, tb=0x7fffc3926cc0 , cpu=0x5555578c19c0) at ../accel/tcg/cpu-exec.c:904 904 tb = cpu_tb_exec(cpu, tb, tb_exit); (gdb) print *tb $1 = {pc = 0, cs_base = 0, flags = 415285939, cflags = 4278321152, size = 7, icount = 1, tc = {ptr = 0x7fffc3926d80 , size = 176}, page_next = {0, 0}, page_addr = {18446744073709551615, 18446744073709551615}, jmp_lock = {value = 0}, jmp_reset_offset = {65535, 65535}, jmp_insn_offset = {65535, 65535}, jmp_target_addr = {0, 0}, jmp_list_head = 140736474540928, jmp_list_next = {0, 0}, jmp_dest = {0, 0}} If the application is run entirely out of MMIO memory, things work fine (the previous patches related to this is in Jonathans branch), so one thought is that it is related to having the code on a mix of regular and CXL memory. Since we previously had issues with code crossing page boundaries where only the second page is MMIO, I tried out the following change to the fix introduced for that issue thinking that reverting to the slow path in the middle of the translation might not correctly update can_do_io: With that change, things work fine. Not saying that this is a valid fix for the issue, but just trying to provide as much information as possible :) Thanks, Jorgen diff --git a/accel/tcg/translator.c b/accel/tcg/translator.c index 38c34009a5..db6ea360e0 100644 --- a/accel/tcg/translator.c +++ b/accel/tcg/translator.c @@ -258,6 +258,7 @@ static void *translator_access(CPUArchState *env, DisasContextBase *db, if (unlikely(new_page1 == -1)) { tb_unlock_pages(tb); tb_set_page_addr0(tb, -1); + set_can_do_io(db, true); return NULL; }