From patchwork Thu Aug 8 15:13:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shiyang Ruan X-Patchwork-Id: 13757852 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0109C52D7B for ; Thu, 8 Aug 2024 15:13:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BE176B00BF; Thu, 8 Aug 2024 11:13:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 26E976B00DB; Thu, 8 Aug 2024 11:13:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E9CD6B00EA; Thu, 8 Aug 2024 11:13:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id DEA076B00BF for ; Thu, 8 Aug 2024 11:13:39 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 8945EA0DFA for ; Thu, 8 Aug 2024 15:13:39 +0000 (UTC) X-FDA: 82429422558.01.63B6BF3 Received: from esa12.hc1455-7.c3s2.iphmx.com (esa12.hc1455-7.c3s2.iphmx.com [139.138.37.100]) by imf04.hostedemail.com (Postfix) with ESMTP id F0CB240007 for ; Thu, 8 Aug 2024 15:13:36 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=fujitsu.com header.s=fj2 header.b=BrfscGuE; dmarc=pass (policy=reject) header.from=fujitsu.com; spf=pass (imf04.hostedemail.com: domain of ruansy.fnst@fujitsu.com designates 139.138.37.100 as permitted sender) smtp.mailfrom=ruansy.fnst@fujitsu.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723129969; a=rsa-sha256; cv=none; b=uhaK6BxLTy9j1bi01CY4preGwzVQQfh6CxJ8gnghDVPXxXWXYz5Ui6X1OkGCCNxQQ9d4GI Iu7Va7NHaxBsf+d1IyCLbNirkL4PW5OgNynByIILjhpLygGMy+ee5+9tRLrh9Uzr/e2WjF JnfvT1yQ5iydioVaPS1Oo0QliLiuVGk= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=fujitsu.com header.s=fj2 header.b=BrfscGuE; dmarc=pass (policy=reject) header.from=fujitsu.com; spf=pass (imf04.hostedemail.com: domain of ruansy.fnst@fujitsu.com designates 139.138.37.100 as permitted sender) smtp.mailfrom=ruansy.fnst@fujitsu.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723129969; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=rHWvGXPiP+8qRxi4NCZMqZ1UTWUnRTJxopwRZetL27g=; b=ij+CQlm7RXaEH0wUirDABB3IxB8kas4xy3XiPyFF3MK66ClDxVbMzUj8A6FidEnb0UXOU8 /9fOM6WB26pB28YRWqNi+lr5P+hBZc+unmQq+m32T37sx/FnsX9rJvPqMOhLT4O3MBphIo mvSb+o0TQTOcoJsVMnBdcxOhVY8+W+A= DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=fujitsu.com; i=@fujitsu.com; q=dns/txt; s=fj2; t=1723130017; x=1754666017; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=iFNP495OinJb2lktMcGqRGAhtEH6mWIcSw6cu316J9c=; b=BrfscGuEsLBc4cXpXlD6F/LMHTO13Ndx56HVTojuiu6IvQYik+GWsG52 bINgln1IcIBmollEPtARujmdpr7W33lJXNihxKdtvOelWailx7RYvqDNS 3mgA10ij8E/QyVjH9iQK6dEtPY36R6JnO5vwZ00MoD+cD2O3TOJWkaeRQ 9Hv1yJf0ETbEaIbCazpKAP1IzWtRlUcS6L0BCYrR4tBonOMmQoeGVwjdJ GZLWZVXkNDPOmlY760EWaLquFufqSIMMbfVlNdPH7qkweOY3h4Xh3zxfF Eds4mC/Rh5WZRf3XP5Jm72KcQP5AP6MgYDljCXJ3BLbOIHtgvyG1NxSnT Q==; X-CSE-ConnectionGUID: z2LOWnzeQSmy1wuv7IdZ/A== X-CSE-MsgGUID: OsOgjVDKRRGKEcRqbrTP4A== X-IronPort-AV: E=McAfee;i="6700,10204,11158"; a="149202101" X-IronPort-AV: E=Sophos;i="6.09,273,1716217200"; d="scan'208";a="149202101" Received: from unknown (HELO yto-r2.gw.nic.fujitsu.com) ([218.44.52.218]) by esa12.hc1455-7.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Aug 2024 00:13:34 +0900 Received: from yto-m4.gw.nic.fujitsu.com (yto-nat-yto-m4.gw.nic.fujitsu.com [192.168.83.67]) by yto-r2.gw.nic.fujitsu.com (Postfix) with ESMTP id 2D9E6C68E2 for ; Fri, 9 Aug 2024 00:13:32 +0900 (JST) Received: from kws-ab3.gw.nic.fujitsu.com (kws-ab3.gw.nic.fujitsu.com [192.51.206.21]) by yto-m4.gw.nic.fujitsu.com (Postfix) with ESMTP id 5C34DD3F20 for ; Fri, 9 Aug 2024 00:13:31 +0900 (JST) Received: from edo.cn.fujitsu.com (edo.cn.fujitsu.com [10.167.33.5]) by kws-ab3.gw.nic.fujitsu.com (Postfix) with ESMTP id CBD5A200649E2 for ; Fri, 9 Aug 2024 00:13:30 +0900 (JST) Received: from irides.g08.fujitsu.local (unknown [10.167.226.114]) by edo.cn.fujitsu.com (Postfix) with ESMTP id 28E1D1A000A; Thu, 8 Aug 2024 23:13:29 +0800 (CST) From: Shiyang Ruan To: qemu-devel@nongnu.org, linux-cxl@vger.kernel.org, linux-edac@vger.kernel.org, linux-mm@kvack.org, dan.j.williams@intel.com, vishal.l.verma@intel.com, Jonathan.Cameron@huawei.com, alison.schofield@intel.com Cc: bp@alien8.de, dave.jiang@intel.com, dave@stgolabs.net, ira.weiny@intel.com, james.morse@arm.com, linmiaohe@huawei.com, mchehab@kernel.org, nao.horiguchi@gmail.com, rric@kernel.org, tony.luck@intel.com, ruansy.fnst@fujitsu.com Subject: [PATCH v4 0/2] cxl: add device reporting poison handler Date: Thu, 8 Aug 2024 23:13:26 +0800 Message-Id: <20240808151328.707869-1-ruansy.fnst@fujitsu.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-TM-AS-Product-Ver: IMSS-9.1.0.1417-9.0.0.1002-28584.000 X-TM-AS-User-Approved-Sender: Yes X-TMASE-Version: IMSS-9.1.0.1417-9.0.1002-28584.000 X-TMASE-Result: 10--25.644800-10.000000 X-TMASE-MatchedRID: lHzeI+asw8M0b21ExFN2+kg5Iem1vm3HTFQnI+epPIbVYWFt1wbX7Eop D+RCCRBkvX+3OiqxW1kU/FrOclzESmD8e8nXbfYy0wmR34xRQc8v5vY1YvMqbujMOEZ5AL0SUZS Ad8uKlMfx79muHEGAix37L/Rv0wwcXOFTT+immjbfh7vdn2pP64zinsSc+o89H8jJDe3vPQN5GD v/ZOF4Nm5/m6vlvI+l/Iblp89iIQD1aPobg60YTKJVTu7sjgg1odQJXaDex6fWeQtrcncLfbIQd fMMKrFBCD9IcjSuAiPykYfoCiS47xyfGYp07KWZTbFVCYPBTqZZDdHiTk9OcLfYIuZsOQ0shj53 gjhYKkQ8I7sVBmNJqozRCiL6QfeVNigjEI6ndRqJJ72DuZB0nDnOm2OHJgpYcdSPHEhb6Fe2NT1 CGlQHxr/5yx5GCNXjb3JmPdq59vkfE8yM4pjsDwtuKBGekqUpPjKoPgsq7cA= X-TMASE-SNAP-Result: 1.821001.0001-0-1-22:0,33:0,34:0-0 X-Rspamd-Queue-Id: F0CB240007 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 1no6kp1n4sbnjgge1exu8y377f3ibjw9 X-HE-Tag: 1723130016-753474 X-HE-Meta: U2FsdGVkX18xQpF2Y6Z+lUYkFSQ1fP2qaiJgdkA6qRERa8dnJdDl2e5uyp7VQKfCmiDI0PdBVswQlv5XmlVkfDJa9Jv+dKH1gCEmcE/uuGDheR0pXbDj12uzvdY4QwSo+/Aaps4ltXgZdQhYBNkAEabvItfiYb62rxUyTFPs9XHPv7d8WwK9KOkcwr4UNKhd7ksEVu29Rq6SXxv89HmulPmDcXTpEginnmK9Xd6znTuCIaYP5W7UtmLAZdeybEdpcBCAEVIxBms/hQ9NmgcxEjx6vzsCdNbCMmOburWF6fBKvPSKOlB2kIv3rsuPdXhDHfhw8KkNQkuGlu2RUGkwuGqMC9rEFdoeUHPVoPzADvrRwfztHogre8Ynffx7O3AEETTaBor4EwymMKfBXPhQ5n2hcC/Yy9ZRpNCndK+You1wR10cgOde1R9L9oZTXSTlMzhcm1MHirnYJJZsNNvzr7YNTiaYDGknfGi2jxLMlpu/G99Hzr5GerQk9lHaK8ioz6JeY6J1dAvYRjNku7izWU7l+shwr4kP086VjJoR3cSx/Wms0cDiyB8St1m8IOS/6AQaX9RsMXAilYQXhVfqVeqCfxAhk+pWjBkDTs3v6/cx+jlcNhmdwBzbDJ75mKyMEQKDSykxn4FQA67RyWIhO6ghfPwhimmgEy+XVKSwUcY74208XkXwFjeEqG3gp5qrTwqsG0canf/LZkl5pEYXgg+dxcnT98Xg0lRrzdgcfAPEjReh0Ve9AutbBbsBTEoicKnxFBcoQ1H8uFqUueP59/woD3qT4cjt9MvTv8G+OVv8fRDFvzGbY5hezsFjtDkPde0R7Ck4cqZu2tyd6h9mx5gQ+rSIAycf0ssEsu+RlfHTxPJPWbWBgEj5kdJrxApe/DLxTWte4DyEJFVwcPbzS5GebHz6OXpcstqSK5BZ85EVtlQrbKyZttsVmhsDEaDlwmwDJONPPQbee/CEj3V CqzeruJ8 HqHz9yCTNtVEhKNlgyoaEmWGWHzm7/hH5ERSRyVCSNtmUDz0sLAfPfsz2mN5hT5Nk9rmaPlBVQzsqy0NANGDEAhyIt7EysK+nEj9ABTvIy/FFPRxDyra6dyz2UIiHAAgTxAHKuRW2Bv9oF9rAVgxmf67nnllcrTPURs/1tz1njPyFBa2sXZIze1k+94ndk0tNR+JKxE8RvU5Tn/qGCcIqxSyrjyIFy67HC508XAF05Vpxh56ss1XVZSJFkZ/pZ8Zp7+7hAMxThk5JSWfwk12912puf7Ui3DO65nUa X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patchset includes "cxl/core: introduce poison creation hanlding" and "cxl: avoid duplicated report from MCE & device", which were posted separately. Here are changes since last version of each patch: P1: 1. since its async memory_failure(), set the flag to 0 2. also handle CXL_EVENT_TRANSACTION_SCAN_MEDIA type P2: 1. use XArray instead of list_head 2. add guard() lock for cxl device iteration P1&P2: Rebase to v6.11-rc1 As is known to us, CXL spec defines POISON feature to notify its status when CXL memory device got a broken page. Basically, there are two major paths for the notification. 1. CPU handling error When a process is accessing this broken page, CXL device returns data with POISON. When CPU consumes the POISON, it raises a kind of error notification. To be precise, "how CPU should behave when it consumes POISON" is architecture dependent. In my understanding, x86-64 raises Machine Check Exception(MCE) via interrupt #18 in this case. 2. CXL device reporting error When CXL device detects the broken page by itself and sends memory error signal to kernel in two optional paths. 2.a. FW-First CXL device sends error via VDM to CXL Host, then CXL Host sends it to System Firmware via interrupt, finally kernel handles the error. 2.b. OS-First CXL device directly sends error via MSI/MSI-X to kernel. Note: Since I'm now focusing on x86_64, basically I'll describe about x86-64 only. The following diagram should describe the 2 major paths and 2 optional sub-paths above. ``` 1. MCE (interrupt #18, while CPU consuming POISON) -> do_machine_check() -> mce_log() -> notify chain (x86_mce_decoder_chain) -> memory_failure() 2.a FW-First (optional, CXL device proactively find&report) -> CXL device -> Firmware -> OS: ACPI->APEI->GHES->CPER -> CXL driver -> trace 2.b OS-First (optional, CXL device proactively find&report) -> CXL device -> MSI -> OS: CXL driver -> trace ``` For "1. CPU handling error" path, the current code seems to work fine. When I used error injection feature on QEMU emulation, the code path is executed certainly. Then, if the CPU certainly raises a MCE when it consumes the POISON, this path has no problem. So, I'm working on making for 2.a and 2.b path, which is CXL device reported POISON error could be handled by kernel. This path has two advantages. - Proactively find&report memory problems Even if a process does not read data yet, kernel/drivers can prevent the process from using corrupted data proactively. AFAIK, the current kernel only traces POISON error event from FW-First/OS-First path, but it doesn't handle them, neither notify processes who are using the POISON page like MCE does. User space tools like rasdaemon reads the trace and log it, but as well, it doesn't handle the POISON page. As a result, user has to read the error log from rasdaemon, distinguish whether the POISON error is from CXL memory or DDR memory, find out which applications are effected. That is not an easy work and cannot be handled in time. Thus, I'd like to add a feature to make the work done automatically and quickly. Once CXL device reports the POISON error (via FW-First/OS-First), kernel handles it immediately, similar to the flow when a MCE is triggered. This is my first motivation. - Architecture independent As the mentioned above, "1. CPU handling error" path is architecture dependent. On the other hand, this route can be architecture independent code. If there is a CPU which does not have similar feature like MCE of x86-64, my work will be essential. (To be honest, I did not notice this advantage at first as mentioned later, but I think this is also important.) Shiyang Ruan (2): cxl/core: introduce device reporting poison hanlding cxl: avoid duplicated report from MCE & device arch/x86/include/asm/mce.h | 1 + drivers/cxl/core/mbox.c | 190 ++++++++++++++++++++++++++++++++++--- drivers/cxl/core/memdev.c | 6 +- drivers/cxl/cxlmem.h | 11 ++- drivers/cxl/pci.c | 4 +- include/linux/cxl-event.h | 16 +++- 6 files changed, 207 insertions(+), 21 deletions(-)