From patchwork Thu Nov 23 09:23:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sumanth Korikkar X-Patchwork-Id: 13466034 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34A0DC61D85 for ; Thu, 23 Nov 2023 09:58:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9273D8D001C; Thu, 23 Nov 2023 04:58:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8D5D38D0002; Thu, 23 Nov 2023 04:58:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7772C8D001C; Thu, 23 Nov 2023 04:58:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 646B68D0002 for ; Thu, 23 Nov 2023 04:58:41 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 314A4B66E6 for ; Thu, 23 Nov 2023 09:58:41 +0000 (UTC) X-FDA: 81488769642.13.44185CE Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf26.hostedemail.com (Postfix) with ESMTP id 5F29E140008 for ; Thu, 23 Nov 2023 09:58:38 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Vc4+O1y+; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf26.hostedemail.com: domain of sumanthk@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=sumanthk@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700733518; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HYgtCZxFyEih59yMBfurLMGboumwZXHsg8XyOU1jf9U=; b=VJPpcXr4uPS0g6YLs/VVbLsMT7Rq8SKWrO27WrUrN7ueJfIhcsA2aCn8VrsCgKCJKjtuQl vhItUrrLtY+lAPE1c47ioSrQFJ2L6N6P7IHqfYqX/HcRe2n9UPKbGImfpuWd5Qz4CFSaHN 145iYvMgafmJPXyiOJintRzvOWkpN/8= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Vc4+O1y+; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf26.hostedemail.com: domain of sumanthk@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=sumanthk@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700733518; a=rsa-sha256; cv=none; b=XP6rf3t1h+5YerWtWXIzw68p3j2udAYbW0EI7GQbQVpRzQ1HDlg16aZhxBxLfXGktiD1zA d/SAKPBF1CCvvgrqv+lIun1CAxbfaICHF0TAxU07F/elrlmKulVvyNGJLIWiGZg0JBuzQH l6CamVKSvSfM+x5ZM1w0hFGVpxinqo4= Received: from pps.filterd (m0353726.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3AN97SOE030489; Thu, 23 Nov 2023 09:24:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=HYgtCZxFyEih59yMBfurLMGboumwZXHsg8XyOU1jf9U=; b=Vc4+O1y+QnWgZoO0+wVbdg6ae+z4jqwMk/Ws7zP2dNJO8E/MxWGbarmZ6q41B2n3aDak oX8SobRQYsO4XYSOFktWOfr6Vqx7H3w0rHahmf1buRqmK7WD2HPkTmuJdpnw0wroQkrP O/rU08cbXksW+pNdMs6n7I7qfKLOUXqAArJemAtJTfNxxB8ivpbTLH/bCas17tYa8P3a hcNnQ4Pj6iCjNxRXr7GlQaEJptjmXoi6rGrsltjuoNzbUac4cDqKaGhdAZ231oU8Ex4t OylgpPPoQLMWfKexJyXEkJc+QJOHv1Rz+lOmfalm7Gz1Ccj9OsxZiEzC7DN84IjRj5P1 Xg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3uj3twrepv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 23 Nov 2023 09:24:01 +0000 Received: from m0353726.ppops.net (m0353726.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3AN98Cl1001750; Thu, 23 Nov 2023 09:24:01 GMT Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3uj3twrep2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 23 Nov 2023 09:24:00 +0000 Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3AN8nG1e021112; Thu, 23 Nov 2023 09:23:59 GMT Received: from smtprelay06.fra02v.mail.ibm.com ([9.218.2.230]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3uf8kp67bb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 23 Nov 2023 09:23:59 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay06.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3AN9Nugb35979694 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 23 Nov 2023 09:23:56 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6387C20040; Thu, 23 Nov 2023 09:23:56 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 34A712004B; Thu, 23 Nov 2023 09:23:56 +0000 (GMT) Received: from tuxmaker.boeblingen.de.ibm.com (unknown [9.152.85.9]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Thu, 23 Nov 2023 09:23:56 +0000 (GMT) From: Sumanth Korikkar To: linux-mm , Andrew Morton , David Hildenbrand Cc: Oscar Salvador , Michal Hocko , "Aneesh Kumar K.V" , Anshuman Khandual , Gerald Schaefer , Sumanth Korikkar , Alexander Gordeev , Heiko Carstens , Vasily Gorbik , linux-s390 , LKML Subject: [PATCH v2 3/7] mm/memory_hotplug: introduce MEM_PREPARE_ONLINE/MEM_FINISH_OFFLINE notifiers Date: Thu, 23 Nov 2023 10:23:39 +0100 Message-Id: <20231123092343.1703707-4-sumanthk@linux.ibm.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231123092343.1703707-1-sumanthk@linux.ibm.com> References: <20231123092343.1703707-1-sumanthk@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: Yp3vEj8jyuT_QEU8ZsHP0Gw5R-tAowx- X-Proofpoint-ORIG-GUID: vfXWQ1jzYRqboH9Tkb3l5IQQkwYio78y X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.987,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-11-23_07,2023-11-22_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 mlxscore=0 suspectscore=0 bulkscore=0 priorityscore=1501 phishscore=0 mlxlogscore=473 adultscore=0 lowpriorityscore=0 spamscore=0 clxscore=1015 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311060000 definitions=main-2311230067 X-Rspam-User: X-Stat-Signature: 6yf35m7q3897dohpuobajp9zxfbo6s5r X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5F29E140008 X-HE-Tag: 1700733518-584358 X-HE-Meta: U2FsdGVkX19vgVEOO8+op9hEsUX2C/Y6z3mf/4y7rsDx3Go6MzI9GuHMnVB5oJNyQZpxbS4EAMYNi9N1vUPT19EnVUDHUJFPBwWXBg3Ghv8qH9mZ+fmV9AbMN8xUhDYC3dU+884rNlEV3aDhfl2HWzUMp7YFYhRZsIUrQeMgXz5ZIqC3pRqHczVTOd8/Gk5AUdFaZZpvPM5mXCisHkRq6CMaiORD+K/UlzFG7TaloCTlo2vH7C4I/SCeCeb56TgR2+bSIgY0EItezbIfCOHH3wtF09uz6ru55FvoQSwCaWBpHLHqA8vgO/GHlnuFQwZyeTlEB5+E+Ga0NxITB+2BCzeFlrsTY0TLKVSsoQLCeGiEo2VndBFg8bDa3Cs8iC1qkBWvreG1lEDHJbZ9NOfHgzX/YOH0spAcWH9Lj6n9VK5p8MZ0eX229EkxGDNdHN7YMJST8GWSuphDTLG5Gb8BVXudh4s3CNXcQhWtrOivEx+HV7TzipYOVaMqZzJ6k2FidsQ1YwyTrRossZHOeReQszaTV6meWgLa/5nw9eX7ZZj2LTKA+w+3qkazrvyX9NcyeLyNCGDT8QnuBlKKzDqz7JS8LCAjADNojVSCG+SKRUuoFhJdRlgxy/dlDYulTCzjimoqCKHbhoGj5bGeFqsttBFs2ld47LAXNRwI65NFO19b7pgoTYhoyzERLv2S+jz5ekW1zdQ/KsfAkZ8yslJjM4S+cnjSj4sSFA7k+vMXSIWBosb+p+K14fpwF88j7sOsz0XUP5/qRRWCNYKUgiaVOoPyILx/Gf4XXxfJ9M3fz3BGknAhS3W461ckfnvJM6+Z/3v/koCAEAkZ1yQrSh+aXIEmEOZlOKI4Jd55GpJJ4XFf82gCBn1kqOXbJs+FN9qzjEJNj4GS7A9o3bH9rjDzt6I+VPKtK9NLd9Y1QvnCXj1v27EvFI8ng/vx7cTnyQ94iTDjsJX5PhL8BZlLi65 cpTcENz2 Sogz17ByC7/tIR3wUs+1gryywko1ZCq1ZdgnpaZp8kkpW8i8ptKkjn9DqUDO+82V9PtrSoAENYbM2OYkjSW+jcs3NDZVD0hkhJNa219ZutFkXrJduNfq+JHPH7U/61wA397GjF5NSRkBNtjTf8yWzAalJyynGyV/x0EDeAOHtXpSmTqrpjJIBXFk5JpIwQSsbXGIN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce MEM_PREPARE_ONLINE/MEM_FINISH_OFFLINE memory notifiers to prepare the transition of memory to and from a physically accessible state. This enhancement is crucial for implementing the "memmap on memory" feature for s390 in a subsequent patch. Platforms such as x86 can support physical memory hotplug via ACPI. When there is physical memory hotplug, ACPI event leads to the memory addition with the following callchain: acpi_memory_device_add() -> acpi_memory_enable_device() -> __add_memory() After this, the hotplugged memory is physically accessible, and altmap support prepared, before the "memmap on memory" initialization in memory_block_online() is called. On s390, memory hotplug works in a different way. The available hotplug memory has to be defined upfront in the hypervisor, but it is made physically accessible only when the user sets it online via sysfs, currently in the MEM_GOING_ONLINE notifier. This is too late and "memmap on memory" initialization is performed before calling MEM_GOING_ONLINE notifier. During the memory hotplug addition phase, altmap support is prepared (but not yet accessed) and during the memory onlining phase s390 requires memory to be physically accessible and then subsequently initiate the "memmap on memory" initialization process. The new MEM_PREPARE_ONLINE notifier allows to work around the problem, by providing a hook to prepare the memory and make it physically accessible. Similarly, the MEM_FINISH_OFFLINE notifier allows to make the memory physically inaccessible at the end of memory_block_offline(). All architectures ignore unknown memory notifiers, so this patch should not introduce any functional changes. Reviewed-by: Gerald Schaefer Signed-off-by: Sumanth Korikkar --- drivers/base/memory.c | 18 +++++++++++++++++- include/linux/memory.h | 2 ++ 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/drivers/base/memory.c b/drivers/base/memory.c index cbff43b2ef44..a06a0b869992 100644 --- a/drivers/base/memory.c +++ b/drivers/base/memory.c @@ -188,6 +188,7 @@ static int memory_block_online(struct memory_block *mem) unsigned long start_pfn = section_nr_to_pfn(mem->start_section_nr); unsigned long nr_pages = PAGES_PER_SECTION * sections_per_block; unsigned long nr_vmemmap_pages = 0; + struct memory_notify arg; struct zone *zone; int ret; @@ -197,6 +198,14 @@ static int memory_block_online(struct memory_block *mem) zone = zone_for_pfn_range(mem->online_type, mem->nid, mem->group, start_pfn, nr_pages); + arg.start_pfn = start_pfn; + arg.nr_pages = nr_pages; + mem_hotplug_begin(); + ret = memory_notify(MEM_PREPARE_ONLINE, &arg); + ret = notifier_to_errno(ret); + if (ret) + goto out_notifier; + /* * Although vmemmap pages have a different lifecycle than the pages * they describe (they remain until the memory is unplugged), doing @@ -207,7 +216,6 @@ static int memory_block_online(struct memory_block *mem) if (mem->altmap) nr_vmemmap_pages = mem->altmap->free; - mem_hotplug_begin(); if (nr_vmemmap_pages) { ret = mhp_init_memmap_on_memory(start_pfn, nr_vmemmap_pages, zone, mem->inaccessible); @@ -232,7 +240,11 @@ static int memory_block_online(struct memory_block *mem) nr_vmemmap_pages); mem->zone = zone; + mem_hotplug_done(); + return ret; out: + memory_notify(MEM_FINISH_OFFLINE, &arg); +out_notifier: mem_hotplug_done(); return ret; } @@ -245,6 +257,7 @@ static int memory_block_offline(struct memory_block *mem) unsigned long start_pfn = section_nr_to_pfn(mem->start_section_nr); unsigned long nr_pages = PAGES_PER_SECTION * sections_per_block; unsigned long nr_vmemmap_pages = 0; + struct memory_notify arg; int ret; if (!mem->zone) @@ -276,6 +289,9 @@ static int memory_block_offline(struct memory_block *mem) mhp_deinit_memmap_on_memory(start_pfn, nr_vmemmap_pages); mem->zone = NULL; + arg.start_pfn = start_pfn; + arg.nr_pages = nr_pages; + memory_notify(MEM_FINISH_OFFLINE, &arg); out: mem_hotplug_done(); return ret; diff --git a/include/linux/memory.h b/include/linux/memory.h index 655714d4e65a..76e5ab68dab7 100644 --- a/include/linux/memory.h +++ b/include/linux/memory.h @@ -97,6 +97,8 @@ int set_memory_block_size_order(unsigned int order); #define MEM_GOING_ONLINE (1<<3) #define MEM_CANCEL_ONLINE (1<<4) #define MEM_CANCEL_OFFLINE (1<<5) +#define MEM_PREPARE_ONLINE (1<<6) +#define MEM_FINISH_OFFLINE (1<<7) struct memory_notify { unsigned long start_pfn;