| Message ID | 20230807213635.3633907-2-terry.bowman@amd.com |
|---|---|
| State | New, archived |
| Headers | show |
| Series | cxl: Script to manage regions during CXL device hotplug remove-reinsert | expand |
Terry Bowman wrote: > Hotplug removing a CXL device and hotplug reinserting the same device > currently requires manual interaction for managing the device > region. The CXL device region must be manually destroyed before > hotplug removal and manually created after hotplug reinsertion. > > Create a script to automatically destroy and recreate the region > during CXL device hotplug remove-reinsert. Save region characteristics to > a file before destroying the region and hotplug remove. Use the region > characteristics stored in the file to recreate the region after > hotplug reinsert. It strikes me that functionality like this might be formalized under a command like "cxl export-region [--destroy]" where import support might be automatic if the exported configuration is saved somewhere udev can find it, otherwise "cxl import-region" could take hot added device back to its defined region. > > Signed-off-by: Terry Bowman <terry.bowman@amd.com> > --- > README.txt | 311 +++++++++++++++++++++++++++++++++++++++++ > cxl-hotplug.py | 366 +++++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 677 insertions(+) > create mode 100644 README.txt > create mode 100755 cxl-hotplug.py > > diff --git a/README.txt b/README.txt > new file mode 100644 > index 0000000..97fa793 > --- /dev/null > +++ b/README.txt > @@ -0,0 +1,311 @@ > + ____________________ > + > + CXL-HOTPLUG.README > + ____________________ mmm, documentation. > + > + > +Table of Contents > +_________________ > + > +1. Purpose > +2. Requirements > +.. 1. Kernel - v6.4.0 > +.. 2. ndctl - v77 or later > +.. 3. QEMU - v8.0.3 + the following patches: > +.. 4. Python modules: > +.. 5. Additional tool details > +3. Usage > +4. Examples > +.. 1. Swap a device > +.. 2. Manually unplug and plugin a device > +.. 3. Manually unplug and plugin a device (w/ step by step details) > + > + > + > + > + > +1 Purpose > +========= > + > + Hotplug adding and removing CXL devices requires region management not > + automatically provided. For instance, if a CXL device is added then a > + region must be 'created' before the memory can be used. Likewise, a > + region must be 'destroyed' before a CXL device can be > + removed. Removing a CXL device before a region is 'destroyed' can > + result in CXL device data loss or corruption. > + > + This tool aims to provide region delete and create automation for an > + existing CXL device. The typical usage is to hotplug remove an > + existing CXL device and then hotplug readd the same device immediately > + or at some later time. > + > + An unplug function is provided that will 'destroy' the region making > + the device ready for hotplug removal. Note, 'destroying' a PMEM region > + incurrs no loss of data. 'destroying' a RAM region will lose the > + region data. This tool saves the region structure information so that > + the device's region can be created in the future. After the device is > + hot hotplug added in the future the region information is used to > + recreate the region. The region information is saved to a default file > + or can be directed to a specific file provided on the tool comandline. > + > + This tool provides a swap function that automatically executes the > + unplug and plugin functions. > + > + This tool is currently limited to non interleaved devices. > + > + > +2 Requirements > +============== > + > + Tested using the following: > + > + > +2.1 Kernel - v6.4.0 > +~~~~~~~~~~~~~~~~~~~ > + > + To include the following kernel config settings: > + ,---- > + | scripts/config --enable LIBNVDIMM > + | scripts/config --enable CONFIG_CXL_BUS > + | scripts/config --enable CONFIG_CXL_PCI > + | scripts/config --enable CONFIG_CXL_ACPI > + | scripts/config --enable CONFIG_CXL_MEM > + | scripts/config --enable CONFIG_CXL_PMEM > + | scripts/config --enable CONFIG_CXL_PORT > + | scripts/config --enable CONFIG_CXL_SUSPEND > + | scripts/config --enable CONFIG_CXL_REGION > + | scripts/config --enable CONFIG_CXL_REGION_INVALIDATION_TEST > + | scripts/config --enable CONFIG_PCIEAER_CXL > + | scripts/config --enable CONFIG_TRANSPARENT_HUGEPAGE > + | scripts/config --enable CONFIG_DEV_DAX > + | scripts/config --enable CONFIG_DEV_DAX_HMEM > + | scripts/config --enable CONFIG_DEV_DAX_KMEM > + | scripts/config --enable CONFIG_DEV_DAX_PMEM > + `---- > + The above configures for statically linked drivers. They could be > + dynamically linked as modules as well. > + > + > +2.2 ndctl - v77 or later > +~~~~~~~~~~~~~~~~~~~~~~~~ > + > + > +2.3 QEMU - v8.0.3 + the following patches: > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > + 20230227163157.6621-1-Jonathan.Cameron@huawei.com > + 20230303150908.27889-1-Jonathan.Cameron@huawei.com > + <https://lore.kernel.org/linux-cxl/20230303152903.28103-1-Jonathan.Cameron@huawei.com/\#r> > + > + > +2.4 Python modules: > +~~~~~~~~~~~~~~~~~~~ > + > + The following python modules are required: json subprocess os argparse > + logging > + > + > +2.5 Additional tool details > +~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > + <https://confluence.amd.com/display/ALK/CXL+QEMU> Internal-only web page? > + > + > +3 Usage > +======= > + > + The utility requires one of the sub-commands: plugin,unplug,list,swap. > + > + Commandline usage information: > + ,---- > + | usage: cxl-hotplug.py [-h] {plugin,unplug,list,swap} ... > + | > + | positional arguments: > + | {plugin,unplug,list,swap} > + | > + | options: > + | -h, --help show this help message and exit > + `---- > + > + Commandline plugin usage information: > + ,---- > + | usage: cxl-hotplug.py plugin [-h] -m MEMDEV [-c CONFIG_FILE] [-d] > + | > + | options: > + | -h, --help show this help message and exit > + | -m MEMDEV CXL memory device to prepare for unplug > + | -c CONFIG_FILE CXL JSON configuration file to use > + | -d, --debug Enable debugging > + `---- > + > + Commandline unplug usage information: > + ,---- > + | usage: cxl-hotplug.py unplug [-h] -m MEMDEV [-c CONFIG_FILE] [-d] > + | > + | options: > + | -h, --help show this help message and exit > + | -m MEMDEV CXL memory device to prepare for unplug > + | -c CONFIG_FILE CXL JSON configuration file to save > + | -d, --debug Enable debugging > + `---- > + > + Commandline swap usage information: > + ,---- > + | usage: cxl-hotplug.py swap [-h] -m MEMDEV [-d] > + | > + | options: > + | -h, --help show this help message and exit > + | -m MEMDEV CXL memory device to swap > + | -d, --debug Enable debugging > + `---- > + > + > +4 Examples > +========== > + > +4.1 Swap a device > +~~~~~~~~~~~~~~~~~ > + > + ,---- > + | # cxl create-region -t ram -m mem0 -d decoder0.0 -w 1 -s 256M > + | # ./cxl-hotplug.py swap -m mem0 > + | Device 'mem0' can now be safely removed. > + | Card is ready for removal. > + | Press any key to continue after card reinsertion. > + | Region created for 'mem0'. > + `---- > + > + > +4.2 Manually unplug and plugin a device > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > + ,---- > + | # cxl create-region -t ram -m mem0 -d decoder0.0 -w 1 -s 256M > + | # ./cxl-hotplug.py unplug -m mem0 -c my-cxl.json > + | # ./cxl-hotplug.py plugin -m mem0 -c my-cxl.json > + `---- > + > + > +4.3 Manually unplug and plugin a device (w/ step by step details) > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > + ,---- > + | # cxl list > + | [ > + | { > + | "memdev":"mem0", > + | "ram_size":268435456, > + | "serial":0, > + | "host":"0000:0d:00.0" > + | } > + | ] > + | > + | # ./cxl-hotplug.py list > + | Device Region Name Region Size Movable > + | ====== =========== =========== ======= > + | mem0 NA NA NA > + | > + | # cxl create-region -t ram -m mem0 -d decoder0.0 -w 1 -s 256M > + | { > + | "region":"region0", > + | "resource":"0x890000000", > + | "size":"256.00 MiB (268.44 MB)", > + | "type":"ram", > + | "interleave_ways":1, > + | "interleave_granularity":256, > + | "decode_state":"commit", > + | "mappings":[ > + | { > + | "position":0, > + | "memdev":"mem0", > + | "decoder":"decoder2.0" > + | } > + | ] > + | } > + | cxl region: cmd_create_region: created 1 region > + | > + | # cxl list > + | [ > + | { > + | "memdevs":[ > + | { > + | "memdev":"mem0", > + | "ram_size":268435456, > + | "serial":0, > + | "host":"0000:0d:00.0" > + | } > + | ] > + | }, > + | { > + | "regions":[ > + | { > + | "region":"region0", > + | "resource":36775657472, > + | "size":268435456, > + | "type":"ram", > + | "interleave_ways":1, > + | "interleave_granularity":256, > + | "decode_state":"commit" > + | } > + | ] > + | } > + | ] > + | > + | # ./cxl-hotplug.py list > + | Device Region Name Region Size Movable > + | ====== =========== =========== ======= > + | mem0 region0 268435456 True > + | > + | # ./cxl-hotplug.py unplug -m mem0 > + | Device 'mem0' can now be safely removed. > + | > + | # cxl list > + | [ > + | { > + | "memdev":"mem0", > + | "ram_size":268435456, > + | "serial":0, > + | "host":"0000:0d:00.0" > + | } > + | ] > + | > + | # ./cxl-hotplug.py list > + | Device Region Name Region Size Movable > + | ====== =========== =========== ======= > + | mem0 NA NA NA > + | > + | # ./cxl-hotplug.py plugin -m mem0 > + | Region created for 'mem0'. > + | > + | # cxl list > + | [ > + | { > + | "memdevs":[ > + | { > + | "memdev":"mem0", > + | "ram_size":268435456, > + | "serial":0, > + | "host":"0000:0d:00.0" > + | } > + | ] > + | }, > + | { > + | "regions":[ > + | { > + | "region":"region0", > + | "resource":36775657472, > + | "size":268435456, > + | "type":"ram", > + | "interleave_ways":1, > + | "interleave_granularity":256, > + | "decode_state":"commit" > + | } > + | ] > + | } > + | ] > + | > + | # ./cxl-hotplug.py list > + | Device Region Name Region Size Movable > + | ====== =========== =========== ======= > + | mem0 region0 268435456 True > + `---- Overall I like wrapping all the details of the focused sub-commands into something that is more goal oriented. I have no heartburn with carrying this in cxl/contrib/ while figuring out if this should also be hooked as a formal sub-command set. > diff --git a/cxl-hotplug.py b/cxl-hotplug.py > new file mode 100755 > index 0000000..971e4e7 > --- /dev/null > +++ b/cxl-hotplug.py > @@ -0,0 +1,366 @@ > +#!/usr/bin/python3 > +# SPDX-License-Identifier: LGPL-2.1 > +# > +# Copyright (C) 2023, Advanced Micro Devices (AMD). All rights reserved. > +# > +# @Author - Terry Bowman > +# > +# Utility to support CXL hotplug removal and re-insertion. The purpose > +# is to recreate a CXL region during insertion that existed before > +# removal. > +# > +# '--unplug' sub-command option will save the CXL region information to > +# a file before a manual hotplug removal. > +# > +# '--plugin' sub-command after hotplug insertion will recreate a device > +# region using the details from the configuration file. > +# > +# '--swap' sub-command will run the same functionality as '--unplug', > +# then prompt for when the card is plugged in, and then execute the > +# '--plugin' functionality. > +# > +# '--list' sub-command will list the CXL devices and associated regions. > +# > + > +import json > +import subprocess > +import os > +import argparse > +import logging > +import tempfile > + > +class cxl_json: > + > + cxl_memdev_json = {} > + decoders_root_json = {} > + regions_decoder_json = {} > + daxregion_json = {} > + daxregion_devices_json = {} > + > + def cxl_list_memdev(self, memdev): > + result = subprocess.run(["cxl", "list", "-m", memdev, "-RBMTXEPD"], > + capture_output=True, text=True) > + if result.returncode != 0: > + print("Error: cxl list command failed for: " + memdev) > + exit(1) > + > + result_json = json.loads(result.stdout) > + > + if not result_json: > + print("CXL list is empty for: " + memdev) > + exit(1) > + > + return(result_json[0]) > + > + # Cache embedded json dictionaries to make more easily accessible. > + # This helps in later processing . > + # > + # If the CXL json dictionary is missing any CXL components than this > + # implies a CXL device is not configured. In this case exit with an > + # error and message. > + def decode_json(self, fatal_error = True): > + for key in self.cxl_memdev_json: > + if key.startswith('decoders:root'): > + self.decoders_root_json = self.cxl_memdev_json[key][0] > + if self.decoders_root_json == None or \ > + len(self.decoders_root_json) == 0: > + if fatal_error == True: > + print("Error: Failed to find decoder root CXL json.") > + exit(1) > + else: > + return > + > + regions_decoder_key = "regions:" + self.decoders_root_json.get('decoder') > + if self.decoders_root_json.get(regions_decoder_key) == None: > + if fatal_error == True: > + print("Error: Failed to find region decoder in CXL json.") > + exit(1) > + else: > + return > + > + self.regions_decoder_json = self.decoders_root_json.get(regions_decoder_key)[0] > + if len(self.regions_decoder_json) == 0: > + if fatal_error == True: > + print("Error: Failed to find region decoder in CXL json.") > + exit(1) > + else: > + return > + > + # PMEM CXL JSON does not include DAX keys searched for below, return early > + if 'pmem' == self.regions_decoder_json.get('type'): > + return; > + > + for key in self.regions_decoder_json: > + if key.startswith('daxregion'): > + self.daxregion_json = self.regions_decoder_json[key] > + if len(self.daxregion_json) == 0: > + if fatal_error == True: > + print("Error: Failed to find daxregion in CXL json.") > + exit(1) > + else: > + return > + > + for key in self.daxregion_json: > + if key.startswith('devices'): > + self.daxregion_devices_json = self.daxregion_json[key][0] > + if len(self.daxregion_devices_json) == 0: > + if fatal_error == True: > + print("Error: Failed to find daxregion devices in CXL json.") > + exit(1) > + else: > + return > + > + def get_block_size_bytes(self): > + result = subprocess.run(["cat", "/sys/devices/system/memory/block_size_bytes"], > + capture_output=True, text=True) > + if result.returncode != 0: > + print("Error: cxl destroy-region command failed. rc = " + > + str(result.returncode)) > + exit(1) > + return int(result.stdout, 16) > + > + def is_region_movable(self): > + > + # PMEM doesn't have movable concept > + if 'pmem' == self.regions_decoder_json.get('type'): > + return True; > + > + block_size_bytes = self.get_block_size_bytes() > + resource = self.regions_decoder_json.get('resource') > + resource_size = self.regions_decoder_json.get('size') > + start_block = int(resource/block_size_bytes) > + stop_block = int((resource+resource_size)/block_size_bytes - 1) > + is_movable = True > + > + for i in range(0,(stop_block-start_block + 1)): > + block = start_block + i > + block_str = ("/sys/devices/system/memory/memory" + str(block) + > + "/valid_zones") > + result = subprocess.run(["cat", block_str], > + capture_output=True, text=True) > + if result.returncode != 0: > + print("Error: Block cat command failed. rc = " + > + str(result.returncode)) > + exit(1) > + > + if "Movable" not in result.stdout: > + is_movable = False > + break; > + > + return is_movable > + > +class cxl_unplug(cxl_json): > + > + def __init__(self, memdev, config_file): > + # Capture the current CXL (and DAX) configurations. Save > + # configurations to file for using later with '--plugin' > + # hot-plug adding the memory device. > + self.cxl_memdev_json = self.cxl_list_memdev(memdev) > + self.serialize_json(self.cxl_memdev_json, config_file) > + self.decode_json() > + > + if (self.regions_decoder_json.get('interleave_ways')>1): > + print("Interleaved devices are not supported.") > + exit(1) > + > + def serialize_json(self, json_str, filename): > + with open( filename , "w" ) as write: > + json.dump(json_str, write) > + > + def is_dax_memory_offline(self, dax_dev_json): > + result = subprocess.run(["daxctl", "list"], > + capture_output=True, text=True) > + if result.returncode != 0: > + print("Error: daxctl offline command failed command. rc = " + > + str(result.returncode)) > + exit(1) > + > + dax_dev_online_mb = json.loads(result.stdout)[0]["online_memblocks"] > + logging.debug("online_memblocks = " + > + str(json.loads(result.stdout)[0]["online_memblocks"])) > + return dax_dev_online_mb == 0 > + > + def offline_dax_memory(self): > + # Note: self.daxregion_devices_json['movable'] JSON key is missing > + # when True (ndctl v77). As a result, use the function region_movable() > + if self.is_region_movable() == False: > + print("Error: Entire region memory is not zone movable.") > + exit(1) > + > + dax_dev = self.daxregion_devices_json['chardev'] > + result=subprocess.run(["daxctl", "offline-memory", dax_dev], > + capture_output=True, text=True) > + if result.returncode != 0: > + if self.is_dax_memory_offline(dax_dev) == False: > + print("Error: daxctl offline command failed. rc = " + > + str(result.returncode)) > + exit(1) > + > + def offline_memory(self): > + if 'ram' == self.regions_decoder_json.get('type'): > + self.offline_dax_memory() > + > + def cxl_destroy_region(self): > + result = subprocess.run(["cxl", "destroy-region", > + str(self.regions_decoder_json.get('region')), > + "--force"], > + capture_output=True, text=True) > + if result.returncode != 0: > + print("Error: cxl destroy-region command failed. rc = " + > + str(result.returncode)) > + exit(1) > + > + def unplug(self): > + unplug_dev.offline_memory() > + unplug_dev.cxl_destroy_region() > + print("Device \'" + args.memdev + "\' can now be safely removed.") > + > +class cxl_plugin(cxl_json): > + > + def __init__(self, config_file): > + f = open(config_file) > + self.cxl_memdev_json = json.load(f) > + self.decode_json() > + > + if (self.regions_decoder_json.get('interleave_ways')>1): > + print("Interleaved devices are not supported.") > + exit(1) > + > + def cxl_create_region(self, cxl_memdev_json, memdev): > + type = self.regions_decoder_json.get('type'); > + decoder = self.decoders_root_json.get('decoder') > + interleave_ways = self.regions_decoder_json.get('interleave_ways') > + interleave_granularity = self.regions_decoder_json.get('interleave_granularity') > + size = self.regions_decoder_json.get('size') > + > + result = subprocess.run(["cxl", "create-region", > + "-m", memdev, > + "-t", type, > + "-d", decoder, > + "-w", str(interleave_ways), > + "-s", str(size), > + "--debug"], > + capture_output=True, text=True) > + if result.returncode!=0: > + print("Error: cxl destroy-region command failed. rc = " + > + str(result.returncode)) > + exit(1) > + > + def plugin(self, memdev): > + self.cxl_create_region(self.cxl_memdev_json, memdev) > + print("Region created for \'" + memdev + "\'.") > + > +class cxl_list(cxl_json): > + > + cxl_memdevs_json = {} > + > + def __init__(self): > + self.cxl_memdevs_json = self.cxl_list_memdevs() > + > + def display_memdevs(self): > + if (self.cxl_memdevs_json == None): > + print("Error: Failed to find memory devices") > + return > + > + if (len(self.cxl_memdevs_json) == 0): > + print("Error: Failed to find memory devices") > + return > + > + print("Device Region Name Region Size Movable") > + print("====== =========== =========== =======") > + > + for memdev in self.cxl_memdevs_json: > + self.cxl_memdev_json = self.cxl_list_memdev(memdev['memdev']) > + self.decode_json(fatal_error = False) > + if len(self.regions_decoder_json) != 0: > + region = self.regions_decoder_json.get('region'); > + size = self.regions_decoder_json.get('size') > + is_movable_str = "True" > + if self.is_region_movable() == 0: > + is_movable_str = "False" > + else: > + region = 'NA' > + size = 'NA' > + is_movable_str = 'NA' > + > + print("%6s %18s %18s %14s" % > + (memdev['memdev'], region, size, is_movable_str)) > + > + def cxl_list_memdevs(self): > + result = subprocess.run(["cxl", "list", "-M"], > + capture_output=True, text=True) > + if result.returncode != 0: > + print("Error: cxl list command failed.") > + exit(1) > + > + logging.debug("cxl_memdevs_json = " + result.stdout) > + self.cxl_memdevs_json = json.loads(result.stdout); > + > + return(self.cxl_memdevs_json) > + > +def init_args(): > + parser = argparse.ArgumentParser() > + subparsers = parser.add_subparsers(dest='subparser_name', required=True) > + > + parser_plugin = subparsers.add_parser('plugin') > + parser_plugin.add_argument('-m', dest='memdev', > + help='CXL memory device to prepare for unplug', > + required=True) > + parser_plugin.add_argument('-c', dest='config_file', > + help='CXL JSON configuration file to save/use', > + required=False, default="cxl.json") > + parser_plugin.add_argument('-d', '--debug', help='Enable debug logging', > + required=False, action='store_true') > + > + parser_unplug = subparsers.add_parser('unplug') > + parser_unplug.add_argument('-m', dest='memdev', > + help='CXL memory device to prepare for unplug', > + required=True) > + parser_unplug.add_argument('-c', dest='config_file', > + help='CXL JSON configuration file to save/use', > + required=False, default="cxl.json") > + parser_unplug.add_argument('-d', '--debug', help='Enable debugging', > + required=False, action='store_true') > + > + parser_list = subparsers.add_parser('list') > + parser_list.add_argument('-d', '--debug', help='Enable debugging', > + required=False, action='store_true') > + > + parser_swap = subparsers.add_parser('swap') > + parser_swap.add_argument('-m', dest='memdev', > + help='CXL memory device to swap', > + required=True) > + parser_swap.add_argument('-d', '--debug', help='Enable debugging', > + required=False, action='store_true') > + > + return parser.parse_args() > + > +args = init_args() > +if args.debug: > + logging.basicConfig(level=logging.DEBUG) > + > +logging.debug("args = " + str(args)) > + > +if args.subparser_name == 'list': > + list_dev = cxl_list(); > + list_dev.display_memdevs() > + > +if args.subparser_name == 'unplug': > + unplug_dev = cxl_unplug(args.memdev, args.config_file); > + unplug_dev.unplug() > + > +if args.subparser_name == 'plugin': > + plugin_dev = cxl_plugin(args.config_file); > + plugin_dev.plugin(args.memdev) > + > +if args.subparser_name == 'swap': > + f, filename = tempfile.mkstemp() > + unplug_dev = cxl_unplug(args.memdev, filename); > + unplug_dev.unplug() > + > + print("Card is ready for removal.") > + input("Press any key to continue after card reinsertion.") > + > + plugin_dev = cxl_plugin(filename); > + plugin_dev.plugin(args.memdev) > + os.close(f) > -- > 2.34.1 >
diff --git a/README.txt b/README.txt new file mode 100644 index 0000000..97fa793 --- /dev/null +++ b/README.txt @@ -0,0 +1,311 @@ + ____________________ + + CXL-HOTPLUG.README + ____________________ + + +Table of Contents +_________________ + +1. Purpose +2. Requirements +.. 1. Kernel - v6.4.0 +.. 2. ndctl - v77 or later +.. 3. QEMU - v8.0.3 + the following patches: +.. 4. Python modules: +.. 5. Additional tool details +3. Usage +4. Examples +.. 1. Swap a device +.. 2. Manually unplug and plugin a device +.. 3. Manually unplug and plugin a device (w/ step by step details) + + + + + +1 Purpose +========= + + Hotplug adding and removing CXL devices requires region management not + automatically provided. For instance, if a CXL device is added then a + region must be 'created' before the memory can be used. Likewise, a + region must be 'destroyed' before a CXL device can be + removed. Removing a CXL device before a region is 'destroyed' can + result in CXL device data loss or corruption. + + This tool aims to provide region delete and create automation for an + existing CXL device. The typical usage is to hotplug remove an + existing CXL device and then hotplug readd the same device immediately + or at some later time. + + An unplug function is provided that will 'destroy' the region making + the device ready for hotplug removal. Note, 'destroying' a PMEM region + incurrs no loss of data. 'destroying' a RAM region will lose the + region data. This tool saves the region structure information so that + the device's region can be created in the future. After the device is + hot hotplug added in the future the region information is used to + recreate the region. The region information is saved to a default file + or can be directed to a specific file provided on the tool comandline. + + This tool provides a swap function that automatically executes the + unplug and plugin functions. + + This tool is currently limited to non interleaved devices. + + +2 Requirements +============== + + Tested using the following: + + +2.1 Kernel - v6.4.0 +~~~~~~~~~~~~~~~~~~~ + + To include the following kernel config settings: + ,---- + | scripts/config --enable LIBNVDIMM + | scripts/config --enable CONFIG_CXL_BUS + | scripts/config --enable CONFIG_CXL_PCI + | scripts/config --enable CONFIG_CXL_ACPI + | scripts/config --enable CONFIG_CXL_MEM + | scripts/config --enable CONFIG_CXL_PMEM + | scripts/config --enable CONFIG_CXL_PORT + | scripts/config --enable CONFIG_CXL_SUSPEND + | scripts/config --enable CONFIG_CXL_REGION + | scripts/config --enable CONFIG_CXL_REGION_INVALIDATION_TEST + | scripts/config --enable CONFIG_PCIEAER_CXL + | scripts/config --enable CONFIG_TRANSPARENT_HUGEPAGE + | scripts/config --enable CONFIG_DEV_DAX + | scripts/config --enable CONFIG_DEV_DAX_HMEM + | scripts/config --enable CONFIG_DEV_DAX_KMEM + | scripts/config --enable CONFIG_DEV_DAX_PMEM + `---- + The above configures for statically linked drivers. They could be + dynamically linked as modules as well. + + +2.2 ndctl - v77 or later +~~~~~~~~~~~~~~~~~~~~~~~~ + + +2.3 QEMU - v8.0.3 + the following patches: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + 20230227163157.6621-1-Jonathan.Cameron@huawei.com + 20230303150908.27889-1-Jonathan.Cameron@huawei.com + <https://lore.kernel.org/linux-cxl/20230303152903.28103-1-Jonathan.Cameron@huawei.com/\#r> + + +2.4 Python modules: +~~~~~~~~~~~~~~~~~~~ + + The following python modules are required: json subprocess os argparse + logging + + +2.5 Additional tool details +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + <https://confluence.amd.com/display/ALK/CXL+QEMU> + + +3 Usage +======= + + The utility requires one of the sub-commands: plugin,unplug,list,swap. + + Commandline usage information: + ,---- + | usage: cxl-hotplug.py [-h] {plugin,unplug,list,swap} ... + | + | positional arguments: + | {plugin,unplug,list,swap} + | + | options: + | -h, --help show this help message and exit + `---- + + Commandline plugin usage information: + ,---- + | usage: cxl-hotplug.py plugin [-h] -m MEMDEV [-c CONFIG_FILE] [-d] + | + | options: + | -h, --help show this help message and exit + | -m MEMDEV CXL memory device to prepare for unplug + | -c CONFIG_FILE CXL JSON configuration file to use + | -d, --debug Enable debugging + `---- + + Commandline unplug usage information: + ,---- + | usage: cxl-hotplug.py unplug [-h] -m MEMDEV [-c CONFIG_FILE] [-d] + | + | options: + | -h, --help show this help message and exit + | -m MEMDEV CXL memory device to prepare for unplug + | -c CONFIG_FILE CXL JSON configuration file to save + | -d, --debug Enable debugging + `---- + + Commandline swap usage information: + ,---- + | usage: cxl-hotplug.py swap [-h] -m MEMDEV [-d] + | + | options: + | -h, --help show this help message and exit + | -m MEMDEV CXL memory device to swap + | -d, --debug Enable debugging + `---- + + +4 Examples +========== + +4.1 Swap a device +~~~~~~~~~~~~~~~~~ + + ,---- + | # cxl create-region -t ram -m mem0 -d decoder0.0 -w 1 -s 256M + | # ./cxl-hotplug.py swap -m mem0 + | Device 'mem0' can now be safely removed. + | Card is ready for removal. + | Press any key to continue after card reinsertion. + | Region created for 'mem0'. + `---- + + +4.2 Manually unplug and plugin a device +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + ,---- + | # cxl create-region -t ram -m mem0 -d decoder0.0 -w 1 -s 256M + | # ./cxl-hotplug.py unplug -m mem0 -c my-cxl.json + | # ./cxl-hotplug.py plugin -m mem0 -c my-cxl.json + `---- + + +4.3 Manually unplug and plugin a device (w/ step by step details) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + ,---- + | # cxl list + | [ + | { + | "memdev":"mem0", + | "ram_size":268435456, + | "serial":0, + | "host":"0000:0d:00.0" + | } + | ] + | + | # ./cxl-hotplug.py list + | Device Region Name Region Size Movable + | ====== =========== =========== ======= + | mem0 NA NA NA + | + | # cxl create-region -t ram -m mem0 -d decoder0.0 -w 1 -s 256M + | { + | "region":"region0", + | "resource":"0x890000000", + | "size":"256.00 MiB (268.44 MB)", + | "type":"ram", + | "interleave_ways":1, + | "interleave_granularity":256, + | "decode_state":"commit", + | "mappings":[ + | { + | "position":0, + | "memdev":"mem0", + | "decoder":"decoder2.0" + | } + | ] + | } + | cxl region: cmd_create_region: created 1 region + | + | # cxl list + | [ + | { + | "memdevs":[ + | { + | "memdev":"mem0", + | "ram_size":268435456, + | "serial":0, + | "host":"0000:0d:00.0" + | } + | ] + | }, + | { + | "regions":[ + | { + | "region":"region0", + | "resource":36775657472, + | "size":268435456, + | "type":"ram", + | "interleave_ways":1, + | "interleave_granularity":256, + | "decode_state":"commit" + | } + | ] + | } + | ] + | + | # ./cxl-hotplug.py list + | Device Region Name Region Size Movable + | ====== =========== =========== ======= + | mem0 region0 268435456 True + | + | # ./cxl-hotplug.py unplug -m mem0 + | Device 'mem0' can now be safely removed. + | + | # cxl list + | [ + | { + | "memdev":"mem0", + | "ram_size":268435456, + | "serial":0, + | "host":"0000:0d:00.0" + | } + | ] + | + | # ./cxl-hotplug.py list + | Device Region Name Region Size Movable + | ====== =========== =========== ======= + | mem0 NA NA NA + | + | # ./cxl-hotplug.py plugin -m mem0 + | Region created for 'mem0'. + | + | # cxl list + | [ + | { + | "memdevs":[ + | { + | "memdev":"mem0", + | "ram_size":268435456, + | "serial":0, + | "host":"0000:0d:00.0" + | } + | ] + | }, + | { + | "regions":[ + | { + | "region":"region0", + | "resource":36775657472, + | "size":268435456, + | "type":"ram", + | "interleave_ways":1, + | "interleave_granularity":256, + | "decode_state":"commit" + | } + | ] + | } + | ] + | + | # ./cxl-hotplug.py list + | Device Region Name Region Size Movable + | ====== =========== =========== ======= + | mem0 region0 268435456 True + `---- diff --git a/cxl-hotplug.py b/cxl-hotplug.py new file mode 100755 index 0000000..971e4e7 --- /dev/null +++ b/cxl-hotplug.py @@ -0,0 +1,366 @@ +#!/usr/bin/python3 +# SPDX-License-Identifier: LGPL-2.1 +# +# Copyright (C) 2023, Advanced Micro Devices (AMD). All rights reserved. +# +# @Author - Terry Bowman +# +# Utility to support CXL hotplug removal and re-insertion. The purpose +# is to recreate a CXL region during insertion that existed before +# removal. +# +# '--unplug' sub-command option will save the CXL region information to +# a file before a manual hotplug removal. +# +# '--plugin' sub-command after hotplug insertion will recreate a device +# region using the details from the configuration file. +# +# '--swap' sub-command will run the same functionality as '--unplug', +# then prompt for when the card is plugged in, and then execute the +# '--plugin' functionality. +# +# '--list' sub-command will list the CXL devices and associated regions. +# + +import json +import subprocess +import os +import argparse +import logging +import tempfile + +class cxl_json: + + cxl_memdev_json = {} + decoders_root_json = {} + regions_decoder_json = {} + daxregion_json = {} + daxregion_devices_json = {} + + def cxl_list_memdev(self, memdev): + result = subprocess.run(["cxl", "list", "-m", memdev, "-RBMTXEPD"], + capture_output=True, text=True) + if result.returncode != 0: + print("Error: cxl list command failed for: " + memdev) + exit(1) + + result_json = json.loads(result.stdout) + + if not result_json: + print("CXL list is empty for: " + memdev) + exit(1) + + return(result_json[0]) + + # Cache embedded json dictionaries to make more easily accessible. + # This helps in later processing . + # + # If the CXL json dictionary is missing any CXL components than this + # implies a CXL device is not configured. In this case exit with an + # error and message. + def decode_json(self, fatal_error = True): + for key in self.cxl_memdev_json: + if key.startswith('decoders:root'): + self.decoders_root_json = self.cxl_memdev_json[key][0] + if self.decoders_root_json == None or \ + len(self.decoders_root_json) == 0: + if fatal_error == True: + print("Error: Failed to find decoder root CXL json.") + exit(1) + else: + return + + regions_decoder_key = "regions:" + self.decoders_root_json.get('decoder') + if self.decoders_root_json.get(regions_decoder_key) == None: + if fatal_error == True: + print("Error: Failed to find region decoder in CXL json.") + exit(1) + else: + return + + self.regions_decoder_json = self.decoders_root_json.get(regions_decoder_key)[0] + if len(self.regions_decoder_json) == 0: + if fatal_error == True: + print("Error: Failed to find region decoder in CXL json.") + exit(1) + else: + return + + # PMEM CXL JSON does not include DAX keys searched for below, return early + if 'pmem' == self.regions_decoder_json.get('type'): + return; + + for key in self.regions_decoder_json: + if key.startswith('daxregion'): + self.daxregion_json = self.regions_decoder_json[key] + if len(self.daxregion_json) == 0: + if fatal_error == True: + print("Error: Failed to find daxregion in CXL json.") + exit(1) + else: + return + + for key in self.daxregion_json: + if key.startswith('devices'): + self.daxregion_devices_json = self.daxregion_json[key][0] + if len(self.daxregion_devices_json) == 0: + if fatal_error == True: + print("Error: Failed to find daxregion devices in CXL json.") + exit(1) + else: + return + + def get_block_size_bytes(self): + result = subprocess.run(["cat", "/sys/devices/system/memory/block_size_bytes"], + capture_output=True, text=True) + if result.returncode != 0: + print("Error: cxl destroy-region command failed. rc = " + + str(result.returncode)) + exit(1) + return int(result.stdout, 16) + + def is_region_movable(self): + + # PMEM doesn't have movable concept + if 'pmem' == self.regions_decoder_json.get('type'): + return True; + + block_size_bytes = self.get_block_size_bytes() + resource = self.regions_decoder_json.get('resource') + resource_size = self.regions_decoder_json.get('size') + start_block = int(resource/block_size_bytes) + stop_block = int((resource+resource_size)/block_size_bytes - 1) + is_movable = True + + for i in range(0,(stop_block-start_block + 1)): + block = start_block + i + block_str = ("/sys/devices/system/memory/memory" + str(block) + + "/valid_zones") + result = subprocess.run(["cat", block_str], + capture_output=True, text=True) + if result.returncode != 0: + print("Error: Block cat command failed. rc = " + + str(result.returncode)) + exit(1) + + if "Movable" not in result.stdout: + is_movable = False + break; + + return is_movable + +class cxl_unplug(cxl_json): + + def __init__(self, memdev, config_file): + # Capture the current CXL (and DAX) configurations. Save + # configurations to file for using later with '--plugin' + # hot-plug adding the memory device. + self.cxl_memdev_json = self.cxl_list_memdev(memdev) + self.serialize_json(self.cxl_memdev_json, config_file) + self.decode_json() + + if (self.regions_decoder_json.get('interleave_ways')>1): + print("Interleaved devices are not supported.") + exit(1) + + def serialize_json(self, json_str, filename): + with open( filename , "w" ) as write: + json.dump(json_str, write) + + def is_dax_memory_offline(self, dax_dev_json): + result = subprocess.run(["daxctl", "list"], + capture_output=True, text=True) + if result.returncode != 0: + print("Error: daxctl offline command failed command. rc = " + + str(result.returncode)) + exit(1) + + dax_dev_online_mb = json.loads(result.stdout)[0]["online_memblocks"] + logging.debug("online_memblocks = " + + str(json.loads(result.stdout)[0]["online_memblocks"])) + return dax_dev_online_mb == 0 + + def offline_dax_memory(self): + # Note: self.daxregion_devices_json['movable'] JSON key is missing + # when True (ndctl v77). As a result, use the function region_movable() + if self.is_region_movable() == False: + print("Error: Entire region memory is not zone movable.") + exit(1) + + dax_dev = self.daxregion_devices_json['chardev'] + result=subprocess.run(["daxctl", "offline-memory", dax_dev], + capture_output=True, text=True) + if result.returncode != 0: + if self.is_dax_memory_offline(dax_dev) == False: + print("Error: daxctl offline command failed. rc = " + + str(result.returncode)) + exit(1) + + def offline_memory(self): + if 'ram' == self.regions_decoder_json.get('type'): + self.offline_dax_memory() + + def cxl_destroy_region(self): + result = subprocess.run(["cxl", "destroy-region", + str(self.regions_decoder_json.get('region')), + "--force"], + capture_output=True, text=True) + if result.returncode != 0: + print("Error: cxl destroy-region command failed. rc = " + + str(result.returncode)) + exit(1) + + def unplug(self): + unplug_dev.offline_memory() + unplug_dev.cxl_destroy_region() + print("Device \'" + args.memdev + "\' can now be safely removed.") + +class cxl_plugin(cxl_json): + + def __init__(self, config_file): + f = open(config_file) + self.cxl_memdev_json = json.load(f) + self.decode_json() + + if (self.regions_decoder_json.get('interleave_ways')>1): + print("Interleaved devices are not supported.") + exit(1) + + def cxl_create_region(self, cxl_memdev_json, memdev): + type = self.regions_decoder_json.get('type'); + decoder = self.decoders_root_json.get('decoder') + interleave_ways = self.regions_decoder_json.get('interleave_ways') + interleave_granularity = self.regions_decoder_json.get('interleave_granularity') + size = self.regions_decoder_json.get('size') + + result = subprocess.run(["cxl", "create-region", + "-m", memdev, + "-t", type, + "-d", decoder, + "-w", str(interleave_ways), + "-s", str(size), + "--debug"], + capture_output=True, text=True) + if result.returncode!=0: + print("Error: cxl destroy-region command failed. rc = " + + str(result.returncode)) + exit(1) + + def plugin(self, memdev): + self.cxl_create_region(self.cxl_memdev_json, memdev) + print("Region created for \'" + memdev + "\'.") + +class cxl_list(cxl_json): + + cxl_memdevs_json = {} + + def __init__(self): + self.cxl_memdevs_json = self.cxl_list_memdevs() + + def display_memdevs(self): + if (self.cxl_memdevs_json == None): + print("Error: Failed to find memory devices") + return + + if (len(self.cxl_memdevs_json) == 0): + print("Error: Failed to find memory devices") + return + + print("Device Region Name Region Size Movable") + print("====== =========== =========== =======") + + for memdev in self.cxl_memdevs_json: + self.cxl_memdev_json = self.cxl_list_memdev(memdev['memdev']) + self.decode_json(fatal_error = False) + if len(self.regions_decoder_json) != 0: + region = self.regions_decoder_json.get('region'); + size = self.regions_decoder_json.get('size') + is_movable_str = "True" + if self.is_region_movable() == 0: + is_movable_str = "False" + else: + region = 'NA' + size = 'NA' + is_movable_str = 'NA' + + print("%6s %18s %18s %14s" % + (memdev['memdev'], region, size, is_movable_str)) + + def cxl_list_memdevs(self): + result = subprocess.run(["cxl", "list", "-M"], + capture_output=True, text=True) + if result.returncode != 0: + print("Error: cxl list command failed.") + exit(1) + + logging.debug("cxl_memdevs_json = " + result.stdout) + self.cxl_memdevs_json = json.loads(result.stdout); + + return(self.cxl_memdevs_json) + +def init_args(): + parser = argparse.ArgumentParser() + subparsers = parser.add_subparsers(dest='subparser_name', required=True) + + parser_plugin = subparsers.add_parser('plugin') + parser_plugin.add_argument('-m', dest='memdev', + help='CXL memory device to prepare for unplug', + required=True) + parser_plugin.add_argument('-c', dest='config_file', + help='CXL JSON configuration file to save/use', + required=False, default="cxl.json") + parser_plugin.add_argument('-d', '--debug', help='Enable debug logging', + required=False, action='store_true') + + parser_unplug = subparsers.add_parser('unplug') + parser_unplug.add_argument('-m', dest='memdev', + help='CXL memory device to prepare for unplug', + required=True) + parser_unplug.add_argument('-c', dest='config_file', + help='CXL JSON configuration file to save/use', + required=False, default="cxl.json") + parser_unplug.add_argument('-d', '--debug', help='Enable debugging', + required=False, action='store_true') + + parser_list = subparsers.add_parser('list') + parser_list.add_argument('-d', '--debug', help='Enable debugging', + required=False, action='store_true') + + parser_swap = subparsers.add_parser('swap') + parser_swap.add_argument('-m', dest='memdev', + help='CXL memory device to swap', + required=True) + parser_swap.add_argument('-d', '--debug', help='Enable debugging', + required=False, action='store_true') + + return parser.parse_args() + +args = init_args() +if args.debug: + logging.basicConfig(level=logging.DEBUG) + +logging.debug("args = " + str(args)) + +if args.subparser_name == 'list': + list_dev = cxl_list(); + list_dev.display_memdevs() + +if args.subparser_name == 'unplug': + unplug_dev = cxl_unplug(args.memdev, args.config_file); + unplug_dev.unplug() + +if args.subparser_name == 'plugin': + plugin_dev = cxl_plugin(args.config_file); + plugin_dev.plugin(args.memdev) + +if args.subparser_name == 'swap': + f, filename = tempfile.mkstemp() + unplug_dev = cxl_unplug(args.memdev, filename); + unplug_dev.unplug() + + print("Card is ready for removal.") + input("Press any key to continue after card reinsertion.") + + plugin_dev = cxl_plugin(filename); + plugin_dev.plugin(args.memdev) + os.close(f)
Hotplug removing a CXL device and hotplug reinserting the same device currently requires manual interaction for managing the device region. The CXL device region must be manually destroyed before hotplug removal and manually created after hotplug reinsertion. Create a script to automatically destroy and recreate the region during CXL device hotplug remove-reinsert. Save region characteristics to a file before destroying the region and hotplug remove. Use the region characteristics stored in the file to recreate the region after hotplug reinsert. Signed-off-by: Terry Bowman <terry.bowman@amd.com> --- README.txt | 311 +++++++++++++++++++++++++++++++++++++++++ cxl-hotplug.py | 366 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 677 insertions(+) create mode 100644 README.txt create mode 100755 cxl-hotplug.py