[0/1] mm/fake-numa: allow later numa node hotplug

Message ID	20250106120659.359610-1-bfaccini@nvidia.com (mailing list archive)
Headers	show Return-Path: <owner-linux-mm@kvack.org> From: Bruno Faccini <bfaccini@nvidia.com> To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, akpm@linux-foundation.org, rppt@kernel.org, david@redhat.com, ziy@nvidia.com, jhubbard@nvidia.com, mrusiniak@nvidia.com, Bruno Faccini <bfaccini@nvidia.com> Subject: [PATCH 0/1] mm/fake-numa: allow later numa node hotplug Date: Mon, 6 Jan 2025 04:06:58 -0800 Message-ID: <20250106120659.359610-1-bfaccini@nvidia.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org Precedence: bulk
Series	mm/fake-numa: allow later numa node hotplug \| expand [0/1] mm/fake-numa: allow later numa node hotplug [1/1] mm/fake-numa: allow later numa node hotplug

Message ID

20250106120659.359610-1-bfaccini@nvidia.com (mailing list archive)

Headers

From: Bruno Faccini <bfaccini@nvidia.com>
To: linux-kernel@vger.kernel.org
Cc: linux-mm@kvack.org,
	akpm@linux-foundation.org,
	rppt@kernel.org,
	david@redhat.com,
	ziy@nvidia.com,
	jhubbard@nvidia.com,
	mrusiniak@nvidia.com,
	Bruno Faccini <bfaccini@nvidia.com>
Subject: [PATCH 0/1] mm/fake-numa: allow later numa node hotplug
Date: Mon,  6 Jan 2025 04:06:58 -0800
Message-ID: <20250106120659.359610-1-bfaccini@nvidia.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?AIyrfWH47jVEC2qU3zShUaMP/Mad?=
	=?utf-8?q?E03q5ed2z1jA2DDqX8a4fUK2GodcGSA6lORxRqhYyRmMhAomQNNPCrorYUurncsJC?=
	=?utf-8?q?o9oF7w7+hGM9AQqcyO+ER1WZ1g7tLgC8RC1X1McK0vkMPTnbFHaYRWGZcbDd6VMW2?=
	=?utf-8?q?DEyKDel4r8gVJmqxPxeGkuEy4DcxjVbddX8d8SJtFfNAnob/uNNeZ0R5mcS5Sjob5?=
	=?utf-8?q?sD/CgBk6QBMxWBVfLoTepCa1c5RD71f9xuRTVQ0iaVHqvVm75JuMQ5Ig+F+0xGBTL?=
	=?utf-8?q?c7KUPb+YW2Ud5uaGEigFzdB6MwfYCf6KQaPSTD3+qP1HQM7qaOLJGwsjdJRojk7dm?=
	=?utf-8?q?uv6/WCn6mLiFzVwzBNWknrTCbWB0tnVpzIsc62gwonbp+HqRpPRDXhU2WbmtzRkRi?=
	=?utf-8?q?i0Dzh8HV5l+MI114xqfnxcreemCJBJSAsehjpObFxZGgdLjUKO/aSqgZg+t5HJ3IJ?=
	=?utf-8?q?AcgSaVof0L5CviXDfXEqg6FWlTiwEv4R95OWOfI83kxGO1uN9tHuSfcw/8JXZMlmH?=
	=?utf-8?q?GL5Z2X87htnYwPmo/9A+yzWotYHpBhouI2SpawEkSuHQiUZnQzFmdpd9h/7a/0gfn?=
	=?utf-8?q?B2Jgh3lrtnuk0dCrayGqueo1ep76osaMMCIU9WjZEpEO7qgQUi8Kf6o380qYinv6j?=
	=?utf-8?q?UsMLf+cpjFrgl0UMs+DZinR7U23nKLeZh1jH4rahlahZ6Dj4v0hZM3rRxaCNJ4oNK?=
	=?utf-8?q?RnplXM8lziz6dABIxMQeDtFsOB3SweLrbkXhXWcn387BcWwLLLB/trVmb7nHfirKs?=
	=?utf-8?q?XNsktkYuZGNz71LsmjU+40JRtrbjXHkgxS5ZSXgWyD0n1TerAtZv8OGQSafwNtA2v?=
	=?utf-8?q?3+7HvavY36NJG96BblvOfHPsCHZFqrWzddLjBNXaiicPA4wUnxqmgLKum1XUqRYEN?=
	=?utf-8?q?gI99RIH7zCpKm5LtDM0h/906zMr9LM++fTQRKRNdybVKm65DQrypsmMTNL0+RZALa?=
	=?utf-8?q?r61QRheQrb7DHMb6hYwGU5NiheDgbcJFTGuAwMCIqOdp29fs1M4aWdSiqlARdKsMY?=
	=?utf-8?q?SllGxRY9NHRHoFl6HLvMfIMRosv/eR7JDefO6YgLJ8ywWGbwlylKbmh/CglqhyUc8?=
	=?utf-8?q?jgZ+Vf9sJerHbzrRXBVP2QweKqjqFFdU2k8kAmJthatbLgysffDU5laXN7gpCnUhR?=
	=?utf-8?q?uBHyPHQGT1Ddxa5ovp+XifmDimCC9JMyf1V2dPf1ur5Rmqg8Id60xsjr5Q7N1uFLf?=
	=?utf-8?q?c0esyr9Rw2WtcHRcyACfSWmjfRTZ+qji5ujrpj2n9n2QABayvrUGh2b3j3e6W33nE?=
	=?utf-8?q?uBgC0TAoZKWHg9DdfkgOObKW6dFQvLiWLyizLA94G+uH/IOElL4yvO2uA3MlI+Pos?=
	=?utf-8?q?yS3PcnHvsCuToSBfFfoilUVFd29V9Va5cOou1a86ASLloPXYXXWjnGR4UUC+7FX1j?=
	=?utf-8?q?eqGUZCyzEmhP8pjKV+M6I622BoswGPcZChhpRY5EAtlN+1KI471VJuTnXGmG08TJ+?=
	=?utf-8?q?CrL7oJ4Kop/AEMC4JXNsO8QEzJoHQsUE41t1tLQaikfWZJgFDXoX2PbmWbmpus7zW?=
	=?utf-8?q?cIOYZQgNX7cC?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 
 682ba125-7725-4c91-84b7-08dd2e4ab5c2
X-MS-Exchange-CrossTenant-AuthSource: MW4PR12MB7261.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jan 2025 12:07:36.9843
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 
 bT6TzYRrVcI7TFoH+T6Gc3RVdbk5faNGFww66g4NF6cHt47cr9ZMclP3WOn+IBHpIjECzUj0vzRXwZ9tcy5lcA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8312
X-Rspamd-Server: rspam04
X-Rspamd-Queue-Id: 7B2231C0008
X-Stat-Signature: ww9e4cjtg4ypa64rsf75ue1snec5uhun
X-Rspam-User: 
X-HE-Tag: 1736165264-479537
X-HE-Meta: 
 U2FsdGVkX1+S7BULXptxGAPvWMmNtd38pW+5vKpwAWgQu4IdxtfVr0HJFB++gu4y7ld6KoizmR7alQAPKkPq1mX9tM1KzcTbpCJt6erEywoAskRExlZrauSPd5pbVMBxRpNlPzZjCLretlDSCSZBvGjlvGKVqmf3XX8ECiE5RYHV7AQv20Zk8kx9I8B3BtjNQOKKwerR5hWzZHbwiJu6vcT3zBW+6cgv/hBcf2qsXm4jsvbKBexfeeftqI1vhpXJwcgDHpvS1+t+TvbwerpMWZFdnT1CEE863HVce1X1onsMyaOBiTfuQcMXfi152lFa5kOutT0oSnOSM6w7hFeH4Uzan5hWWvecdH77x2dvl6EwH0f2yijWm0WussMMQmCRYyaoaP2WEcTT6RGMcIJGKSjPWyzxzbAHpyjxvDAeWWYqOup51PdhJ/EIMHoTVl0myiUGjmBzT9G++NUQiyiFvIdYtl4QFxorUNK8SGA9WFijqAHVz5HaqxeMXBEV2q6Az8hO10Rqx3BEPGbE51pTA0lhOfia+4xbO5MTDFHhRIGr1uwE5K6/98E41CQmFr2ZObx574BthBHrAEYYmvHM83wYKmrSqy017V67Wf6U0QgeIufBC/iM8eSv0y2I4VIpEcqpXdYUa1k4BRN9DSDkCZ2G9xK6cIAbQ0iF8fbYUg7Va6iMh2C8yrMUtxR1nYlz+Xey35SMBOaOvZh4GDFH19P5osYwm5vJiuGsKxt5Q5hgtB6nT1OeZlrJVjgLplATSRE1edLnrgBvUD7g6lkx4HQedcGhDT/G+lPRAb9eHsBbCMVaioLVI+E9/7w7Ooh8VHOGZMqY5YQi9iVfesBKribdSIBj7j5HJo4SU7bhDTIFcI8mltunpzHXGgzqPypPqNYTfeYAPdVGyBR6hpsqCVaYdDEhipdWaSiIoQKIUnIcCDq9Xx02mVkoPd59efsQRFawbcHB1gPHpmldvL4
 r5oYEjqS
 b5VozKpoa5dqopkqAazQUMJJdzeY6PyruapaLYSA/+os/fOkhJG43juHQMAx5+h+kKjChHKuDp1NLHW/4JwQ1w0yLBw+16Q8tB93GnQiTKN5sN/1j6nOV30WuLhgcW73ZwDKzGpRTyZoOCDbCCjj/m0SLuXhSbyYDeBgs21URANrUoe5fZRIPxv22a9h6GPnJayjvfaqeu6Cw2urFLgpsfc+NPYPP+lEBCs4c8lOePFhgkQSvLScwgNqM5T2P2kyE4+9dmT8Brj4JydhbLty4vPPvDgTOVrYyiMYcb6+1K8oD4s2xutQGnmXp0w8peBKtFnxVkfeZQDQJ6RJjSkeAkY7pQbHsh+mBWJqg
X-Bogosity: Ham, tests=bogofilter, spamicity=0.002190, version=1.2.4
Sender: owner-linux-mm@kvack.org
Precedence: bulk
X-Loop: owner-majordomo@kvack.org
List-ID: <linux-mm.kvack.org>
List-Subscribe: <mailto:majordomo@kvack.org>
List-Unsubscribe: <mailto:majordomo@kvack.org>

Series

mm/fake-numa: allow later numa node hotplug | expand

Message

Bruno Faccini Jan. 6, 2025, 12:06 p.m. UTC

When trying to use fake-numa feature on our system where new
Numa nodes are being "hot-plugged" upon driver load, this fails
with the following type of message and warning with stack :

node 8 was absent from the node_possible_map
WARNING: CPU: 61 PID: 4259 at mm/memory_hotplug.c:1506
add_memory_resource+0x3dc/0x418

This issue prevents the use of the fake-NUMA debug feature with
the system's full configuration, when it has proven to be sometimes
extremely useful for performance testing of multi-tasked,
memory-bound applications, as it enables better isolation of
processes/ranks compared to fat NUMA nodes.

Usual numactl output after driver has “hot-plugged”/unveiled some
new Numa nodes with and without memory :
$ numactl --hardware
available: 9 nodes (0-8)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
65 66 67 68 69 70 71
node 0 size: 490037 MB
node 0 free: 484432 MB
node 1 cpus:
node 1 size: 97280 MB
node 1 free: 97279 MB
node 2 cpus:
node 2 size: 0 MB
node 2 free: 0 MB
node 3 cpus:
node 3 size: 0 MB
node 3 free: 0 MB
node 4 cpus:
node 4 size: 0 MB
node 4 free: 0 MB
node 5 cpus:
node 5 size: 0 MB
node 5 free: 0 MB
node 6 cpus:
node 6 size: 0 MB
node 6 free: 0 MB
node 7 cpus:
node 7 size: 0 MB
node 7 free: 0 MB
node 8 cpus:
node 8 size: 0 MB
node 8 free: 0 MB
node distances:
node   0   1   2   3   4   5   6   7   8
  0:  10  80  80  80  80  80  80  80  80
  1:  80  10  255  255  255  255  255  255  255
  2:  80  255  10  255  255  255  255  255  255
  3:  80  255  255  10  255  255  255  255  255
  4:  80  255  255  255  10  255  255  255  255
  5:  80  255  255  255  255  10  255  255  255
  6:  80  255  255  255  255  255  10  255  255
  7:  80  255  255  255  255  255  255  10  255
  8:  80  255  255  255  255  255  255  255  10


With recent M.Rapoport set of fake-numa patches in mm-everything
and using numa=fake=4 boot parameter :
$ numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
65 66 67 68 69 70 71
node 0 size: 122518 MB
node 0 free: 117141 MB
node 1 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
65 66 67 68 69 70 71
node 1 size: 219911 MB
node 1 free: 219751 MB
node 2 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
65 66 67 68 69 70 71
node 2 size: 122599 MB
node 2 free: 122541 MB
node 3 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
65 66 67 68 69 70 71
node 3 size: 122479 MB
node 3 free: 122408 MB
node distances:
node   0   1   2   3
  0:  10  10  10  10
  1:  10  10  10  10
  2:  10  10  10  10
  3:  10  10  10  10


With recent M.Rapoport set of fake-numa patches in mm-everything,
this patch on top, using numa=fake=4 boot parameter :
# numactl —hardware
available: 12 nodes (0-11)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
65 66 67 68 69 70 71
node 0 size: 122518 MB
node 0 free: 116429 MB
node 1 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
65 66 67 68 69 70 71
node 1 size: 122631 MB
node 1 free: 122576 MB
node 2 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
65 66 67 68 69 70 71
node 2 size: 122599 MB
node 2 free: 122544 MB
node 3 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
65 66 67 68 69 70 71
node 3 size: 122479 MB
node 3 free: 122419 MB
node 4 cpus:
node 4 size: 97280 MB
node 4 free: 97279 MB
node 5 cpus:
node 5 size: 0 MB
node 5 free: 0 MB
node 6 cpus:
node 6 size: 0 MB
node 6 free: 0 MB
node 7 cpus:
node 7 size: 0 MB
node 7 free: 0 MB
node 8 cpus:
node 8 size: 0 MB
node 8 free: 0 MB
node 9 cpus:
node 9 size: 0 MB
node 9 free: 0 MB
node 10 cpus:
node 10 size: 0 MB
node 10 free: 0 MB
node 11 cpus:
node 11 size: 0 MB
node 11 free: 0 MB
node distances:
node   0   1   2   3   4   5   6   7   8   9  10  11
  0:  10  10  10  10  80  80  80  80  80  80  80  80
  1:  10  10  10  10  80  80  80  80  80  80  80  80
  2:  10  10  10  10  80  80  80  80  80  80  80  80
  3:  10  10  10  10  80  80  80  80  80  80  80  80
  4:  80  80  80  80  10  255  255  255  255  255  255  255
  5:  80  80  80  80  255  10  255  255  255  255  255  255
  6:  80  80  80  80  255  255  10  255  255  255  255  255
  7:  80  80  80  80  255  255  255  10  255  255  255  255
  8:  80  80  80  80  255  255  255  255  10  255  255  255
  9:  80  80  80  80  255  255  255  255  255  10  255  255
 10:  80  80  80  80  255  255  255  255  255  255  10  255
 11:  80  80  80  80  255  255  255  255  255  255  255  10


Bruno Faccini (1):
  mm/fake-numa: allow later numa node hotplug

 drivers/acpi/numa/srat.c     | 86 ++++++++++++++++++++++++++++++++++++
 include/acpi/acpi_numa.h     |  5 +++
 include/linux/numa_memblks.h |  3 ++
 mm/numa_emulation.c          | 45 ++++++++++++++++---
 mm/numa_memblks.c            |  2 +-
 5 files changed, 133 insertions(+), 8 deletions(-)


base-commit: 4c10320ffbe7d6273b153b112a6e5f9b61ac008a

Comments

David Hildenbrand Jan. 7, 2025, 10:08 a.m. UTC | #1

Hi,

> 
> With recent M.Rapoport set of fake-numa patches in mm-everything
> and using numa=fake=4 boot parameter :
> $ numactl --hardware
> available: 4 nodes (0-3)
> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
> 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
> 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
> 65 66 67 68 69 70 71
> node 0 size: 122518 MB
> node 0 free: 117141 MB
> node 1 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
> 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
> 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
> 65 66 67 68 69 70 71
> node 1 size: 219911 MB
> node 1 free: 219751 MB
> node 2 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
> 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
> 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
> 65 66 67 68 69 70 71
> node 2 size: 122599 MB
> node 2 free: 122541 MB
> node 3 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
> 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
> 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
> 65 66 67 68 69 70 71
> node 3 size: 122479 MB
> node 3 free: 122408 MB

Why are all CPUs indicated as belonging to all nodes? Is that expected 
or a BUG?

I would have thought, just like memory, that one resource only belongs 
to one NUMA node.


> 
> With recent M.Rapoport set of fake-numa patches in mm-everything,
> this patch on top, using numa=fake=4 boot parameter :
> # numactl —hardware
> available: 12 nodes (0-11)
> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
> 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
> 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
> 65 66 67 68 69 70 71
> node 0 size: 122518 MB
> node 0 free: 116429 MB
> node 1 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
> 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
> 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
> 65 66 67 68 69 70 71
> node 1 size: 122631 MB
> node 1 free: 122576 MB
> node 2 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
> 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
> 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
> 65 66 67 68 69 70 71
> node 2 size: 122599 MB
> node 2 free: 122544 MB
> node 3 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
> 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
> 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
> 65 66 67 68 69 70 71
> node 3 size: 122479 MB
> node 3 free: 122419 MB
> node 4 cpus:
> node 4 size: 97280 MB
> node 4 free: 97279 MB


^ Is this where your driver hotplugged a single node and hotplugged memory?