Thanks, this was enabled (by default once no failures were found in the lab) to find buggy machines/buggy code ahead of enabling NUMA support in 6074aef (2023).

The code could be changed to disable NUMA support (numa-affinity-policy) when a buggy table is found
(if one value is invalid then we can't trust the rest of the table either).

For now reverting the commit should make it work again for the affected user

Although reverting the commit might cause xenopsd to call Lazy.force from a thread, which will require the ocaml/ocaml@ed4695a backported OCaml runtime commit to avoid 'Lazy.force' from causing a segfault.
(having that patch would be a good idea anyway).

NUMAPlacement.make should return an option, IMO.

Tentative commit: 5f49ba1

btw here is the ACPI spec https://uefi.org/specs/ACPI/6.5_A/06_Device_Configuration.html#system-locality-information-table that says (i, i) is always a value of 10.

@stormi would be useful to see the full xl-dmesg.out and an acpidump of the relevant SRAT/SLIT tables.
See __node_distance function in Xen: 0xff (-1) could mean an unreachable node, or a value 0-9.

If that is indeed an unreachable NUMA node (e.g. one that has no online CPUs) then we can safely ignore it and still use the rest of the distance matrix.

The tentative commit from yesterday fixed the issue for the user.

Jan 21 11:01:49 v-server-1 xenopsd-xc: [debug||0 ||topology] Distances: [|[|10; 4294967295|]; [|4294967295; 4294967295|]|]
Jan 21 11:01:49 v-server-1 xenopsd-xc: [debug||0 ||topology] CPU2Node: [|0; 0; 0; 0; 0; 0; 1; 1; 1; 1; 1; 1|]
Jan 21 11:01:49 v-server-1 xenopsd-xc: [debug||0 ||xenops] Host NUMA information: Some\x0A  { "distances" = [|[|10; 4294967295|]; [|4294967295; 4294967295|]|];\x0A    "cpu2node" = [|0; 0; 0; 0; 0; 0; 1; 1; 1; 1; 1; 1|];\x0A    "node_cpus" = [|[0; 1; 2; 3; 4; 5]; [6; 7; 8; 9; 10; 11]|] }
Jan 21 11:01:49 v-server-1 xenopsd-xc: [debug||0 ||xenops] NUMA node 0: 30836379648/36507222016 memory free
Jan 21 11:01:49 v-server-1 xenopsd-xc: [debug||0 ||xenops] NUMA node 1: 0/0 memory free
Jan 21 11:01:49 v-server-1 xenopsd-xc: [debug||0 ||xenops] xenstore is responding to requests

I've got a xen-bugtool from the user so I'll try to find the data you asked.

@edwintorok

acpidump.out.txt
xl-dmesg.out.txt

@psafont do you need tests from the user with your latest patch?

(XEN) [0000002112563a15] SRAT: Node 0 PXM 0 [0000000000000000, 000000007fffffff]
(XEN) [0000002112564b11] SRAT: Node 0 PXM 0 [0000000100000000, 000000087fffffff]
(XEN) [00000021125670a5] SRAT: node 1 has no memory

OK, so maybe not a buggy ACPI table after all. If node 1 has no memory then we should be ignoring it. Probably quite rare to have such a system (if it has no memory, then what is its purpose?)

A node is a node whether it has memory or not.

(Mis)configurations like this were very common in the AMD MagnyCours days, and one OEM shipped a lot of systems with DIMMs populated in a less-optimal configuration.

FWIW here is the decoded SLIT and SRAT table using xxd -r and iasl -d

slit.dsl.txt
srat.dsl.txt

ah can we also have NUMA nodes with no CPUs in it?
We should probably filter the matrix we receive to:

• eliminate unreachable nodes (i.e. those with distance -1 in (i,i)). Although we also need to fix how we interpret the entry, because there seems to be some unsigned/signed int mismatch there (we expect the value to be unsigned I think, maybe that is wrong)
• eliminate nodes without CPUs, or nodes without any online CPUs

ah can we also have NUMA nodes with no CPUs in it?

Yes. If you e.g. downcore to a single CPU, and the package has multiple memory controllers, then you'll get a NUMA node with IO and RAM, but no (online) CPUs. Also, manually playing with xen-hptool cpu-offline can create a configuration with no online CPUs on a node.

Furthermore, In principle, a memory controller more hops than usual away from the cores could manifest like this, but I'm not aware of a platform which looks like this naturally.

What's weird here is that the SLIT declares a single locality, while the SRAT lists two proximity domains. I expect Xen has filled in the blanks with -1, including part of the leading diagonal of the matrix.

@psafont do you need tests from the user with your latest patch?

If they're up for it, sure. there was a bug in the previous one