Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable (more than) 1 TiB RAM. #78

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Enable (more than) 1 TiB RAM. #78

wants to merge 2 commits into from

Conversation

daym
Copy link
Contributor

@daym daym commented May 9, 2023

See also #51

Note: This PR is on top of issue-64 and issue-73 as well--and that is how it was tested.

Danny Milosavljevic added 2 commits May 5, 2023 20:38
#64 APCB configuration should not include BMC early link training
#51 APCB configuration should support 1 TiB RAM
@daym daym requested a review from jclulow May 9, 2023 20:58
@daym
Copy link
Contributor Author

daym commented May 9, 2023

Test result (on b9): success, and these messages (Note: 1 TiB = 0x10000000000 B):

../../oxide/os/startup.c:909: 'econtig' is 0xfffffffffc518000
MEMLIST: boot physinstalled:
        Address 0x0, size 0x80000000
        Address 0x100000000, size 0xfc00000000
        Address 0x10000000000, size 0x7ef00000
../../oxide/os/startup.c:922: 'physmax' is 0x1007eeff
../../oxide/os/startup.c:923: 'physinstalled' is 0xfcfef00
../../oxide/os/startup.c:924: 'memblocks' is 0x3
MEMLIST: boot reserved mem:
        Address 0x80000000, size 0x80000000
        Address 0xfd00000000, size 0x300000000
        Address 0x1007ef00000, size 0x11100000
MEMLIST: phys_install:
        Address 0x0, size 0x80000000
        Address 0x100000000, size 0xfc00000000
        Address 0x10000000000, size 0x7ef00000
MEMLIST: phys_avail:
        Address 0x0, size 0x200000
        Address 0x600000, size 0x200000
        Address 0xa00000, size 0x200000
        Address 0xe00000, size 0x400000
        Address 0x1400000, size 0x10f000
        Address 0x160f000, size 0x1000
        Address 0x1810000, size 0x1000
        Address 0x1911000, size 0x7e6ef000
        Address 0x100000000, size 0xfc00000000
        Address 0x10000000000, size 0x7ef00000
MEMLIST: phys_rsvd:
        Address 0x80000000, size 0x80000000
        Address 0xfd00000000, size 0x300000000
        Address 0x1007ef00000, size 0x11100000
MEMSEG addr=0x0 pgs=0x200 pfn 0x0-0x200
MEMSEG addr=0x600000 pgs=0x200 pfn 0x600-0x800
MEMSEG addr=0xa00000 pgs=0x200 pfn 0xa00-0xc00
MEMSEG addr=0xe00000 pgs=0x400 pfn 0xe00-0x1200
MEMSEG addr=0x1400000 pgs=0x10f pfn 0x1400-0x150f
MEMSEG addr=0x160f000 pgs=0x1 pfn 0x160f-0x1610
MEMSEG addr=0x1810000 pgs=0x1 pfn 0x1810-0x1811
MEMSEG addr=0x1911000 pgs=0x7e6ef pfn 0x1911-0x80000
MEMSEG addr=0x100000000 pgs=0xfc00000 pfn 0x100000-0xfd00000
MEMSEG addr=0x10000000000 pgs=0x7eeff pfn 0x10000000-0x1007eeff
../../oxide/os/startup.c:2245: 'availrmem_initial' is 0xfcfe0fd
../../oxide/os/startup.c:2246: 'availrmem' is 0xfcfe0fd
../../oxide/os/startup.c:2247: 'freemem' is 0xfcfe0fd
../../oxide/os/startup.c:1196: kphysm_init() done
../../oxide/os/startup.c:1197: 'npages' is 0xfcfe0ff
../../oxide/os/startup.c:1207: startup_memlist() done

@daym daym changed the title Enable more than 1 TiB RAM. Enable (more than) 1 TiB RAM. May 9, 2023
@daym daym requested a review from rmustacc May 9, 2023 21:01
@rmustacc
Copy link
Contributor

rmustacc commented May 9, 2023

What is the impact, if any, on the resulting memory channel configuration? Are we changing how we're interleaving as a result of changing these tokens? In particular, the main concern and I think the thing we need to understand before we move forward is:

  • What changes here to the memory interleaving rules, if anything?
  • If no changes, how certain are we that there is no performance impact. If there are changes, we need to quantify that impact on the memory subsystem.

We can get at some of this data through the zen_umc driver work that I did and the big theory statement has some background there.

@daym
Copy link
Contributor Author

daym commented May 26, 2023

Comparing the version before enabling 1 TiB to after enabling 1 TiB, we get no change in interleaving parameters on gimlet b9:

--- UMC_old5	2023-05-25 23:42:01.555873987 +0200
+++ UMC_mew4	2023-05-25 23:57:07.569732410 +0200
@@ -25,362 +25,761 @@
 	0x8078 apob-address = len=8 0000000004000000
 	0x8028 reset-vector = len=4 7ffefff0
 Loading kmdb...
-../../oxide/milan/milan_fabric.c:3001: milan_fabric_topo_init() starting...

-Socket 0 SMU Version: 45.93.0

-Socket 0 DXIO Version: 45.682

-Socket 0 SMU features 0x0690fbfd enabled

-cpu0: microcode has been updated from version 0x0 to 0xa0011ce

-../../oxide/os/startup.c:673: startup_init() starting...

-../../oxide/os/startup.c:695: startup_init() done

-../../oxide/os/startup.c:884: startup_memlist() starting...

-../../oxide/os/startup.c:905: 'modtext' is 0xfffffffffbe00000

-../../oxide/os/startup.c:906: 'e_modtext' is 0xfffffffffbe00000

-../../oxide/os/startup.c:907: 'moddata' is 0xfffffffffc4cc000

-../../oxide/os/startup.c:908: 'e_moddata' is 0xfffffffffc517000

-../../oxide/os/startup.c:909: 'econtig' is 0xfffffffffc517000

-MEMLIST: boot physinstalled:

-        Address 0x0, size 0x80000000

-        Address 0x100000000, size 0xfc00000000

-        Address 0x10000000000, size 0x7ef00000

-../../oxide/os/startup.c:922: 'physmax' is 0x1007eeff

-../../oxide/os/startup.c:923: 'physinstalled' is 0xfcfef00

-../../oxide/os/startup.c:924: 'memblocks' is 0x3

-MEMLIST: boot reserved mem:

-        Address 0x80000000, size 0x80000000

-        Address 0xfd00000000, size 0x300000000

-        Address 0x1007ef00000, size 0x11100000

-../../oxide/os/startup.c:935: 'rsvd_high_pfn' is 0x1008ffff

-../../oxide/os/startup.c:936: 'rsvd_pgcnt' is 0x391100

-../../oxide/os/startup.c:937: 'rsvdmemblocks' is 0x3

-../../oxide/os/startup.c:961: 'mmu.pt_nx' is 0x8000000000000000

-../../oxide/os/startup.c:978: 'npages' is 0xfcfe0ff

-../../oxide/os/startup.c:979: 'obp_pages' is 0x400

-../../oxide/os/startup.c:991: 'physmem' is 0xfcfe0ff

-../../oxide/os/startup.c:1005: 'memseg_sz' is 0x1000

-../../oxide/os/startup.c:1014: 'memlist_sz' is 0x1000

-../../oxide/os/startup.c:1022: 'rsvdmemlist_sz' is 0x1000

-../../oxide/os/startup.c:1035: 'pagehash_sz' is 0x20000000

-../../oxide/os/startup.c:1040: 'npages' is 0xfcfe0ff

-../../oxide/os/startup.c:1043: 'pp_sz' is 0x769718000

-../../oxide/os/startup.c:1053: 'pagecolor_memsz' is 0x1000

-../../oxide/os/startup.c:1057: 'page_ctrs_size' is 0x105000

-../../oxide/os/startup.c:1095: 'adjustment' is 0x603f7800000

-../../oxide/os/startup.c:1114: 'segkpm_base' is 0xfffff68000000000

-../../oxide/os/startup.c:1115: 'valloc_base' is 0xfffff78080000000

-../../oxide/os/startup.c:520: 'valloc_base' is 0xfffff78080000000

-../../oxide/os/startup.c:521: 'valloc_sz' is 0x789c00000

-MEMLIST: phys_install:

-        Address 0x0, size 0x80000000

-        Address 0x100000000, size 0xfc00000000

-        Address 0x10000000000, size 0x7ef00000

-../../oxide/os/startup.c:1136: Building phys_avail:

-

-        Filter: in: a=0, s=80000000

-        Filter out: a=0, s=200000

-        Filter: in: a=200000, s=7fe00000

-                trim: a=600000, s=7fa00000

-        Filter out: a=600000, s=200000

-        Filter: in: a=800000, s=7f800000

-                trim: a=a00000, s=7f600000

-        Filter out: a=a00000, s=200000

-        Filter: in: a=c00000, s=7f400000

-                trim: a=e00000, s=7f200000

-        Filter out: a=e00000, s=400000

-        Filter: in: a=1200000, s=7ee00000

-                trim: a=1400000, s=7ec00000

-        Filter out: a=1400000, s=10e000

-        Filter: in: a=150e000, s=7eaf2000

-                trim: a=160e000, s=7e9f2000

-        Filter out: a=160e000, s=1000

-        Filter: in: a=160f000, s=7e9f1000

-                trim: a=180f000, s=7e7f1000

-        Filter out: a=180f000, s=1000

-        Filter: in: a=1810000, s=7e7f0000

-                trim: a=1910000, s=7e6f0000

-        Filter out: a=1910000, s=7e6f0000

-        Filter: in: a=100000000, s=fc00000000

-        Filter out: a=100000000, s=fc00000000

-        Filter: in: a=10000000000, s=7ef00000

-        Filter out: a=10000000000, s=7ef00000

-MEMLIST: phys_avail:

-        Address 0x0, size 0x200000

-        Address 0x600000, size 0x200000

-        Address 0xa00000, size 0x200000

-        Address 0xe00000, size 0x400000

-        Address 0x1400000, size 0x10e000

-        Address 0x160e000, size 0x1000

-        Address 0x180f000, size 0x1000

-        Address 0x1910000, size 0x7e6f0000

-        Address 0x100000000, size 0xfc00000000

-        Address 0x10000000000, size 0x7ef00000

-MEMLIST: phys_rsvd:

-        Address 0x80000000, size 0x80000000

-        Address 0xfd00000000, size 0x300000000

-        Address 0x1007ef00000, size 0x11100000

-../../oxide/os/startup.c:1194: Calling kphysm_init()...

-MEMSEG addr=0x0 pgs=0x200 pfn 0x0-0x200

-MEMSEG addr=0x600000 pgs=0x200 pfn 0x600-0x800

-MEMSEG addr=0xa00000 pgs=0x200 pfn 0xa00-0xc00

-MEMSEG addr=0xe00000 pgs=0x400 pfn 0xe00-0x1200

-MEMSEG addr=0x1400000 pgs=0x10e pfn 0x1400-0x150e

-MEMSEG addr=0x160e000 pgs=0x1 pfn 0x160e-0x160f

-MEMSEG addr=0x180f000 pgs=0x1 pfn 0x180f-0x1810

-MEMSEG addr=0x1910000 pgs=0x7e6f0 pfn 0x1910-0x80000

-MEMSEG addr=0x100000000 pgs=0xfc00000 pfn 0x100000-0xfd00000

-MEMSEG addr=0x10000000000 pgs=0x7eeff pfn 0x10000000-0x1007eeff

-../../oxide/os/startup.c:2245: 'availrmem_initial' is 0xfcfe0fd

-../../oxide/os/startup.c:2246: 'availrmem' is 0xfcfe0fd

-../../oxide/os/startup.c:2247: 'freemem' is 0xfcfe0fd

-../../oxide/os/startup.c:1196: kphysm_init() done

-../../oxide/os/startup.c:1197: 'npages' is 0xfcfe0ff

-../../oxide/os/startup.c:1207: startup_memlist() done

-../../oxide/os/startup.c:1209: 'valloc_sz' is 0x789c00000

-../../oxide/os/startup.c:1227: startup_kmem() starting...

-../../oxide/os/startup.c:1236: 'core_base' is 0xffffffffc0000000

-../../oxide/os/startup.c:1237: 'core_size' is 0x3bbeb000

-../../oxide/os/startup.c:1238: 'kernelbase' is 0xfffff60000000000

-../../oxide/os/startup.c:1241: 'ekernelheap' is 0xffffffffc0000000

-../../oxide/os/startup.c:1258: '_kernelbase' is 0xfffff60000000000

-../../oxide/os/startup.c:1259: '_userlimit' is 0xfffff5ffffe00000

-../../oxide/os/startup.c:1260: '_userlimit32' is 0xfefff000

-../../oxide/os/startup.c:1526: layout_kernel_va() starting...

-../../oxide/os/startup.c:1532: 'kpm_size' is 0x1007f000000

-../../oxide/os/startup.c:1533: 'kpm_vbase' is 0xfffff68000000000

-../../oxide/os/startup.c:1551: 'segkp_base' is 0xfffff78809c00000

-../../oxide/os/startup.c:1552: 'segkpsize' is 0x80000

-../../oxide/os/startup.c:1567: 'segkvmmsize' is 0x3f3f8400

-../../oxide/os/startup.c:1568: 'segkvmm_base' is 0xfffff78889c00000

-../../oxide/os/startup.c:1583: 'segziosize' is 0x17b7d200

-../../oxide/os/startup.c:1584: 'segzio_base' is 0xfffffb7c82000000

-../../oxide/os/startup.c:1592: 'toxic_addr' is 0xfffffcf7ff200000

-../../oxide/os/startup.c:1602: 'segmap_start' is 0xfffffcf84f200000

-../../oxide/os/startup.c:1603: 'segmapsize' is 0x4000000

-../../oxide/os/startup.c:1605: 'kernelheap' is 0xfffffcf853200000

-../../oxide/os/startup.c:1606: layout_kernel_va() done...

-
Oxide Helios Version stlouis-0-gd2d480bbc4 64-bit (onu)

-DEBUG enabled

-../../oxide/os/startup.c:1311: startup_kmem() done

-../../oxide/os/startup.c:1620: startup_vm() starting...

-../../oxide/os/startup.c:1631: Calling hat_kern_alloc()...

-../../oxide/os/startup.c:1633: hat_kern_alloc() done

-../../oxide/os/startup.c:1647: Protecting boot pages

-../../oxide/os/startup.c:1514: 'boot_protect_cnt' is 0x0

-../../oxide/os/startup.c:1514: 'boot_protect_cnt' is 0x93bf

-../../oxide/os/startup.c:1667: Calling hat_kern_setup()...

-../../oxide/os/startup.c:1675: hat_kern_setup() done

-../../oxide/os/startup.c:1682: Calling kvm_init()...

-../../oxide/os/startup.c:2277: attaching kvseg_core

-../../oxide/os/startup.c:2283: attaching segkvmm

-../../oxide/os/startup.c:2289: attaching segzio

-../../oxide/os/startup.c:2306: protecting redzone

-../../oxide/os/startup.c:1684: kvm_init() done

-../../oxide/os/startup.c:1741: Calling hat_init_finish()...

-../../oxide/os/startup.c:1743: hat_init_finish() done

-../../oxide/os/startup.c:1749: Attaching segkp

-../../oxide/os/startup.c:1757: Doing segkp_create()

-../../oxide/os/startup.c:1762: 'segkp' is 0xfffffffffbe415a0

-../../oxide/os/startup.c:791: about to create segkpm

-../../oxide/os/startup.c:1780: 'segmap' is 0xfffffffffbe41600

-../../oxide/os/startup.c:1795: startup_vm() done

-../../oxide/os/startup.c:1813: startup_tsc() starting...

-../../oxide/os/timestamp.c:913: Calibrating the TSC...

-../../oxide/os/timestamp.c:988: 'tsccpp[i]->tscc_source' is 'PIT'

-../../oxide/os/startup.c:1816: 'tsc_freq' is 0x76fba659

-../../oxide/os/smm.c:191: 'tseg_pa' is 0x11c0000

-../../oxide/os/smm.c:192: 'smh_protop->smh_ksmmpa' is 0x1c17000

-../../oxide/os/startup.c:1322: startup_modules() starting...

-NOTICE: pciex dev 0x0, func 0x0

-NOTICE: pciex dev 0x0, func 0x1

-NOTICE: pciex dev 0x0, func 0x2

-NOTICE: pciex dev 0x0, func 0x3

-NOTICE: pciex dev 0x0, func 0x4

-NOTICE: pciex dev 0x0, func 0x5

-NOTICE: pciex dev 0x0, func 0x6

-NOTICE: pciex dev 0x0, func 0x7

-NOTICE: pciex dev 0x1, func 0x0

-NOTICE: pciex dev 0x1, func 0x1

-NOTICE: pciex dev 0x1, func 0x2

-NOTICE: pciex dev 0x1, func 0x3

-PCI-Express (0,1,3) capability found

-Found PCI-Ex in the system

-PCI-Express (0,1,3) capability found

-PCI-Express (0,3,2) capability found

-PCI-Express (0,3,3) capability found

-PCI-Express (0,3,4) capability found

-PCI-Express (0,7,1) capability found

-PCI-Express (0,8,1) capability found

-PCI-Express (0,8,2) capability found

-PCI-Express (0,8,3) capability found

-PCI-Express (1,0,0) capability found

-PCI-Express (2,0,0) capability found

-PCI-Express (3,0,0) capability found

-PCI-Express (4,0,0) capability found

-PCI-Express (6,0,0) capability found

-PCI-Express (7,0,0) capability found

-PCI-Express (8,0,0) capability found

-NOTICE: pciex dev 0x0, func 0x0

-NOTICE: pciex dev 0x0, func 0x1

-NOTICE: pciex dev 0x0, func 0x2

-NOTICE: pciex dev 0x0, func 0x3

-NOTICE: pciex dev 0x0, func 0x4

-NOTICE: pciex dev 0x0, func 0x5

-NOTICE: pciex dev 0x0, func 0x6

-NOTICE: pciex dev 0x0, func 0x7

-NOTICE: pciex dev 0x1, func 0x0

-NOTICE: pciex dev 0x1, func 0x1

-PCI-Express (40,1,1) capability found

-Found PCI-Ex in the system

-PCI-Express (40,1,1) capability found

-PCI-Express (40,1,2) capability found

-PCI-Express (40,1,3) capability found

-PCI-Express (40,3,3) capability found

-PCI-Express (40,7,1) capability found

-PCI-Express (40,8,1) capability found

-PCI-Express (40,8,2) capability found

-PCI-Express (40,8,3) capability found

-PCI-Express (41,0,0) capability found

-PCI-Express (42,0,0) capability found

-PCI-Express (43,0,0) capability found

-PCI-Express (44,0,0) capability found

-PCI-Express (45,0,0) capability found

-PCI-Express (46,0,0) capability found

-PCI-Express (47,0,0) capability found

-NOTICE: pciex dev 0x0, func 0x0

-NOTICE: pciex dev 0x0, func 0x1

-NOTICE: pciex dev 0x0, func 0x2

-NOTICE: pciex dev 0x0, func 0x3

-NOTICE: pciex dev 0x0, func 0x4

-NOTICE: pciex dev 0x0, func 0x5

-NOTICE: pciex dev 0x0, func 0x6

-NOTICE: pciex dev 0x0, func 0x7

-NOTICE: pciex dev 0x1, func 0x0

-NOTICE: pciex dev 0x1, func 0x1

-PCI-Express (80,1,1) capability found

-Found PCI-Ex in the system

-PCI-Express (80,1,1) capability found

-PCI-Express (80,7,1) capability found

-PCI-Express (80,8,1) capability found

-PCI-Express (80,8,2) capability found

-PCI-Express (80,8,3) capability found

-PCI-Express (81,0,0) capability found

-PCI-Express (82,0,0) capability found

-PCI-Express (83,0,0) capability found

-PCI-Express (84,0,0) capability found

-PCI-Express (85,0,0) capability found

-PCI-Express (85,0,1) capability found

-PCI-Express (85,0,2) capability found

-PCI-Express (85,0,3) capability found

-PCI-Express (85,0,4) capability found

-PCI-Express (85,0,5) capability found

-PCI-Express (85,0,6) capability found

-PCI-Express (85,0,7) capability found

-NOTICE: pciex dev 0x0, func 0x0

-NOTICE: pciex dev 0x0, func 0x1

-NOTICE: pciex dev 0x0, func 0x2

-NOTICE: pciex dev 0x0, func 0x3

-NOTICE: pciex dev 0x0, func 0x4

-NOTICE: pciex dev 0x0, func 0x5

-NOTICE: pciex dev 0x0, func 0x6

-NOTICE: pciex dev 0x0, func 0x7

-NOTICE: pciex dev 0x1, func 0x0

-NOTICE: pciex dev 0x1, func 0x1

-PCI-Express (c0,1,1) capability found

-Found PCI-Ex in the system

-PCI-Express (c0,1,1) capability found

-PCI-Express (c0,1,2) capability found

-PCI-Express (c0,1,3) capability found

-PCI-Express (c0,1,4) capability found

-PCI-Express (c0,3,2) capability found

-PCI-Express (c0,7,1) capability found

-PCI-Express (c0,8,1) capability found

-PCI-Express (c0,8,2) capability found

-PCI-Express (c0,8,3) capability found

-PCI-Express (c1,0,0) capability found

-PCI-Express (c2,0,0) capability found

-PCI-Express (c3,0,0) capability found

-PCI-Express (c4,0,0) capability found

-PCI-Express (c5,0,0) capability found

-PCI-Express (c6,0,0) capability found

-PCI-Express (c7,0,0) capability found

-PCI-Express (c8,0,0) capability found

-../../oxide/os/startup.c:1429: startup_modules: calling prom_setup...

-../../oxide/os/startup.c:1431: startup_modules: done

-../../oxide/os/startup.c:1436: startup_modules: calling psm_modload...

-../../oxide/os/startup.c:1439: startup_modules() done

-../../oxide/os/startup.c:1828: startup_end() starting...

-../../oxide/os/startup.c:1855: Calling configure()...

-NOTICE: Bus 0x00, bridges 0x8, buffer mem 0x1ebcde00

-NOTICE: Bus 0x40, bridges 0x8, buffer mem 0x1fdc6000

-NOTICE: Bus 0x80, bridges 0x5, buffer mem 0x1e7ce000

-NOTICE: Bus 0xc0, bridges 0x9, buffer mem 0x1ffae000

-../../oxide/os/startup.c:1857: configure() done

-../../oxide/os/startup.c:1863: xsave_setup_msr()

-../../oxide/os/startup.c:1872: cpu_intr_alloc()

-../../oxide/os/startup.c:1874: psm_install()

-../../oxide/os/mp_implfuncs.c:368: 'swp->psw_infop->p_mach_idstring' is 'apix'

-../../oxide/os/mp_implfuncs.c:371: psm_probe()

-../../oxide/io/apix/apic_common.c:333: apic_probe_common()

-../../oxide/io/apix/apic_common.c:343: apic_probe_raw()

-../../oxide/io/apix/apic_common.c:345: 'retval' is 0x0

-../../oxide/io/apix/apic_common.c:350: apic_ioapic_method_probe()

-../../oxide/io/apix/apic_common.c:352: SUCCESS

-../../oxide/os/mp_implfuncs.c:373: psm_probe() PSM_SUCCESS

-../../oxide/os/mp_implfuncs.c:396: psminitf()

-../../oxide/os/mp_machdep.c:1003: mach_construct_info()

-../../oxide/os/mp_machdep.c:1028: psm_softinit()

-../../oxide/os/mp_machdep.c:1044: 'idle_cpu_use_hlt' is 0x1

-../../oxide/os/mp_machdep.c:1077: mach_smpinit()

-../../oxide/os/startup.c:1881: NULLing out bootops

-../../oxide/os/startup.c:1887: Enabling interrupts

-../../oxide/os/startup.c:1891: Installing SMI handler

-../../oxide/os/startup.c:1909: startup_end() done

-NOTICE: Starting Oxide boot

-oxb=fffffd0211777a80

-NOTICE:     Phase 1 wants: baa5bd6432f8fd9a48803158e09c2c143cc1279dff5cf05706d0037963315a47

-NOTICE: TRYING: boot net

-attaching stuff...

-FCH peripheral: dwu@0, dwu0

-FCH peripheral: dwu@1, dwu1

-FCH peripheral: dwu@2, dwu2

-FCH peripheral: dwu@3, dwu3

-Ethernet interfaces:

-    igb0

-opening igb0 handle

-opening client handle

-NOTICE: MAC address is E8:EA:6A:09:6B:FD (igb0)

-NOTICE: listening for packets...

-NOTICE: hello...

-NOTICE: hello...

-NOTICE:     in image: baa5bd6432f8fd9a48803158e09c2c143cc1279dff5cf05706d0037963315a47

-NOTICE: received offer from 34:17:eb:d3:5c:51  -- size 4294967296 data size 838860800 dataset rpool/ROOT/ramdisk

-opening ramdisk control device

-creating ramdisk of size 4294967296

-opening ramdisk device: /devices/pseudo/ramdisk@1024:rpool

-

-
 receiving 0000000000000000 / 0000000032000000 (  0%)    

 receiving 0000000005160000 / 0000000032000000 ( 10%)    

 receiving 0000000009ce0000 / 0000000032000000 ( 19%)    

 receiving 000000000e2e0000 / 0000000032000000 ( 28%)    

 receiving 0000000013120000 / 0000000032000000 ( 38%)    

 receiving 0000000017b60000 / 0000000032000000 ( 47%)    

 receiving 000000001cc20000 / 0000000032000000 ( 57%)    

 receiving 0000000021720000 / 0000000032000000 ( 66%)    

 receiving 0000000026340000 / 0000000032000000 ( 76%)    

 receiving 000000002abe0000 / 0000000032000000 ( 85%)    

 receiving 000000002f760000 / 0000000032000000 ( 94%)    
NOTICE: reached EOF at offset 838860800 after 10 seconds           

-all done!

-closing unicast handle

-closing client handle

-freeing remaining messages

-closing handle

-ramdisk data size = 838860800

-checksum ok!

-NOTICE: vdev_disk_open("/dev/dsk/c3t1d0p0"): both DKIOCGMEDIAINFO{,EXT} calls failed, 25

-

-NOTICE: vdev_disk_open("/dev/dsk/c3t1d0p0"): both DKIOCGMEDIAINFO{,EXT} calls failed, 25

-

-strplumb: failed to initialize drv/ip

-../../oxide/os/startup.c:2024: Unmapping lower boot pages

-../../oxide/os/startup.c:2043: Releasing boot pages

-../../oxide/os/startup.c:2060: Boot pages released

+../../oxide/milan/milan_fabric.c:3001: milan_fabric_topo_init() starting...
+
+Socket 0 SMU Version: 45.93.0
+
+Socket 0 DXIO Version: 45.682
+
+Socket 0 SMU features 0x0690fbfd enabled
+
+cpu0: microcode has been updated from version 0x0 to 0xa0011ce
+
+../../oxide/os/startup.c:673: startup_init() starting...
+
+../../oxide/os/startup.c:695: startup_init() done
+
+../../oxide/os/startup.c:884: startup_memlist() starting...
+
+../../oxide/os/startup.c:905: 'modtext' is 0xfffffffffbe00000
+
+../../oxide/os/startup.c:906: 'e_modtext' is 0xfffffffffbe00000
+
+../../oxide/os/startup.c:907: 'moddata' is 0xfffffffffc4cc000
+
+../../oxide/os/startup.c:908: 'e_moddata' is 0xfffffffffc517000
+
+../../oxide/os/startup.c:909: 'econtig' is 0xfffffffffc517000
+
+MEMLIST: boot physinstalled:
+
+        Address 0x0, size 0x80000000
+
+        Address 0x100000000, size 0xfc00000000
+
+        Address 0x10000000000, size 0x7ef00000
+
+../../oxide/os/startup.c:922: 'physmax' is 0x1007eeff
+
+../../oxide/os/startup.c:923: 'physinstalled' is 0xfcfef00
+
+../../oxide/os/startup.c:924: 'memblocks' is 0x3
+
+MEMLIST: boot reserved mem:
+
+        Address 0x80000000, size 0x80000000
+
+        Address 0xfd00000000, size 0x300000000
+
+        Address 0x1007ef00000, size 0x11100000
+
+../../oxide/os/startup.c:935: 'rsvd_high_pfn' is 0x1008ffff
+
+../../oxide/os/startup.c:936: 'rsvd_pgcnt' is 0x391100
+
+../../oxide/os/startup.c:937: 'rsvdmemblocks' is 0x3
+
+../../oxide/os/startup.c:961: 'mmu.pt_nx' is 0x8000000000000000
+
+../../oxide/os/startup.c:978: 'npages' is 0xfcfe0ff
+
+../../oxide/os/startup.c:979: 'obp_pages' is 0x400
+
+../../oxide/os/startup.c:991: 'physmem' is 0xfcfe0ff
+
+../../oxide/os/startup.c:1005: 'memseg_sz' is 0x1000
+
+../../oxide/os/startup.c:1014: 'memlist_sz' is 0x1000
+
+../../oxide/os/startup.c:1022: 'rsvdmemlist_sz' is 0x1000
+
+../../oxide/os/startup.c:1035: 'pagehash_sz' is 0x20000000
+
+../../oxide/os/startup.c:1040: 'npages' is 0xfcfe0ff
+
+../../oxide/os/startup.c:1043: 'pp_sz' is 0x769718000
+
+../../oxide/os/startup.c:1053: 'pagecolor_memsz' is 0x1000
+
+../../oxide/os/startup.c:1057: 'page_ctrs_size' is 0x105000
+
+../../oxide/os/startup.c:1095: 'adjustment' is 0x603f7800000
+
+../../oxide/os/startup.c:1114: 'segkpm_base' is 0xfffff68000000000
+
+../../oxide/os/startup.c:1115: 'valloc_base' is 0xfffff78080000000
+
+../../oxide/os/startup.c:520: 'valloc_base' is 0xfffff78080000000
+
+../../oxide/os/startup.c:521: 'valloc_sz' is 0x789c00000
+
+MEMLIST: phys_install:
+
+        Address 0x0, size 0x80000000
+
+        Address 0x100000000, size 0xfc00000000
+
+        Address 0x10000000000, size 0x7ef00000
+
+../../oxide/os/startup.c:1136: Building phys_avail:
+
+
+
+        Filter: in: a=0, s=80000000
+
+        Filter out: a=0, s=200000
+
+        Filter: in: a=200000, s=7fe00000
+
+                trim: a=600000, s=7fa00000
+
+        Filter out: a=600000, s=200000
+
+        Filter: in: a=800000, s=7f800000
+
+                trim: a=a00000, s=7f600000
+
+        Filter out: a=a00000, s=200000
+
+        Filter: in: a=c00000, s=7f400000
+
+                trim: a=e00000, s=7f200000
+
+        Filter out: a=e00000, s=400000
+
+        Filter: in: a=1200000, s=7ee00000
+
+                trim: a=1400000, s=7ec00000
+
+        Filter out: a=1400000, s=10e000
+
+        Filter: in: a=150e000, s=7eaf2000
+
+                trim: a=160e000, s=7e9f2000
+
+        Filter out: a=160e000, s=1000
+
+        Filter: in: a=160f000, s=7e9f1000
+
+                trim: a=180f000, s=7e7f1000
+
+        Filter out: a=180f000, s=1000
+
+        Filter: in: a=1810000, s=7e7f0000
+
+                trim: a=1910000, s=7e6f0000
+
+        Filter out: a=1910000, s=7e6f0000
+
+        Filter: in: a=100000000, s=fc00000000
+
+        Filter out: a=100000000, s=fc00000000
+
+        Filter: in: a=10000000000, s=7ef00000
+
+        Filter out: a=10000000000, s=7ef00000
+
+MEMLIST: phys_avail:
+
+        Address 0x0, size 0x200000
+
+        Address 0x600000, size 0x200000
+
+        Address 0xa00000, size 0x200000
+
+        Address 0xe00000, size 0x400000
+
+        Address 0x1400000, size 0x10e000
+
+        Address 0x160e000, size 0x1000
+
+        Address 0x180f000, size 0x1000
+
+        Address 0x1910000, size 0x7e6f0000
+
+        Address 0x100000000, size 0xfc00000000
+
+        Address 0x10000000000, size 0x7ef00000
+
+MEMLIST: phys_rsvd:
+
+        Address 0x80000000, size 0x80000000
+
+        Address 0xfd00000000, size 0x300000000
+
+        Address 0x1007ef00000, size 0x11100000
+
+../../oxide/os/startup.c:1194: Calling kphysm_init()...
+
+MEMSEG addr=0x0 pgs=0x200 pfn 0x0-0x200
+
+MEMSEG addr=0x600000 pgs=0x200 pfn 0x600-0x800
+
+MEMSEG addr=0xa00000 pgs=0x200 pfn 0xa00-0xc00
+
+MEMSEG addr=0xe00000 pgs=0x400 pfn 0xe00-0x1200
+
+MEMSEG addr=0x1400000 pgs=0x10e pfn 0x1400-0x150e
+
+MEMSEG addr=0x160e000 pgs=0x1 pfn 0x160e-0x160f
+
+MEMSEG addr=0x180f000 pgs=0x1 pfn 0x180f-0x1810
+
+MEMSEG addr=0x1910000 pgs=0x7e6f0 pfn 0x1910-0x80000
+
+MEMSEG addr=0x100000000 pgs=0xfc00000 pfn 0x100000-0xfd00000
+
+\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00../../oxide/os/startup.c:2245: 'availrmem_initial' is 0xfcfe0fd
+
+../../oxide/os/startup.c:2246: 'availrmem' is 0xfcfe0fd
+
+../../oxide/os/startup.c:2247: 'freemem' is 0xfcfe0fd
+
+../../oxide/os/startup.c:1196: kphysm_init() done
+
+../../oxide/os/startup.c:1197: 'npages' is 0xfcfe0ff
+
+../../oxide/os/startup.c:1207: startup_memlist() done
+
+../../oxide/os/startup.c:1209: 'valloc_sz' is 0x789c00000
+
+../../oxide/os/startup.c:1227: startup_kmem() starting...
+
+../../oxide/os/startup.c:1236: 'core_base' is 0xffffffffc0000000
+
+../../oxide/os/startup.c:1237: 'core_size' is 0x3bbeb000
+
+../../oxide/os/startup.c:1238: 'kernelbase' is 0xfffff60000000000
+
+../../oxide/os/startup.c:1241: 'ekernelheap' is 0xffffffffc0000000
+
+../../oxide/os/startup.c:1258: '_kernelbase' is 0xfffff60000000000
+
+../../oxide/os/startup.c:1259: '_userlimit' is 0xfffff5ffffe00000
+
+../../oxide/os/startup.c:1260: '_userlimit32' is 0xfefff000
+
+../../oxide/os/startup.c:1526: layout_kernel_va() starting...
+
+../../oxide/os/startup.c:1532: 'kpm_size' is 0x1007f000000
+
+../../oxide/os/startup.c:1533: 'kpm_vbase' is 0xfffff68000000000
+
+../../oxide/os/startup.c:1551: 'segkp_base' is 0xfffff78809c00000
+
+../../oxide/os/startup.c:1552: 'segkpsize' is 0x80000
+
+../../oxide/os/startup.c:1567: 'segkvmmsize' is 0x3f3f8400
+
+../../oxide/os/startup.c:1568: 'segkvmm_base' is 0xfffff78889c00000
+
+../../oxide/os/startup.c:1583: 'segziosize' is 0x17b7d200
+
+../../oxide/os/startup.c:1584: 'segzio_base' is 0xfffffb7c82000000
+
+../../oxide/os/startup.c:1592: 'toxic_addr' is 0xfffffcf7ff200000
+
+../../oxide/os/startup.c:1602: 'segmap_start' is 0xfffffcf84f200000
+
+../../oxide/os/startup.c:1603: 'segmapsize' is 0x4000000
+
+../../oxide/os/startup.c:1605: 'kernelheap' is 0xfffffcf853200000
+
+../../oxide/os/startup.c:1606: layout_kernel_va() done...
+
+
+Oxide Helios Version stlouis-0-gd2d480bbc4 64-bit (onu)
+
+DEBUG enabled
+
+../../oxide/os/startup.c:1311: startup_kmem() done
+
+../../oxide/os/startup.c:1620: startup_vm() starting...
+
+../../oxide/os/startup.c:1631: Calling hat_kern_alloc()...
+
+../../oxide/os/startup.c:1633: hat_kern_alloc() done
+
+../../oxide/os/startup.c:1647: Protecting boot pages
+
+../../oxide/os/startup.c:1514: 'boot_protect_cnt' is 0x0
+
+../../oxide/os/startup.c:1514: 'boot_protect_cnt' is 0x93bf
+
+../../oxide/os/startup.c:1667: Calling hat_kern_setup()...
+
+../../oxide/os/startup.c:1675: hat_kern_setup() done
+
+../../oxide/os/startup.c:1682: Calling kvm_init()...
+
+../../oxide/os/startup.c:2277: attaching kvseg_core
+
+../../oxide/os/startup.c:2283: attaching segkvmm
+
+../../oxide/os/startup.c:2289: attaching segzio
+
+../../oxide/os/startup.c:2306: protecting redzone
+
+../../oxide/os/startup.c:1684: kvm_init() done
+
+../../oxide/os/startup.c:1741: Calling hat_init_finish()...
+
+../../oxide/os/startup.c:1743: hat_init_finish() done
+
+../../oxide/os/startup.c:1749: Attaching segkp
+
+../../oxide/os/startup.c:1757: Doing segkp_create()
+
+../../oxide/os/startup.c:1762: 'segkp' is 0xfffffffffbe415a0
+
+../../oxide/os/startup.c:791: about to create segkpm
+
+../../oxide/os/startup.c:1780: 'segmap' is 0xfffffffffbe41600
+
+../../oxide/os/startup.c:1795: startup_vm() done
+
+../../oxide/os/startup.c:1813: startup_tsc() starting...
+
+../../oxide/os/timestamp.c:913: Calibrating the TSC...
+
+../../oxide/os/timestamp.c:988: 'tsccpp[i]->tscc_source' is 'PIT'
+
+../../oxide/os/startup.c:1816: 'tsc_freq' is 0x76fb9920
+
+../../oxide/os/smm.c:191: 'tseg_pa' is 0x11c0000
+
+../../oxide/os/smm.c:192: 'smh_protop->smh_ksmmpa' is 0x1c17000
+
+../../oxide/os/startup.c:1322: startup_modules() starting...
+
+NOTICE: pciex dev 0x0, func 0x0
+
+NOTICE: pciex dev 0x0, func 0x1
+
+NOTICE: pciex dev 0x0, func 0x2
+
+NOTICE: pciex dev 0x0, func 0x3
+
+NOTICE: pciex dev 0x0, func 0x4
+
+NOTICE: pciex dev 0x0, func 0x5
+
+NOTICE: pciex dev 0x0, func 0x6
+
+NOTICE: pciex dev 0x0, func 0x7
+
+NOTICE: pciex dev 0x1, func 0x0
+
+NOTICE: pciex dev 0x1, func 0x1
+
+NOTICE: pciex dev 0x1, func 0x2
+
+NOTICE: pciex dev 0x1, func 0x3
+
+PCI-Express (0,1,3) capability found
+
+Found PCI-Ex in the system
+
+PCI-Express (0,1,3) capability found
+
+PCI-Express (0,3,2) capability found
+
+PCI-Express (0,3,3) capability found
+
+PCI-Express (0,3,4) capability found
+
+PCI-Express (0,7,1) capability found
+
+PCI-Express (0,8,1) capability found
+
+PCI-Express (0,8,2) capability found
+
+PCI-Express (0,8,3) capability found
+
+PCI-Express (1,0,0) capability found
+
+PCI-Express (2,0,0) capability found
+
+PCI-Express (3,0,0) capability found
+
+PCI-Express (4,0,0) capability found
+
+PCI-Express (6,0,0) capability found
+
+PCI-Express (7,0,0) capability found
+
+PCI-Express (8,0,0) capability found
+
+NOTICE: pciex dev 0x0, func 0x0
+
+NOTICE: pciex dev 0x0, func 0x1
+
+NOTICE: pciex dev 0x0, func 0x2
+
+NOTICE: pciex dev 0x0, func 0x3
+
+NOTICE: pciex dev 0x0, func 0x4
+
+NOTICE: pciex dev 0x0, func 0x5
+
+NOTICE: pciex dev 0x0, func 0x6
+
+NOTICE: pciex dev 0x0, func 0x7
+
+NOTICE: pciex dev 0x1, func 0x0
+
+NOTICE: pciex dev 0x1, func 0x1
+
+PCI-Express (40,1,1) capability found
+
+Found PCI-Ex in the system
+
+PCI-Express (40,1,1) capability found
+
+PCI-Express (40,1,2) capability found
+
+PCI-Express (40,1,3) capability found
+
+PCI-Express (40,3,3) capability found
+
+PCI-Express (40,7,1) capability found
+
+PCI-Express (40,8,1) capability found
+
+PCI-Express (40,8,2) capability found
+
+PCI-Express (40,8,3) capability found
+
+PCI-Express (41,0,0) capability found
+
+PCI-Express (42,0,0) capability found
+
+PCI-Express (43,0,0) capability found
+
+PCI-Express (44,0,0) capability found
+
+PCI-Express (45,0,0) capability found
+
+PCI-Express (46,0,0) capability found
+
+PCI-Express (47,0,0) capability found
+
+NOTICE: pciex dev 0x0, func 0x0
+
+NOTICE: pciex dev 0x0, func 0x1
+
+NOTICE: pciex dev 0x0, func 0x2
+
+NOTICE: pciex dev 0x0, func 0x3
+
+NOTICE: pciex dev 0x0, func 0x4
+
+NOTICE: pciex dev 0x0, func 0x5
+
+NOTICE: pciex dev 0x0, func 0x6
+
+NOTICE: pciex dev 0x0, func 0x7
+
+NOTICE: pciex dev 0x1, func 0x0
+
+NOTICE: pciex dev 0x1, func 0x1
+
+PCI-Express (80,1,1) capability found
+
+Found PCI-Ex in the system
+
+PCI-Express (80,1,1) capability found
+
+PCI-Express (80,7,1) capability found
+
+PCI-Express (80,8,1) capability found
+
+PCI-Express (80,8,2) capability found
+
+PCI-Express (80,8,3) capability found
+
+PCI-Express (81,0,0) capability found
+
+PCI-Express (82,0,0) capability found
+
+PCI-Express (83,0,0) capability found
+
+PCI-Express (84,0,0) capability found
+
+PCI-Express (85,0,0) capability found
+
+PCI-Express (85,0,1) capability found
+
+PCI-Express (85,0,2) capability found
+
+PCI-Express (85,0,3) capability found
+
+PCI-Express (85,0,4) capability found
+
+PCI-Express (85,0,5) capability found
+
+PCI-Express (85,0,6) capability found
+
+PCI-Express (85,0,7) capability found
+
+NOTICE: pciex dev 0x0, func 0x0
+
+NOTICE: pciex dev 0x0, func 0x1
+
+NOTICE: pciex dev 0x0, func 0x2
+
+NOTICE: pciex dev 0x0, func 0x3
+
+NOTICE: pciex dev 0x0, func 0x4
+
+NOTICE: pciex dev 0x0, func 0x5
+
+NOTICE: pciex dev 0x0, func 0x6
+
+NOTICE: pciex dev 0x0, func 0x7
+
+NOTICE: pciex dev 0x1, func 0x0
+
+NOTICE: pciex dev 0x1, func 0x1
+
+PCI-Express (c0,1,1) capability found
+
+Found PCI-Ex in the system
+
+PCI-Express (c0,1,1) capability found
+
+PCI-Express (c0,1,2) capability found
+
+PCI-Express (c0,1,3) capability found
+
+PCI-Express (c0,1,4) capability found
+
+PCI-Express (c0,3,2) capability found
+
+PCI-Express (c0,7,1) capability found
+
+PCI-Express (c0,8,1) capability found
+
+PCI-Express (c0,8,2) capability found
+
+PCI-Express (c0,8,3) capability found
+
+PCI-Express (c1,0,0) capability found
+
+PCI-Express (c2,0,0) capability found
+
+PCI-Express (c3,0,0) capability found
+
+PCI-Express (c4,0,0) capability found
+
+PCI-Express (c5,0,0) capability found
+
+PCI-Express (c6,0,0) capability found
+
+PCI-Express (c7,0,0) capability found
+
+PCI-Express (c8,0,0) capability found
+
+../../oxide/os/startup.c:1429: startup_modules: calling prom_setup...
+
+../../oxide/os/startup.c:1431: startup_modules: done
+
+../../oxide/os/startup.c:1436: startup_modules: calling psm_modload...
+
+../../oxide/os/startup.c:1439: startup_modules() done
+
+../../oxide/os/startup.c:1828: startup_end() starting...
+
+../../oxide/os/startup.c:1855: Calling configure()...
+
+NOTICE: Bus 0x00, bridges 0x8, buffer mem 0x1ebcde00
+
+NOTICE: Bus 0x40, bridges 0x8, buffer mem 0x1fdc6000
+
+NOTICE: Bus 0x80, bridges 0x5, buffer mem 0x1e7ce000
+
+NOTICE: Bus 0xc0, bridges 0x9, buffer mem 0x1ffae000
+
+../../oxide/os/startup.c:1857: configure() done
+
+../../oxide/os/startup.c:1863: xsave_setup_msr()
+
+../../oxide/os/startup.c:1872: cpu_intr_alloc()
+
+../../oxide/os/startup.c:1874: psm_install()
+
+../../oxide/os/mp_implfuncs.c:368: 'swp->psw_infop->p_mach_idstring' is 'apix'
+
+../../oxide/os/mp_implfuncs.c:371: psm_probe()
+
+../../oxide/io/apix/apic_common.c:333: apic_probe_common()
+
+../../oxide/io/apix/apic_common.c:343: apic_probe_raw()
+
+../../oxide/io/apix/apic_common.c:345: 'retval' is 0x0
+
+../../oxide/io/apix/apic_common.c:350: apic_ioapic_method_probe()
+
+../../oxide/io/apix/apic_common.c:352: SUCCESS
+
+../../oxide/os/mp_implfuncs.c:373: psm_probe() PSM_SUCCESS
+
+../../oxide/os/mp_implfuncs.c:396: psminitf()
+
+../../oxide/os/mp_machdep.c:1003: mach_construct_info()
+
+../../oxide/os/mp_machdep.c:1028: psm_softinit()
+
+../../oxide/os/mp_machdep.c:1044: 'idle_cpu_use_hlt' is 0x1
+
+../../oxide/os/mp_machdep.c:1077: mach_smpinit()
+
+../../oxide/os/startup.c:1881: NULLing out bootops
+
+../../oxide/os/startup.c:1887: Enabling interrupts
+
+../../oxide/os/startup.c:1891: Installing SMI handler
+
+../../oxide/os/startup.c:1909: startup_end() done
+
+NOTICE: Starting Oxide boot
+
+oxb=fffffd0211777a80
+
+NOTICE:     Phase 1 wants: 9fc5d2b8a69e9d5c25322fe86a0eb4fc585493589cfb05f3ec01c07ecf98c2ae
+
+NOTICE: TRYING: boot net
+
+attaching stuff...
+
+FCH peripheral: dwu@0, dwu0
+
+FCH peripheral: dwu@1, dwu1
+
+FCH peripheral: dwu@2, dwu2
+
+FCH peripheral: dwu@3, dwu3
+
+Ethernet interfaces:
+
+    igb0
+
+opening igb0 handle
+
+opening client handle
+
+NOTICE: MAC address is E8:EA:6A:09:6B:FD (igb0)
+
+NOTICE: listening for packets...
+
+NOTICE: hello...
+
+NOTICE: hello...
+
+NOTICE: hello...
+
+NOTICE: hello...
+
+NOTICE: hello...
+
+NOTICE: hello...
+
+NOTICE: hello...
+
+NOTICE: hello...
+
+NOTICE: hello...
+
+NOTICE: hello...
+
+NOTICE: hello...
+
+NOTICE: hello...
+
+NOTICE:     in image: 9fc5d2b8a69e9d5c25322fe86a0eb4fc585493589cfb05f3ec01c07ecf98c2ae
+
+NOTICE: received offer from 34:17:eb:d3:5c:51  -- size 4294967296 data size 838860800 dataset rpool/ROOT/ramdisk
+
+opening ramdisk control device
+
+creating ramdisk of size 4294967296
+
+opening ramdisk device: /devices/pseudo/ramdisk@1024:rpool
+
+
+
+
+ receiving 0000000000000000 / 0000000032000000 (  0%)    
+
+ receiving 0000000004d80000 / 0000000032000000 (  9%)    
+
+ receiving 0000000009a00000 / 0000000032000000 ( 19%)    
+
+ receiving 000000000e2c0000 / 0000000032000000 ( 28%)    
+
+ receiving 0000000013120000 / 0000000032000000 ( 38%)    
+
+ receiving 0000000017700000 / 0000000032000000 ( 46%)    
+
+ receiving 000000001c780000 / 0000000032000000 ( 56%)    
+
+ receiving 0000000020ea0000 / 0000000032000000 ( 65%)    
+
+ receiving 0000000025920000 / 0000000032000000 ( 75%)    
+
+ receiving 000000002a080000 / 0000000032000000 ( 84%)    
+
+ receiving 000000002e820000 / 0000000032000000 ( 93%)    
+NOTICE: reached EOF at offset 838860800 after 10 seconds           
+
+all done!
+
+closing unicast handle
+
+closing client handle
+
+freeing remaining messages
+
+closing handle
+
+ramdisk data size = 838860800
+
+checksum ok!
+
+\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00v/dsk/c3t1d0p0"): both DKIOCGMEDIAINFO{,EXT} calls failed, 25
+
+
+
+NOTICE: vdev_disk_open("/dev/dsk/c3t1d0p0"): both DKIOCGMEDIAINFO{,EXT} calls failed, 25
+
+
+
+strplumb: failed to initialize drv/ip
+
+../../oxide/os/startup.c:2024: Unmapping lower boot pages
+
+../../oxide/os/startup.c:2043: Releasing boot pages
+
+../../oxide/os/startup.c:2060: Boot pages released
+
 Configuring devices.
 WARNING: ipcc0: Could not determine uart path
 NOTICE: vdev_disk_open("/dev/ramdisk/rpool"): both DKIOCGMEDIAINFO{,EXT} calls failed, 25
 
 Hostname: unknown
-

-unknown console login: root
-Dec 28 00:00:13 unknown login: ROOT LOGIN /dev/console

+
+
+unknown console login: Dec 28 00:00:14 unknown fch: FCH peripheral: dwu@1, dwu1
+
+Dec 28 00:00:14 unknown fch: FCH peripheral: dwu@2, dwu2
+
+Dec 28 00:00:14 unknown fch: FCH peripheral: dwu@3, dwu3
+
+Dec 28 00:00:26 unknown svc.startd[100003]: site/compliance/dump:default failed: transitioned to maintenance (see 'svcs -xv' for details)
+
+root
+Dec 28 00:04:28 unknown login: ROOT LOGIN /dev/console
+
 
     �[32;42m�[48;5;78m�[38;5;78m#####�[0m
    �[32;42m�[48;5;78m�[38;5;78m##�[0m   �[32;42m�[48;5;78m�[38;5;78m##�[0m
@@ -390,18 +789,13 @@
    �[32;42m�[48;5;78m�[38;5;78m##�[0m   �[32;42m�[48;5;78m�[38;5;78m##�[0m    �[32;42m�[48;5;78m�[38;5;78m##�[0m �[32;42m�[48;5;78m�[38;5;78m##�[0m    
     �[32;42m�[48;5;78m�[38;5;78m#####�[0m    �[32;42m�[48;5;78m�[38;5;78m##�[0m   �[32;42m�[48;5;78m�[38;5;78m##�[0m    Gimlet
 
-�[01;31munknown �[00m�[1m#�[00m Dec 28 00:00:14 unknown fch: FCH peripheral: dwu@1, dwu1

-Dec 28 00:00:14 unknown fch: FCH peripheral: dwu@2, dwu2

-Dec 28 00:00:14 unknown fch: FCH peripheral: dwu@3, dwu3

-modload /kernel/drv/amd64/zen_umc
-�[01;31munknown �[00m�[1m#�[00m �[KmdbDec 28 00:00:26 unknown svc.startd[100003]: site/compliance/dump:default failed: transitioned to maintenance (see 'svcs -xv' for details)

- -k
-�[?40s�[?40h�[?12l�[?25h�[?7hLoading modules: [ unix genunix specfs dtrace mac cpu.generic apix cpc crypto mm random smbios zfs sata ip hook neti sockfs lofs scsi_vhci arp ufs logindmux nsmb ptm ]
+�[01;31munknown �[00m�[1m#�[00m modload /kernel/drv/amd64/zen_umc
+�[01;31munknown �[00m�[1m#�[00m �[K
 > *zen_umc::print zen_umc_t
 {
     umc_tom = 0x80000000
     umc_tom2 = 0x10080000000
-    umc_dip = 0xfffffd03f59fbba0
+    umc_dip = 0xfffffd03f5a76ba0
     umc_family = 0t39 (X86_PF_AMD_MILAN)
     umc_df_rev = 0x2 (DF_REV_3)
     umc_fdata = zen_umc_fam_data+0xa0
@@ -421,6 +815,7 @@
             zud_flags = 0xd (ZEN_UMC_DF_F_H{OLE_VALID|ASH_21_23|ASH_30_32})
             zud_dfno = 0
             zud_ccm_inst = 0x10
+            zud_dram_nrules = 0x10
             zud_nchan = 0x8
             zud_cs_nremap = 0x2
             zud_hole_raw = 0x80000001
@@ -31024,4 +31419,10 @@
     umc_decoder_buf = 0
     umc_decoder_len = 0
 }
-> 
+> 
+�[?40r�(B�[m�[01;31munknown �[00m�[1m#�[00m prtconf -m
+1036271
+�[01;31munknown �[00m�[1m#�[00m logout
+
+
+unknown console login: 

@rmustacc
Copy link
Contributor

rmustacc commented May 26, 2023

So in that output, I don't see any change in npages or the memory segments that we have present. Given the comment in the host-software channel that prtconf -m returned 1036271 and all the memory pieces lined up, it appears we didn't actually get the 12 GiB back that this was trying to get at.

I expect that we'll need to go investigate changing the interleaving to do so.

@wesolows
Copy link
Contributor

The change that's necessary to make this effective is not directly changing the interleaving settings. The critical guidance from AMD here is this tidbit from pub 55483 rev 1.70, 4.1.2.9 APCB_TOKAN_UID_DF_REMAP_AT_1TB:

If the region that crosses the problematic boundary is based at system address 0x0000, remapping is not possible.

This goes on to mention the example of interleaving set to 'die' where each socket (sic) has 1TB of DRAM, in which the remapping won't work. Let's examine then the tokens we might expect to be of interest. The first is APCB_TOKEN_UID_DF_MEM_INTERLEAVING which is documented to accept the values socket, die, channel, and none as well as two different automatic modes distinguished by their behaviour on different processor families. Tellingly, the only one of these modes that is documented at all for Rome is Auto - [F17M30] Interleaving is based upon the NUMA topology per NUMA Node Per Socket (NPSx). See reference Socket SP3 NUMA Topology; none of these modes at all is documented for Milan. I have tried some of these settings in the past and found them ineffective (in fact, my notes say that the changes I made to the APCB invariably failed to boot). In any case, for Milan, only channel would be useful: there is always a 1-1 mapping of die to socket, and our platform always has exactly one socket, so those two modes are equivalent both to each other and to the default behaviour we see, which is always:

BASE             LIMIT            FLAGS   INTERLEAVE      DESTINATION
0                0x807fffffff     VH-     8/COD-1 8/1/1   0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0

Let's start with the only setting of this token that can possibly be of use: channel. It's relatively inexpensive to run these tests; I built a normal working phase1 image and then simply repeated the same command used by the helios builder while making small modifications to the APCB JSON input. Each image was then differentially written to flash and booted with kmdb enabled. While I do not have a 1 TiB Gimlet, it's very easy to tell whether a configuration change has altered the interleaving in a manner that would satisfy the documented restriction and allow recovery of the 12 GiB in question; the only difference is that each channel has 2x the total RAM in the larger configuration. We don't need to boot to phase2 at all but can simply use ::df_route -d in kmdb to tell us whether we've identified a configuration that will work. With that in mind, the use of channel interleave mode results in this:

BASE             LIMIT            FLAGS   INTERLEAVE      DESTINATION
0                0x807fffffff     VH-     8/COD-1 8/1/1   0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0

The astute reader will notice that this is exactly the same interleaving configuration we get by default. We shouldn't really be altogether surprised, because the documentation for this parameter tells us it's valid on [F17M00][F17M10][F17M20] which is to say Naples and a pair of other very old and uninteresting processor families (Raven Ridge and Dali).

If this obvious setting doesn't do anything, what might? Let's consider instead a pair of tokens APCB_TOKEN_UID_DF_DRAM_INTERLEAVE_SIZE and APCB_TOKEN_UID_DF_MEM_INTERLEAVING_SIZE. Yes, they are different. In fact they accept essentially the same set of possible values and have the same semantics, except that 4 KiB is accepted only for the second. It's not clear why AMD felt the need to introduce the second token, but our documentation does tell us that the first token is valid only on Naples/RV/RV2X while the second is valid on Rome and Milan. The possible meanings of these options can be derived from the discussion at the top of https://github.com/oxidecomputer/illumos-gate/blob/master/usr/src/uts/intel/io/amdzen/zen_umc.c in part because we can't quote here at any length from the AMD document but moreso because it doesn't really tell us anything anyway. Our options for this are taken from https://github.com/oxidecomputer/amd-apcb/blob/6b1a0677ce187336a02f1b9311ee1c9031310e9b/src/ondisk.rs#L6558. Why is this interesting? A note in the AMD document suggests that the values 1KiB and larger cause "channel interleaving hash" to be disabled; i.e., it causes the automatic selection of interleaving mode to change. Does that help us? Sort of, but not in the way we care about. With this set to 1Ki, we end up with:

BASE             LIMIT            FLAGS   INTERLEAVE      DESTINATION
0                0x807fffffff     VH-     11/8/1/1        0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0

While we can probably get the ABL to choose slightly different bits for interleaving this way (and that might indeed be useful if we want to explore the overall configuration space while observing performance changes), this isn't going to get us what we really need which is for the rule containing the IOMMU hole to begin at some address other than 0.

The key observation here is that from Naples to Rome, the essential control knob for this changed from these tokens to the "NUMA" configuration, Nodes per Socket or APCB_TOKEN_UID_DF_DRAM_NPS. Again, while we can't quote from the AMD doc, it doesn't really tell us that this is so, only hints at it. Instead, consult the (public!) AMD pub. 56795 rev 1.00, especially table 1. In the configurations of interest, every quadrant (indeed, every channel) always has the exact same population, which simplifies interpretation of this information. The critical piece of information is that this NPS setting has real effects; one might easily imagine that on PCs it merely dictates the construction of ACPI tables (SLIT and SRAT, mainly), but in fact it causes pre-boot interleaving configuration to be altered fundamentally, which is described in some detail in the document although not directly as the concept of DF routing is never mentioned at all and instead most of the document discusses uninteresting PC implementation details.

Armed with this essential conclusion, we can set DfDramNumaPerSocket to something other than Auto. For our purposes, None (or AMD's NPS0) is the same as One since this is a 1S machine, and the default of Auto will result in the same configuration for all supported DIMM populations. That leaves us with NPS2 (Two) and NPS4 (Four). At this point we can actually predict what we're going to see, and sure enough it's pretty obvious. With Two:

BASE             LIMIT            FLAGS   INTERLEAVE      DESTINATION
0                0x407fffffff     VH-     8/COD-2 4/1/1   0 (0/0) -- UMC0
0x4080000000     0x807fffffff     V--     8/COD-2 4/1/1   0x4 (0/0x4) -- UMC4
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0

and with Four:

BASE             LIMIT            FLAGS   INTERLEAVE      DESTINATION
0                0x207fffffff     VH-     8/COD-4 2/1/1   0 (0/0) -- UMC0
0x2080000000     0x407fffffff     V--     8/COD-4 2/1/1   0x2 (0/0x2) -- UMC2
0x4080000000     0x607fffffff     V--     8/COD-4 2/1/1   0x4 (0/0x4) -- UMC4
0x6080000000     0x807fffffff     V--     8/COD-4 2/1/1   0x6 (0/0x6) -- UMC6
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0
0                0xfffffff        ---     8/1/1/1         0 (0/0) -- UMC0

With the full 1 TiB population, each of these rules would be twice as large but the basic organisation will remain the same.

In other words, setting this parameter to Two or Four should enable remapping of the DRAM from the IOMMU hole, opening up an additional 12 GiB of RAM on fully populated Gimlets that's otherwise completely inaccessible.

That leaves us with two fundamental questions, one much more important than the other:

  1. Although these use the same hashing mechanism (unless other parameters are changed) as the default configuration, they do somewhat restrict the number of channels across which a linear access pattern will be hashed. Does this materially degrade performance?
  2. Is any of the guidance in AMD pub 56795 meaningful at all in software terms? The impression I took from this is that the ABL's use of the NPS value to configure interleaving is a matter of enabling the binding of CCXs, NBIOs, and DRAM regions but does not by itself influence the actual cost of a particular access, which is still dependent entirely on whether the access crosses the xGMI interface between sockets (and perhaps to a lesser extent the internal structure of a given processor's dies within a socket). In other words, this makes it easier for software that wants to bind these resources together to do so but does not make it obligatory; that's important, because we don't have any NUMA support at all on this platform (yet). Our second question thus can be distilled to: is there some benefit, beyond recovering the 12 GiB, to implementing some kernel software here in order to take advantage of/minimise the cost of configuring interleaving for NPS2 or NPS4?

Obviously, if the answer to the first part is no, then the second question is largely moot on this 1S platform. In any event, we now know what is possible in terms of interleaving configuration so at least we can measure what is measurable. It would seem that the obvious path forward here, with lowest impact, is to configure NPS2 and leave the mode selection at the default (i.e., COD-4). In the limit it may be advantageous to go to NPS4 and take advantage of our knowledge of this to improve locality for VMs that fit in a quadrant, though the effect (if any) on globally-bound PCIe devices doing DMA access may not be conducive to great results.

Copy link
Contributor

@wesolows wesolows left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my notes on interleaving; in order to make this work on Milan, one must set DfDramNumaPerSocket to Two, or else identify some convincing alternative.

Since this change includes the fixes for additional bugs, please include them in the synopsis.

Then there's the big part: how will you test performance of the alternate interleaving configuration in a way that will be representative of how the machine is being used in the field? This would mean that a pattern of linear accesses to a physical address range would be interleaved over 4 channels rather than 8. How common is that access pattern? Does the kmem allocator end up mostly papering over this anyway? If not, are guest performance or other specific tasks measurably slowed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants