Format/mount CXL devices

[christian-pinto]

Hi,

I have managed to run QEMU using this script in combination with:

• Linux: https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git - tag cxl-for-5.16
• QEMU: https://gitlab.com/bwidawsk/qemu.git - branch cxl-2.0v4

QEMU runs fine, I can see the cxl devices in /dev/cxl/.

ndctl reports the below

[root@localhost ndctl]# ndctl list -BDiu -b CXL
{
  "provider":"CXL",
  "dev":"ndbus0",
  "dimms":[
    {
      "dev":"nmem1"
    },
    {
      "dev":"nmem0"
    }
  ]
}

After that I have not been able to format (e.g., with xfs) and mount/use the devices. Any suggestions on how to make actual use of those devices?

Thanks

[hishine6]

I'm trying to do the same thing, but keep on getting error messages from the script.. I couldn't make the qemu start running at all. could you please provide some information about the mkosi version you have used? Or the host OS that you have used? I am trying on Ubuntu 18.04(mkosi version 3+17) which makes alot of errors. Thank you in advance.

[sscargal]

@christian-pinto We're waiting for the Region driver & feature to become available. Once we have regions, we can then create namespaces which we can then format. There are active discussions on the mailing list - CXL Region Driver

[christian-pinto]

Thanks @sscargal. Also, any chance these CXL devices can be used as "normal" non volatile memory? I see the QEMU device presents capacity only for non-volatile case. Looks like the initial code for QEMU CXL devices was kind of supporting volatile memory but it then disappeared.

I'd be interested in using a CXL device as a regular memory, hotplug them, etc.

Christian

[bgoglin]

Hello, is there any updates on this? I tried playing with this with 5.19rc and Dan's patches for CXL/PMEM. What I am missing is how to actually configure the volatile/pmem partitioning of CXL memory devices. Are we supposed to configure that before booting Qemu? In the "LSA"? Or using cxl command-lines in Linux inside Qemu?

[stellarhopper]

@bgoglin the 'set-partition' command of cxl-cli is nominally used to configure persistent capacity vs. volatile. However I don't think the current qemu implementation actually supports that. It (qemu) provides both persistent and volatile capacities on all the memdevs. Are you asking about the command line bits to actually create a region using Dan's create-region patches? I have an equivalent patchset for cxl-cli that will add a cxl create-region command, that should be posted this week.

[bgoglin]

Yes, I'd like the command-line bits to make a CXL pmem region appear as pmem in Linux with Dan's patches (and unrelated to this issue, bits to make a CXL volatile mem region appear as a NUMA node in Linux). IIRC CXL mem is pmem by default, I indeed tried set-partition inside qemu and it failed as you said. Is your cxl create-region for volatile, pmem or both?

[stellarhopper]

@bgoglin currently cxl create-region only has pmem, but eventually it will grow support for volatile as well.

[jack-chen1688]

Using run_qemu.sh, cxl-for-6.0 and latest qemu + two minor patches related to get_lsa and set_lsa, I created an region using sysfs with configuration of interleave ways = 2, and then create namespace with fsdax mod. After this, I tried dd command and also mkfs.ext4. It seems that dd command works fine without data corruption. But formating the namespace using mkfs.ext4 and mount it will fail. This is very strange. Could anyone provide some advice on what could be wrong? Thanks.

[Yemaoxin]

@stellarhopper Now，I see the newest cxl-cli in ndctl can create-region -t ram. My question is How can I use this ram region？ndctl is only work for PMEM.

[bgoglin]

@stellarhopper Now，I see the newest cxl-cli in ndctl can create-region -t ram. My question is How can I use this ram region？ndctl is only work for PMEM.

Once it's created, Linux will create a DAX device on top of it (will appear in daxctl list, as /dev/dax... and in /sys/bus/dax/devices/). DAX devices are special files that you may mmap directly, or can be configured as new NUMA nodes than application may use to allocate memory. It depends on the Linux kernel and system configuration, you may use daxctl to switch between modes (eg daxctl reconfigure-device --mode=system-ram dax0.0) and to online/offline memory in a NUMA node (daxctl online-memory dax0.0).

[Yemaoxin]

@bgoglin Thank，I fixed this problem . Thanks for your help, I found out that it was because the DAX support was not turned on in the Kernel before. I have solved it by myself and 2 NUMA nodes appeared successfully. Your explanation gave me a clearer understanding of CXL's region management, thanks!

[Yemaoxin]

@bgoglin Sorry for bother you again. When I run numactl --cpubind=0 --membind=1 memcached -m 30000 -p 11212 -t 8 -c 1024, membind=1 is the CXL Memory,but I got Illegal instruction. How can I do?

[bgoglin]

No idea, I never actually used CXL memory, even in Qemu. I only played with configuration things.

[Yemaoxin]

Thanks @bgoglin