Why the value address pass to pmemkv_get_v_callback function is not on pmem?(Tree3 engine)

[yijieZ]

QUESTION:

In tree3::get function, using pmem_is_pmem() to test the value address but returns 0.

Details

Four real optane pm devices are interleaving and added to OS in 'fsdax' mode.
I mount the pm device with ext4-dax filesystem.

mkfs.ext4 /dev/pmem0
mount -o dax /dev/pmem0 /mnt/pmem0

The PMDK version is 1.9.1.

I add pmem_is_pmem() to the tree3::get function in pmemkv/src/engines-experimental/tree3.c,
and found that pmem_is_pmem() returns 0.

status tree3::get(string_view key, get_v_callback *callback, void *arg)
{
	LOG("get using callback for key=" << std::string(key.data(), key.size()));
	check_outside_tx();
	// XXX - do not create temporary string
	auto leafnode = LeafSearch(std::string(key.data(), key.size()));
	if (leafnode) {
		const uint8_t hash = PearsonHash(key.data(), key.size());
		for (int slot = LEAF_KEYS; slot--;) {
			if (leafnode->hashes[slot] == hash) {
				LOG("   found hash match, slot=" << slot);
				if (leafnode->keys[slot].compare(
					    std::string(key.data(), key.size())) == 0) {
					auto kv = leafnode->leaf->slots[slot].get_ro();

                                        printf("tree3 get is_pmem=%d\n", pmem_is_pmem(kv.val(), kv.valsize()));

					LOG("   found value, slot="
					    << slot
					    << ", size=" << std::to_string(kv.valsize()));
					callback(kv.val(), kv.valsize(), arg);
					return status::OK;
				}
			}
		}
	}
	LOG("   could not find key");
	return status::NOT_FOUND;
}

The leafnode->leaf->slots[slot] is allocated in tree3::put function by calling make_persistent<internal::tree3::KVLeaf>()
which means it is allocated from pm.
It is confused that the pmem_is_pmem() return 0.

The explain of pmemkv_get function in pmemkv manpage says that the
Value points to the location where data is actually stored (no copy occurs).
Does it means kv.val() is a address to pm?

[igchor]

From the manpage of pmem_is_pmem: "Calling this function with a memory range that originates from a source different than pmem_map_file() is undefined."

Pmemkv uses pmemobj to manage a persistent memory pool, and pmemobj does not use pmem_map_file but some dedicated function. I don't think there is any function like pmem_is_pmem for pmemobj. But if you'd share some more details on what you are trying to do we might come up with some suggestions.

[yijieZ]

Sorry for my question, I should read doc and code before ask.
I insert KV pairs(1,000,000 and value size=128B) to tree3 and search them with hotness distribute(like workloadc in ycsb).
I find the data size read from PMs(by using pcm-memory tool) is far more less than I requested(10,000,000 get).
There must be a buffer for hot data and I am looking for it.
But I think is not pmemkv, sorry for my question again.

[igchor]

No problem :)

Are you actually using ycsb in your experiments? No sure if you know but we have made integration with it (not yet upstreamed): https://github.com/pmem/ycsb

[yijieZ]

I use YCSB to generate trace file which provides operation type and key
and modify pmemkv/examples/pmemkv_basic_c/pmemkv_basic.c to load the trace file.
It would be great to have pmemkv in YCSB!

[lukaszstolarczuk]

If you want, you can bump our PR on their repo:
brianfrankcooper/YCSB#1545

perhaps it will speed up the wait there :-)

[lukaszstolarczuk]

I hope it helped. In case of more questions don't hesitate to ask. Closing this one for now.