post warning "found unrecognized file"

[countvonzero]

Description

when node restart, post module warn about the following files unexpected but they are. cc @poszu

{"L":"WARN","T":"2023-08-08T17:24:07.287Z","N":"6e5ab.post","M":"found unrecognized file","node_id":"6e5ab097e4e0e91afb04058743a409af53072b2c74148045248881ba818ea210","module":"post","fileName":"post.bin"}
{"L":"WARN","T":"2023-08-08T17:24:07.287Z","N":"6e5ab.post","M":"found unrecognized file","node_id":"6e5ab097e4e0e91afb04058743a409af53072b2c74148045248881ba818ea210","module":"post","fileName":"nipost_challenge.bin"}
{"L":"WARN","T":"2023-08-08T17:24:07.287Z","N":"6e5ab.post","M":"found unrecognized file","node_id":"6e5ab097e4e0e91afb04058743a409af53072b2c74148045248881ba818ea210","module":"post","fileName":"nipost_builder_state.bin"}
{"L":"WARN","T":"2023-08-08T17:24:07.287Z","N":"6e5ab.post","M":"found unrecognized file","node_id":"6e5ab097e4e0e91afb04058743a409af53072b2c74148045248881ba818ea210","module":"post","fileName":"key.bin"} | ```

[poszu]

@countvonzero, this log message is produced by post that is not expecting any extra files in the POS data directory (containing postdata_N.bin and postdata_metadata.json files). IMHO, it's wrong that we store state files and keys in the same directory and a proper solution is to move these files elsewhere. What do you think?

[countvonzero]

i recall storing key in post dir was a deliberate decision so that it ties all things identity in one directory.
also it make checkpoint restore easier (simply preserving everything in that directory).
what do you think if we push the POST data one directory down? @poszu

[poszu]

what do you think if we push the POST data one directory down

👍 SGTM

[fasmat]

what do you think if we push the POST data one directory down

I think this is only a symptomatic fix for a problem that needs a better solution.

[countvonzero]

what do you think if we push the POST data one directory down

I think this is only a symptomatic fix for a problem that needs a better solution.

can you state the problem?

[fasmat]

I think we should consider moving eveything that is not postdata_N.bin or postdata_metadata.bin out of the data directory. If a user makes this directory read only the node should still be able to smesh without issues. We have users that did exactly this and I consider that a valid solution to accidentally deleting files (by the user or the node).

The key.bin file is not required for PoST. Information about the identity that the data belongs to is already in postdata_metadata.json. Additionally having it in the PoST data dir lead to 2 implementations of generating / parsing an identity (one in the node and one in PoST). PoST doesn't need to know about identity handling, it only needs the NodeID to generate the data for, so in my opinion it should be moved out of the directory and maybe to the same place where the node stores the SQL database.

Some of the other files should in my opinion also be moved out of the directory - nipost_builder_state.bin does not belong into the PoST directory, as it holds information on the state the current ATX building process - which I think only concerns the node but not PoST. I could make similar arguments for nipost_challenge.bin.

post.bin arguably belongs to PoST since it holds a generated proof. Here it would be nice if in the future when we introduce pausing and resuming of post generation this file is replaced with more detailed information about the progress proof generation (challenge used as input, last checked index & nonce, found indices below threshold, etc.). The file itself should allow PoST to continue generating a proof if not finished yet or verify the completeness of one without external information.

[countvonzero]

do you agree tho the life cycle of key.bin and post data should be the same? one should not be able to exist without the other?

i will be onboard with separating node's own state files from the post data and key.bin.
tho that is not what this issue is about. please file a separate issue if making post dir read-only is important in your view.

[fasmat]

do you agree tho the life cycle of key.bin and post data should be the same?

I'm not sure. PoST doesn't care about an identity. It only needs a commitment which can be anything.

Correct me if I'm wrong but a node can have an identity without smeshing (and therefore having PoST data) I think. A node can receive and send transactions but doesn't participate in the consensus (i.e. a light node)?

[fasmat]

As a quick fix I would suggest changing the logging level from "WARN" to "DEBUG" or even removing the logs about unrecognised files completely. Since they are just misleading to someone not familiar with the internals of PoST and do not indicate any actual problem.

[countvonzero]

but a node can have an identity without smeshing

we need to separate node identity and smesher identity. currently the code doesn't do a good job of that. ideally those are separate public keys. in the future, a node could be running the consensus protocol for multiple identities even, as that's the most demanding part of a full node.

The key.bin file is not required for PoST. Information about the identity that the data belongs to is already in postdata_metadata.json.

are you saying that if the public key in postdata_metadata.json doesn't match key.bin, a miner can still use this post data to create atxs and ballots?

[fasmat]

are you saying that if the public key in postdata_metadata.json doesn't match key.bin, a miner can still use this post data to create atxs and ballots?

PoST doesn't even read the file key.bin and only cares about what is inside postdata_metadata.json:

The code is quite convoluted at the moment: The node has to read key.bin to get the private key and from that the NodeID and then pass it to PoST for initialization and proof generation. PoST then checks if the passed NodeID matches the one in the postdata_metadata.json and returns an error if it doesn't.

This design issue will be addressed by: spacemeshos/post#116

[countvonzero]

@fasmat and i had an offline conversation. we agreed that short term pushing the data to a subdir or just changing to dbg log are fine.

long term:
in the context of a node, it is important for identity related data (post, atx build state, key.bin) to stay in one top directory for the purpose checkpoint recovery. but the post data should/need not be accompanied by key.bin outside the context of the mining node. e.g in the context of initialization service, it doesn't need a key.bin file.

[Stizerg]

If you move key.bin outside of POS directory there will be more mistakes made by people moving POS files between nodes. Currently it's easy to move just one folder and to point the node to that folder.

[fasmat]

@Stizerg please be aware: you cannot re-use the same identity (key.bin & POS files) to smesh with two different nodes. You will not get twice the rewards and you risk your identity being flagged as malicious.

EDIT: In case I misunderstood you and you only initialize on one node and use the data on another: that's not an issue 🙂

[Stizerg]

@fasmat Yes I know, that's why I said ".. moving POS files .."
Obviously moving identity as well if it's together with POS files.

[countvonzero]

from an user's log on windows. note the ".jso" files

2023-08-11T19:59:43.314+0300    INFO    e2090.post      initialization started  {"node_id": "e20905c6e85cd32f7a35e3a915c3e99aa6f19e978cf43b9e40e0efda9e9c26f4", "module": "post", "datadir": "Z:\\L", "numUnits": 29, "maxFileSize": 2147483648, "labelsPerUnit": 4294967296}
2023-08-11T19:59:43.314+0300    INFO    e2090.post      initialization file layout      {"node_id": "e20905c6e85cd32f7a35e3a915c3e99aa6f19e978cf43b9e40e0efda9e9c26f4", "module": "post", "labelsPerFile": 134217728, "labelsLastFile": 134217728, "firstFileIndex": 0, "lastFileIndex": 927}
2023-08-11T19:59:43.326+0300    WARN    e2090.post      found unrecognized file {"node_id": "e20905c6e85cd32f7a35e3a915c3e99aa6f19e978cf43b9e40e0efda9e9c26f4", "module": "post", "fileName": "$RECYCLE.BIN"}
2023-08-11T19:59:43.326+0300    WARN    e2090.post      found unrecognized file {"node_id": "e20905c6e85cd32f7a35e3a915c3e99aa6f19e978cf43b9e40e0efda9e9c26f4", "module": "post", "fileName": "1"}
2023-08-11T19:59:43.326+0300    WARN    e2090.post      found unrecognized file {"node_id": "e20905c6e85cd32f7a35e3a915c3e99aa6f19e978cf43b9e40e0efda9e9c26f4", "module": "post", "fileName": "System Volume Information"}
2023-08-11T19:59:43.326+0300    WARN    e2090.post      found unrecognized file {"node_id": "e20905c6e85cd32f7a35e3a915c3e99aa6f19e978cf43b9e40e0efda9e9c26f4", "module": "post", "fileName": "key.bin"}
2023-08-11T19:59:43.326+0300    WARN    e2090.post      found unrecognized file {"node_id": "e20905c6e85cd32f7a35e3a915c3e99aa6f19e978cf43b9e40e0efda9e9c26f4", "module": "post", "fileName": "nipost_builder_state.bin"}
2023-08-11T19:59:43.326+0300    WARN    e2090.post      found unrecognized file {"node_id": "e20905c6e85cd32f7a35e3a915c3e99aa6f19e978cf43b9e40e0efda9e9c26f4", "module": "post", "fileName": "nipost_challenge.bin"}
2023-08-11T19:59:43.326+0300    WARN    e2090.post      found unrecognized file {"node_id": "e20905c6e85cd32f7a35e3a915c3e99aa6f19e978cf43b9e40e0efda9e9c26f4", "module": "post", "fileName": "post.bin"}
2023-08-11T19:59:43.337+0300    WARN    e2090.post      found unrecognized file {"node_id": "e20905c6e85cd32f7a35e3a915c3e99aa6f19e978cf43b9e40e0efda9e9c26f4", "module": "post", "fileName": "postdata_metadata.jso"}
2023-08-11T19:59:43.337+0300    WARN    e2090.post      found unrecognized file {"node_id": "e20905c6e85cd32f7a35e3a915c3e99aa6f19e978cf43b9e40e0efda9e9c26f4", "module": "post", "fileName": "smeshing_metadata.jso"}
2023-08-11T19:59:43.337+0300    WARN    e2090.post      found unrecognized file {"node_id": "e20905c6e85cd32f7a35e3a915c3e99aa6f19e978cf43b9e40e0efda9e9c26f4", "module": "post", "fileName": "smeshing_metadata.json"}
2023-08-11T19:59:43.337+0300    WARN    e2090.post      found unrecognized file {"node_id": "e20905c6e85cd32f7a35e3a915c3e99aa6f19e978cf43b9e40e0efda9e9c26f4", "module": "post", "fileName": "tmp_read_write.xml"}

[fasmat]

Here are my thoughts: Z:\ is either a removable device or the files on it have been copied using a removable device from another computer. Removable devices compatible with Windows can have any of multiple different file systems: FAT32, exFAT, NTFS, etc.

Some of those filesystems (specifically FAT32) do not support file-extensions with more than 3 letters natively (files are stored as 8.3 and metadata keeps track of their "real" filename). This would explain where the postdata_metadata.jso file comes from -> during the copy from the removable device the file was copied with its "real" name (.json) and the name it is stored in the FS (.jso). Since there is no log about a missing postdata_metadata.json that file is also present in the folder.

key.bin, nipost_builder_state.bin, nipost_challenge.bin and post.bin are all created by our node in the process of creating an ATX.

$RECYCLE.BIN, and System Volume Information are folders created by Windows and contain the recycle bin for and information about the removable device (that's why I believe Z: is a removable device). They should not be in the post folder (here Z:\L it seems) but rather in the root of the removable device and have probably been copied accidentally.

I don't know what smeshing_metadata.json is for - that file is not created by our node or postcli. The same goes for 1 and tmp_read_write.xml.

[fasmat]

spacemeshos/post#215 reduced the logging level from WARN to DEBUG and is expected to be included in the next release of post / go-spacemesh

[fasmat]

The log from the windows user also makes me believe the right way to go is to just ignore any file PoST doesn't know instead of logging them. There might always be files in the POS folder that are put there by the OS (or OS tools) that we shouldn't complain about:

• desktop.ini metadata about the folder, created by Windows explorer when User changes how files are displayed in Explorer (list, icons, etc.) or if the Folder is assigned a custom icon.
• .DS_Store same file as desktop.ini but for MacOS, contains meta information about the folder and is created by Finder.
• $recycle.bin, System Volume Information folders that are in the root of a device if used by Windows (independent of file system)
• autorun.inf another file that can be in the root of a removable device and is managed by Windows
• ***.chk files are created by windows when Scandisk checks a disk for errors
• I believe rsync creates a hidden folder with meta information when it is used to copy files between computers.
• possibly more... depending on OS, file system and tools a user might use to manage their data...

I believe the right way to "deal" with those is to just ignore anything that is not postdata_metadata.json and postdata_*.bin, not log any other files/folders that might be present and operate as if they weren't there.