Unknown Attestation in OTS proof

[RebekkalPangras]

Hi, one of the timestamps made on 2022-08-19 has verify Unknown Attestation in it's info. Timestamping was done using the multistamp method with a total of 23 document hashes

The proof : 004f70656e54696d657374616d7073000050726f6f6600bf89e2e884e892940108a0733aadc1dbe15d74410ecace34b8f6decc357c5c99825b23421ae458d9a94af010ca7d36ebc0fe1498eecb71047080dba808f1203bad873856885bae567a2458ccb7caabfe909e0a64d7f522f48006672468954108f12078ba28a25bb603ce9e047c4f9ce339b944ff2b84a8015fe27525c483636894b008f02032b27027e360ae98e1145071d9c9916c99dfad7fc3538c9c2acd43aa4af5490508f020440bb1c7f15a276d68710fabf3ad0492f872789bcf17ab92e8395990f9a7d08c08f020d79a5bc4c75e5e8661d9e70ac18b7f51f2463accee92966e0cb010e682e016a20800

Document Hash : a0733aadc1dbe15d74410ecace34b8f6decc357c5c99825b23421ae458d9a94a

[petertodd]

Weird. Looks like you have a corrupted proof. Running ots info on that proof gives the error message: Tried to read 8 bytes but got only 0 bytes

Is this error repeatable?

[RebekkalPangras]

Hi @petertodd , we have multistamped more than 25,000 hashes so far using Opentimestamps and this is the first time we got such an error. Have you seen this error before? If so, kindly let us know if there is any fix for this.

[petertodd]

On Tue, Oct 18, 2022 at 03:38:33AM -0700, Rebekkal wrote: Hi @petertodd , we have multistamped more than 25,000 hashes so far using Opentimestamps and this is the first time we got such an error. Have you seen this error before? If so, kindly let us know if there is any fix for this.

Sorry, I've never seen this error before. It's quite possible there was some kind of hardware or other error that resulted in the timestamp being truncated. Since it was truncated, there's probably no way to recover that particular timestamp.

[RebekkalPangras]

Thank you for your response.

[petertodd]

If this happens again, please let us know!

…

On October 27, 2022 12:25:20 PM GMT+02:00, Rebekkal ***@***.***> wrote: Thank you for your response. -- Reply to this email directly or view it on GitHub: #58 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[RebekkalPangras]

Hi @petertodd, the above issue seems to have reoccurred. I apologize for any inconvenience this may cause and would greatly appreciate any assistance you can provide in resolving this issue.

Following are the dates in which it has happened.

2022-11-25
2022-11-27
2022-12-16
2022-12-21
2022-12-22
2023-01-01
2023-01-02
2023-01-10
2023-01-11
2023-01-18

An example ots-proof received with Unknown attestation.

Ots Proof : 004f70656e54696d657374616d7073000050726f6f6600bf89e2e884e892940108006875e61d982e8541e73c9c0ea913ea473dcaae812934e0e54ea27ecbe4338ff010d8aafae63599379d0bda403859b6c3f508f020446f05f572070fe0334b03a8750392f31c3e682495e4e22decae23292e0213d408f120fc11847023ad3bb0344586243bda3bd16eb470bddbae72a9a0ada0e0b6597fb608f120dff1e4bcb08bcfbd107dda9f346de735a54a73a3a8a4f9e2cde0e760efb31afb08f0203c1bc4af0dbf1587873f55f1538f4e79520104e4eb40db738012dfe27c8fd08a0800

Document Hash : 006875e61d982e8541e73c9c0ea913ea473dcaae812934e0e54ea27ecbe4338f

[petertodd]

Can you post some of the code used to make timestamps on your end? I have another thing that makes tens of thousands of timestamps regularly with python-opentimestamps and it hasn't seen a similar issue. So I suspect the issue has something to do with your specific setup. Something that could help here is to try to deserialize every timestamp you receive, and retry the timestamp if deserialization fails.

…

On January 23, 2023 6:53:48 AM EST, Rebekkal ***@***.***> wrote: Reopened #58. -- Reply to this email directly or view it on GitHub: #58 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[RebekkalPangras]

Thank you for your quick response, @petertodd

public static List<OtsTimestamp> stampHashes(List<DocHash> pendingHashData) throws AppException {
        List<OtsTimestamp> otsTimestamps = new ArrayList<OtsTimestamp>();
        HashMap<String, String> privateUrls = new HashMap<>();
        List<String> calendarsUrl = new ArrayList<>();
        HashMap<DocHash, DetachedTimestampFile> mapFiles = new HashMap<>();
        Integer m = 0;

        // convert hash to DetachedTimestampFile
        for (DocHash pendingHash : pendingHashData) {
            try {
                String hashValue = pendingHash.getDocumentHash();
                byte[] shasum = Utils.hexToBytes(hashValue);
                mapFiles.put(pendingHash, DetachedTimestampFile.from(new Hash(shasum, OpSHA256._TAG)));
            } catch (Exception e) {
                LOGGER.log(Level.SEVERE, "Crypto Error", e);
            }
        }

        // Stamping
        Timestamp stampResult;
        String merkleRoot = "";

        try {
            List<DetachedTimestampFile> detaches = new ArrayList<DetachedTimestampFile>(mapFiles.values());
            stampResult = OpenTimestamps.stamp(detaches, calendarsUrl, m, privateUrls);
            merkleRoot = Utils.bytesToHex(stampResult.msg);
        } catch (IOException e) {
            LOGGER.log(Level.SEVERE, "Stamp Error", e);
        }

        // Generate ots proofs
        for (Map.Entry<DocHash, DetachedTimestampFile> entry : mapFiles.entrySet()) {
            DocHash timestampData = entry.getKey();
            DetachedTimestampFile detached = entry.getValue();
            try {
                // convert the bytes to hex
                byte[] otsBytes = detached.serialize();

                LOGGER.info("CHECKING FOR INVALID ATTESTATION IN THE RECEIVED PROOF");
                DetachedTimestampFile.deserialize(otsBytes); // Temp fix for the attestation issue
                LOGGER.info("NO INVALID ATTESTATION FOUND");

                String otsHex = Utils.bytesToHex(otsBytes);
                java.sql.Timestamp submittedTime = new java.sql.Timestamp(timestampData.getSubmittedTime());

                otsTimestamps.add(new OtsTimestamp(timestampData, otsHex, submittedTime));
            } catch (NullPointerException npe) {
                LOGGER.log(Level.INFO, npe.getMessage());
                throw npe;
            } catch (Exception e) {
                LOGGER.log(Level.SEVERE, e.getLocalizedMessage(), e);
            }
        }
        return otsTimestamps;
    }

[petertodd]

So I decoded your invalid timestamp by hand:

004f70656e54696d657374616d7073000050726f6f6600bf89e2e884e8929401 # header

08 # File hash type: OpSHA256
006875e61d982e8541e73c9c0ea913ea473dcaae812934e0e54ea27ecbe4338f # file digest

f0 # OpAppend
10 # 16 bytes
d8aafae63599379d0bda403859b6c3f5

08 # OpSHA256

f0 # OpAppend
20 # 32 bytes
446f05f572070fe0334b03a8750392f31c3e682495e4e22decae23292e0213d4

08 # OpSHA256

f1 # OpPrepend
20 # 32 bytes
fc11847023ad3bb0344586243bda3bd16eb470bddbae72a9a0ada0e0b6597fb6

08 # OpSha256

f1 # OpPrepend
20 # 32 bytes
dff1e4bcb08bcfbd107dda9f346de735a54a73a3a8a4f9e2cde0e760efb31afb

08 # OpSHA256
f0 # OpAppend
20 # 32 bytes
3c1bc4af0dbf1587873f55f1538f4e79520104e4eb40db738012dfe27c8fd08a

08 # OpSha256

00 # Attestation

# End-of-file, when there should be an attestation!

I'm not familiar with the java-opentimestamps code base. But I suspect the problem has something to do with a lack of attestations.

@RCasatta It seems that the codebase allows a timestamp to have zero attestations:

java-opentimestamps/src/main/java/com/eternitywall/ots/Timestamp.java

Line 118 in 8186391

if (sortedAttestations.size() > 1) {

This is incorrect: a valid timestamp must have at least one attestation at all times.

So maybe the issue is that the OpenTimestamps.stamp function isn't raising an exception properly when the calendar fails?

@RebekkalPangras I assume you're only timestamping with a single calendar right?

[RebekkalPangras]

Hi @petertodd , Yes timestamping was done with single calendar before. Now I have changed it to two.

[petertodd]

On February 2, 2023 5:21:17 AM GMT+01:00, Rebekkal ***@***.***> wrote: Hi @petertodd , Yes timestamping was done with single calendar before. Now I have changed it to two.

Are you saving calendar data locally, eg by upgrading the timestamps when the transactions confirm?

[RebekkalPangras]

Yes, I save the ots proof locally.

[petertodd]

Yes, I save the ots proof locally.

Good! So my suggestion to you for now would be to continue timestamping with multiple calendars, and use the above code to verify it.

Long term we should fix java-opentimestamps to deal with errors better to avoid others making this mistake. I don't know any Java though, so hopefully one of the people who wrote this library can help. You're also welcome to come up with a pull-req. :)

[RebekkalPangras]

Thank you for your suggestion @petertodd . As for fixing the java-opentimestamps library, I will look into contributing a pull request or reaching out to the developers for assistance.

[petertodd]

On Mon, Feb 06, 2023 at 09:25:36PM -0800, Rebekkal wrote: Thank you for your suggestion @petertodd . As for fixing the java-opentimestamps library, I will look into contributing a pull request or reaching out to the developers for assistance.

Thanks!