More compact format

[splitice]

Currently JSON is used for serialization. This leads to alot of additional space to represent an object. A more compact format might perform better through reducing the number of blocks that need to be written (and possibly encrypted) on disk.

SHORT LINE (amd64):

real    0m14.598s
user    0m0.000s
sys     0m5.584s

LONG LINE (amd64):

real    0m17.016s
user    0m0.000s
sys     0m5.706s

These tests were performed under considerable existing load (consistent between tests)

Code:

#include <stdio.h>
#include <unistd.h>

#define LINE "through or flushing a disk cache if present. The call blocks until the device reports that the transfer has test line\n"

void main(){
        FILE* fp = fopen("write-test.bin", "w");
        for(int i=0;i<10000;i++){
                fwrite(LINE, 1, sizeof(LINE), fp);
                fdatasync(fileno(fp));
        }
        fclose(fp);
}

Indicates that ~15% could be gained by reducing the LINE by 70% (best case)

[splitice]

Armv7 shows even more savings from a reduction in disk storage.

LONG LINE (armv7):

real    0m4.056s
user    0m0.024s
sys     0m0.383s

SHORT LINE (armv7):

real    0m0.864s
user    0m0.001s
sys     0m0.114s

Difference is even more pronounced here!

This is likely because the CPU cost of encrypting the data is of significantly higher cost. The disk is also considerably slower under high rate transfer so further savings expected.