[TsFile] What is new from v0.7.0 to Kill_Thanos

In version v0.7.0, TsFile uses Thrift to help serializing metadata. Tsfile's format is as following:

In a TsFile, a ChunkGroup only stores data that belongs to one device. And a Chunk only stores data that belongs to one measurement.

Level 1 and Level 2 index (called as metadata in the source code) form a Thrift Object, called as TsFileMetadata. PageHeader is another Thrift Object.

Level 1 index in TsFileMetadata records the positions of each ChunkGroup, and the corresponding positions in Level 2 index. Level 2 index records the positions of each Chunk.

In many cases, TsFileMetadata can be very large. Suppose that device 1 and device 2 writing data concurrently. Then a TsFile may looks like:

In this way, there may be many chunk groups (8 groups in the figure), and therefore the TsFileMetadata records 8 positions for the 8 ChunkGroups.

We need to remain the TsFileMetadata in memory all the time for serving queries. To reduce the memory cost, only loading Level 1 index into memory is optimal (e.g., Apache Cassandra has the similar behavior). Unfortunately, we have to load Level 1 and Level 2 in one breath because they belong to one TsFileMetadata.

Therefore, the first problem is that too much memory is used in this way. (We can fix the problem by split TsFileMetadta into two objects.)

Secondly, because Page Header is serialized by Thrift, while others parts are not, it is hard to know the file format in bytes clearly. It may be harmful if we want to fix a broken file manually in product environment (for saving data).

Thirdly, only Page has a header structure, and the structure stores summary information in the header, all other summary information of ChunkGroups and Chunks is stored in TsFileMetadata, which is asymmetric.

Fourth, in a TsFileMedata, there are two kinds of information: (1) data definition, e.g., the data type, the encoding method, the compression method, the measurement name, and the device name; (2) summary information, e.g., the min timestamp, the max timestamp, the min data point and the max data point value. Splitting them helps source code more clear.

Besides, TsFileMetadata is flushed on disk in the end. If the program is stopped before the TsMetadata is flushed, then the whole file is unable to be read. However, if we split data definition and summary, and flush the data definition along with the data, we can rescue much data when the program is stopped before a TsMetadta is flushed.

Fifth, in v0.7.0, there are codes for converting a Thrift object to a JavaBean object. Developers may be confusing about that.

Considering all of them, a new tsfile format is introduced: (1) Ability to restore a broken file. TsFileMetadata is only used for indexing data. All the data definitions are along with data. (2) Better code organization. All the data structures have a header or a footer. Besides, there are many other code refactoring, which makes the project structure more clear. (3) Embrace Java NIO.

Now the new TsFile format looks like:

Notice that, we have to add a marker to distinguish what kind of structure the next part of data is: 0 is chunk group footer, 1 is chunk header and 2 means no data left.

For more information, you can run tsfile/example/src/main/java/cn/edu/tsinghua/tsfile/TsFileSequenceRead. The class helps to print the file format: e.g.,

file length: 1070
file magic head: TsFilev0.8.0
file magic tail: TsFilev0.8.0
Level 1 metadata position: 899
Level 1 metadata size: 899
[Chunk Group]
position: 12
	[Chunk]
	position: 13
	Measurement: sensor_3
		[Page]
 		Page head position: 47
		Page data position: 99
		points in the page: 5
		Uncompressed page data size: 45
			time, value: 2, 50
			time, value: 4, 51
			time, value: 6, 11
			time, value: 7, 21
			time, value: 8, 31
	[Chunk]
	position: 145
	Measurement: sensor_1
		[Page]
 		Page head position: 179
		Page data position: 231
		points in the page: 6
		Uncompressed page data size: 41
			time, value: 1, 1.2
			time, value: 3, 1.4
			time, value: 4, 1.2
			time, value: 6, 7.2
			time, value: 7, 6.2
			time, value: 8, 9.2
	[Chunk]
	position: 273
	Measurement: sensor_2
		[Page]
 		Page head position: 307
		Page data position: 359
		points in the page: 7
		Uncompressed page data size: 46
			time, value: 1, 20
			time, value: 2, 20
			time, value: 3, 21
			time, value: 4, 20
			time, value: 6, 10
			time, value: 7, 20
			time, value: 8, 30
Chunk Group Footer position: 406
device: device_1
[Metadata]
	[Device]File Offset: 431, Device device_1, Number of Chunk Groups 1
		Measurement:sensor_3
		File offset:12
		Measurement:sensor_1
		File offset:144
		Measurement:sensor_2
		File offset:272