FEATURES

Map of processed files.

The size of the processed files and their name is saved on a map. This map is written to disk. When the flume agent is started, it is loaded. Before processing a file, the flume source consults this map to find out if the number of lines in the file has grown. If it has done so, the file is marked as modified and therefore the line difference is processed. If it has not changed, the file is simply not processed but will be moved to the processed directory. This movement is susceptible to generate an exception if the file already existed in a directory of processed, simply preventing movement.