FilePtr Linked lists -> vec or post-process?

[hacklint]

Hi!
I'm playing with the idea of an binrw based MDF reader (MDF as the measurement format, not the database one).
The standard is sadly not open, but essentially it is much based on trees and linked lists of blocks. A block can hold data and then "first links" to start element of linked lists, the link being the absolute file position. Blocks in the list have a "next block" pointer to the next element in the list. Each block may also have links to other "first blocks" of other lists. The last element in a list is identified by its "next" pointer having value 0. There are no cycles. Using 16-bit pointers for small test data, real ones are 64 bit.
I'm fairly new to rust and new to binrw but the use of file positions as link address seems to call out for FilePtr. To avoid a recursive infinite-size type I use Box for the linked-to blocks. As far as I can see there is no special handling of 0 in FilePtr, but I have solved that by using a free function parser.
However, there are some things I struggle with (minimal example below, thoughts refer to that)
Ideally I would like to read linked lists of blocks into Vec. Now I get them as Option<Box> which gives a "lopsided" type where the Option for the next element is part of the "current" block. Is there a reasonable way to get that "directly" into a Vec? I see no way to pass state (eg a Vec to fill) between calls to the parsing function and since there may be other lists read between reads of consecutive elements that may not be an issue... But then the next question is if there is some "supported" way to post-process parsed data (eg "unroll" the partially nested blocks into a Vec? Eg some "hook" after reading a complete struct? Final thought is if there is a way to only parse parts of the tree (ie wait with some lists to a "second parsing stage" which is a central idea in MDF - read only what you need.
This probably turned into a mix of binrw and Rust issues - hope it made sense, code below.

use std::fmt::Debug;
use binrw::{binread, BinRead, BinReaderExt, BinResult, FilePtr16};

#[binrw::parser(reader, endian)]
fn linked_elements<'a, T :Debug + BinRead<Args<'a> = ()>>() -> BinResult<Option<Box<T>>> {
    if reader.stream_position()? != 0 {  // Zero signals "no more elements in list"
        println!("At pos: {:?}, read {}"
            ,reader.stream_position().unwrap()
            ,std::any::type_name::<T>());
        let element = <_>::read_options(reader, endian, ())?;
        println!(" Got element: {:?}", element);
        return Ok(Some(Box::new(element)));
    } else {
        Ok(None)
    }
}

#[derive(Debug)]
#[binread]
struct Meta {  // The metadata struct - this one is the linked list
    meta: u16,
    #[br(parse_with = FilePtr16::parse_with(linked_elements))]
    meta_next: Option<Box<Meta>>  // No recursion, using Box
    // Possible more fields, including other linked lists...
}

#[derive(Debug)]
#[binread]
struct Root {  // Top level struct - MAY point to a Meta
    root_data: u16,
    #[br(parse_with = FilePtr16::parse_with(linked_elements))]
    meta_first: Option<Box<Meta>>  // Points to first meta element (at pos 8)
    // What I _really_ want here: meta: Vec<Meta>;  // Holding all the elements of the Meta linked list, 2 in example.
    // Possible more fields, including other linked lists
}

fn main() {
    let mut filedata = binrw::io::Cursor::new(b"\x00\x07\x00\x08\x00\x7f\x00\x00\x00\x0e\x00\x04");

    let root:Root = filedata.read_be().unwrap();
    assert!(root.root_data == 7);
    println!("struct root: {:?}", root);
}

[kitlith]

Since you're already using a custom parser, I'd probably hoist the recursive data structure into a loop handled by the custom parser. I'll provide an example, but keep in mind that I haven't tried compiling it, I'm just typing into github:

#[binrw::parser(reader, endian)]
fn linked_elements(mut next: u16) -> BinResult<Vec<Meta>> {
    let mut data = Vec::new();
    if next == 0 { // this is a micro-optimization, we don't need to check/restore position if we're not going anywhere.
        return Ok(data);
    }
    
    let restore_pos = reader.stream_position()?;
    while next != 0 {
        reader.seek(SeekFrom::Start(next as usize))?;
        let element = Meta::read_options(reader, endian, ())?;
        next = element.meta_next;
        // consider converting into version of meta that doesn't contain next pointer here
        data.push(element);
    }

    reader.seek(SeekFrom::Start(restore_position))?;
    Ok(data)
}

If you really need it to be generic (because there's a bunch of other linked list types) you could add some trait machinery.

trait LinkedListItem: BinRead<Args = ()> {
    fn get_next(&self) -> Option<SeekFrom>;
}

impl LinkedListItem for Meta {
    fn get_next(&self) -> Option<SeekFrom> {
        if self.meta_next == 0 {
            None
        } else {
            Some(SeekFrom::Start(self.meta_next as usize))
        }
    }
}

#[binrw::parser(reader, endian)]
fn linked_elements<T: LinkedListItem>(start: usize) -> BinResult<Vec<Meta>> {
    let mut data = Vec::new();
    if start == 0 { // this is a micro-optimization, we don't need to check/restore position if we're not going anywhere.
        return Ok(data);
    }

    let mut next = Some(SeekFrom::Start(start));
    
    let restore_pos = reader.stream_position()?;
    while let Some(seek) = next {
        reader.seek(seek)?;
        let element = <T>::read_options(reader, endian, ())?;
        next = element.get_next();
        // consider converting into version of meta that doesn't contain next pointer here
        data.push(element);
    }

    reader.seek(SeekFrom::Start(restore_position))?;
    Ok(data)
}

Usage of these would look something like:

#[derive(Debug)]
#[binread]
struct Root {  // Top level struct - MAY point to a Meta
    root_data: u16,
    #[br(temp)]
    meta_first: u16, // Points to first meta element (at pos 8)
    #[br(parse_with = linked_elements, args(meta_first as usize))]
    meta: Vec<Meta> 
    // Possible more fields, including other linked lists
}

There might be a better way to handle the start, e.g. by assuming that the current position is the first element and wrapping it in a FilePtr, or assuming that the root ptr is present at the current position.