Add serialization support to FstAddOn #162
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds the possibility to serialize/deserialize a
FstAddOn
from binary data. This makes it possible to load a lookahead FST (specialMatcherFst
) from binary data.Changes
SerializeBinary
requirement forSerializableFst
trait. This made it possible to read/write an FST from/to binary data slice, instead of a path.SerializeBinary
forIntInterval
,VectorIntervalStore
,IntervalSet
andLabelReachableData
. These implementations are ported from relevant OpenFst implementations to make it compatible with files created by OpenFst.Label
is assumed as 32 bit data, because OpenFst usesint
forLabel
.Fst<W>
requirement forfst
field ofFstAddOn
. I think this is better to add this requirement by definition of the struct.SerializeBinary
forFstAddOn
with AddOnPair ((Option<Arc<AO1>>, Option<Arc<AO2>>)
). For now, this seems to be the only variant ofFstAddOn
used in the project. More generic implementation may be added.fst_type
field is added toFstAddOn
to implement this, OpenFst has a type name field inAddOnImpl
as well. I have given names{i,o}label_lookahead
toFstAddOn
variables defined inmatcher_fst.rs
. Names are taken from OpenFst, and they seem compatible with the binary output of OpenFst.MatcherFst
allowing to create it from already computed (or read)FstAddOn
.Status
olabel_lookahead
FST file created with OpenFst. I did not added a proper test yet. I think it may be necessary to add it.