Test data format

[Janai2019]

X_test = [sent2features(s) for s in test_sents]
Looking at the format of the test data, it seems to require a tagged test data to extract features especially, current tag. In reality, the purpose is to tag new data where such information is not present except word features. How do we tag new data?

[mani2106]

Normally you would tag a set of sentences and split them to train and test/eval sets.
To ensure that the model does not overfit (memorize) the training data. We predict with the test data and calculate the scores/metrics and decide whether it is suitable for real-world data.

This is what the example in the documentation does.