Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make scamper1 mda output to single JSON object (autoloadable) #169

Open
stephen-soltesz opened this issue Oct 8, 2024 · 0 comments
Open

Comments

@stephen-soltesz
Copy link
Contributor

The scamper1 traceroute format by default is four JSON objects, creating a JSONL file per trace. This format is not supported by the autoloading system. Ideally, traceroute-caller would support a native JSON output format for scamper1 traces.

The native scamper structure uses a double array, [][]ScamperLink which the bigquery.InferSchema cannot work with. The etl parser transforms the raw object into a new object that is structured compatibly with BigQuery schemas.

To achive an autoloadable, JSON output for scamper mda traces, this transformation process must be local to the traceroute caller. This does not necessarily imply we must remain backward compatible with scamper1 schema, only that the transform is local and the output is autoloadable JSON.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant