input_filename is always <stdin>

[qistoph]

I would expect input_filename to work like it does in jq, with yq however it always outputs <stdin>.

Actual result:

$ yq 'input_filename' */problem.yaml
"<stdin>"
"<stdin>"
"<stdin>"
"<stdin>"
"<stdin>"
"<stdin>"

Expected result:

$ yq 'input_filename' */problem.yaml
"project_alpha/problem.yaml"
"project_zero/problem.yaml"
"questions/problem.yaml"
"research/problem.yaml"
"stuff/problem.yaml"
"test/problem.yaml"

[kislyuk]

Thanks for pointing this out. Although the value of input_filename can be set externally in jq, I'm not sure how one would do that with multiple input documents. Open to suggestions.

[qistoph]

Just took a look at your code.
https://github.com/kislyuk/yq/blob/master/yq/__init__.py#L172

It seems to me you are concatenating all input files into one big stream. My guess would be that jq would need you to supply multiple separate input stream to make input_filename work. Don't know exactly how, but I'll try to figure out some more.

[kislyuk]

Right. Literally supplying multiple input streams is technically possible (for example, via a bunch of temporary FIFOs), but cumbersome and could be less performant.

[vergenzt]

Although the value of input_filename can be set externally in jq, I'm not sure how one would do that with multiple input documents. Open to suggestions.

@kislyuk how would you set the value of jq's input_filename externally without passing the path to a real file or pipe via command line arguments? I've spelunked in the jq codebase for a bit but can't figure out how one would cause input_filename to be a string other than one that was passed via command line args (either explicitly or through a glob).

[kislyuk]

@vergenzt you can set the input_filename variable via a jq command line argument.

[vergenzt]

@kislyuk which command line option are you seeing that does that? I don't see anything mentioned about this in the documentation.

And as far as I could tell in the codebase, the value returned by input_filename is the same as the value passed to fopen (with the exception of setting the filename to "<stdin>" if input == stdin). There's -f / --from-file, but that just reads the jq filter from that file, it doesn't set input_filename.

Code references:

• https://github.com/stedolan/jq/blob/jq-1.6/src/util.c#L291-L305
• https://github.com/stedolan/jq/blob/jq-1.6/src/builtin.c#L1653
• https://github.com/stedolan/jq/blob/jq-1.6/src/builtin.c#L1553
• https://github.com/stedolan/jq/blob/jq-1.6/src/util.c#L393

[vergenzt]

My current understanding is that the only way to control input_filename is by actually creating a file on the filesystem with the given path/name.

Would you be open to a PR adding some sort of --use-temp-fifos option to create temporary named FIFOs for each input file? Could use mktemp with a format of <filename>.temp_XXXXX.json so that the original filename is recoverable by input_filename | sub(".temp_\w+.json$", "").

(Or personally I'd be a fan of just testing whether there's a noticeable performance hit with a large number of files, and if there's not then just defaulting to making the FIFOs so that the default is for input_filename to be something based on the original filename.)

E.g. for original files ./myfile1.yaml and ./myfile2.yaml, create FIFOs ./myfile.yaml.temp_In49e.json and ./myfile2.yaml.temp_coHi9.json and pass those to jq instead of concatenating them on stdin.

[kislyuk]

@vergenzt I am open to the design you describe in principle, as long as it's not the default (since there are a lot more things that can potentially break with the fifo approach, I don't want to make it the default for now).

[kislyuk]

@vergenzt you're right, there is no way to set input_filename - in the test I ran earlier, I confused input_filename with $input_filename.

[qistoph]

How about checking with the jq authors to see if they're willing in include an argument of variable to enable this functionality?

[kislyuk]

@qistoph please feel free to do that. However, I don't think it will solve this issue unless you also want to design a protocol for naming all documents in a stream and passing the names of those documents as a list to jq.

[qistoph]

Omg! I'm so sorry! I really didn't notice it was a different tool that I'm using...