Bug? --reference-vfilter *maybe* needs some more massage?

[mr44er]

I have a grainy anamorph source.mkv, denoising and deanamorphing via ffmpeg and exporting lossless as ffv1 as preparation-step:

ffmpeg -i source.mkv -vcodec ffv1 -level 3 -vf "fieldmatch,crop=716:556:2:10,split[a][b],[a]bm3d=sigma=20:block=8:bstep=2:group=1:estim=basic[a],[b][a]bm3d=sigma=20:block=8:bstep=2:group=16:estim=final:ref=1,scale=764:556:flags=lanczos,setsar=1:1" -an -sn -dn output_denoised.mkv
This works and looks as intended. But that is slow AF, does not even use half of the cores and is a huuuge "temp"-file.

crf-search with it:

ab-av1 crf-search --svt tune=3 --svt enable-overlays=1 --svt enable-dlf=2 --svt enable-tf=0 --min-crf 5 --thorough -i output_denoised.mkv
- crf 30 VMAF 87.94 (4%)
- crf 17 VMAF 94.18 (9%)
- crf 5 VMAF 98.96 (32%)
- crf 15 VMAF 95.08 (10%)
- crf 16 VMAF 94.61 (10%)
  00:00:25 ################################################################################################ (sampling crf 16, eta 0s)
Encode with: ab-av1 encode -i output_denoised.mkv --crf 15 --svt tune=3 --svt enable-overlays=1 --svt enable-dlf=2 --svt enable-tf=0

crf 15 VMAF 95.08 predicted video stream size 132.13 MiB (10%) taking 38 seconds

Yeah, also working as intended, VMAF-score of source.mkv is completely irrelevant at this point and it is my understanding of --reference-vfilter.

But his does not work:
ab-av1 crf-search --svt tune=3 --svt enable-overlays=1 --svt enable-dlf=2 --svt enable-tf=0 --min-crf 5 --thorough --reference-vfilter "fieldmatch,crop=716:556:2:10,split[a][b],[a]bm3d=sigma=20:block=8:bstep=2:group=1:estim=basic[a],[b][a]bm3d=sigma=20:block=8:bstep=2:group=16:estim=final:ref=1,scale=764:556:flags=lanczos,setsar=1:1" -i source.mkv

Error:

Metadata:
    ENCODER         : Lavf60.16.100
  Duration: 00:00:20.04, start: 0.000000, bitrate: 6005 kb/s
  Stream #1:0(eng): Video: mpeg2video (Main), yuv420p(tv, bt470bg, top first), 720x576 [SAR 16:15 DAR 4:3], 6000 kb/s, 25 fps, 25 tbr
, 1k tbn
    Metadata:
      BPS-eng         : 5999876
      DURATION-eng    : 00:06:17.680000000
      NUMBER_OF_FRAMES-eng: 9442
      NUMBER_OF_BYTES-eng: 283254162
      SOURCE_ID-eng   : 0100E0
      _STATISTICS_WRITING_APP-eng: MakeMKV v1.17.5 win(x64-release)
      _STATISTICS_WRITING_DATE_UTC-eng: 2024-09-09 14:05:03
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES SOURCE_ID
      DURATION        : 00:00:20.040000000
    Side data:
      cpb: bitrate max/min/avg: 6000000/0/0 buffer size: 1835008 vbv_delay: N/A
[libdav1d @ 0x24f49d511400] libdav1d 1.4.3
Stream mapping:
  Stream #0:0 (libdav1d) -> format:default
  Stream #1:0 (mpeg2video) -> format:default
  libvmaf:default -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
[Parsed_libvmaf_15 @ 0x24f49d4255c0] input width must match.
[Parsed_libvmaf_15 @ 0x24f49d4255c0] Failed to configure input pad on Parsed_libvmaf_15
[fc#0 @ 0x24f49d46b080] Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
Error while filtering: Invalid argument
[Parsed_libvmaf_15 @ 0x24f4aadefe00] input width must match.
[Parsed_libvmaf_15 @ 0x24f4aadefe00] Failed to configure input pad on Parsed_libvmaf_15
[fc#0 @ 0x24f49d46b080] Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
[out#0/null @ 0x24f49d4240c0] Nothing was written into output file, because at least one of its streams received no packets.
frame=    0 fps=0.0 q=0.0 Lsize=       0kB time=N/A bitrate=N/A speed=N/A    
Conversion failed!

I read #213 , but it isn't clear to me if this would be a bug or not possible at all. If it is possible, this would save a lot of time, load up every core and avoid the huge temp-file/the prep-step at all.

[alexheretic]

--reference-vfilter is used only for vmaf analysis, mostly exists to allow overriding the vfilter for vmaf.

In this case I think you'd just use --vfilter which would apply it for sample encoding and for vmaf analysis, so you should get an equivalent crf-search to your example but without the preparation step.

[mr44er]

Yes, --vfilter works. Tested both again with preset0 in hope for more differences. This file is short (~6 min) and one sample file is not enough it seems. I swear with previous files, the difference was too big to ignore and this way aspect ratio change alone punishes VMAF-calculation unneccesarily and this is something I want to override with --reference-vfilter, so source.mkv should not be included in reference at all. Reference should be everything after the whole filter chain.

ab-av1 auto-encode --svt tune=3 --svt enable-overlays=1 --svt enable-dlf=2 --svt enable-tf=0 --min-crf 5 --thorough -i output_denoised.mkv --preset 0 --output test1_correct.mkv
  Searching 00:00:01 ################################################################################################################# (crf 18, VMAF 95.30, size 9%)
  Encoding  00:18:04 ############################################################################################################################# (8.7 fps, eta 0s)
Encoded 120.34 MiB (9%)

auto-encode --svt tune=3 --svt enable-overlays=1 --svt enable-dlf=2 --svt enable-tf=0 --min-crf 5 --thorough --vfilter "fieldmatch,crop=716:556:2:10,split[a][b],[a]bm3d=sigma=20:block=8:bstep=2:group=1:estim=basic[a],[b][a]bm3d=sigma=20:block=8:bstep=2:group=16:estim=final:ref=1,scale=764:556:flags=lanczos,setsar=1:1" -i source.mkv --preset 0 --video-only --output test2_notwanted.mkv
  Searching 00:35:41 ################################################################################################################ (crf 18, VMAF 95.26, size 43%)
  Encoding  01:23:19 ############################################################################################################################# (1.9 fps, eta 0s)
Encoded 119.49 MiB (40%)

ffmpeg -i source.mkv -vcodec ffv1 -level 3 -vf "fieldmatch,crop=716:556:2:10,scale=764:556:flags=lanczos,setsar=1:1" -an -sn -dn output_notdenoised.mkv

gives me a 2,1GB file to begin with and

ffmpeg -i source.mkv -vcodec ffv1 -level 3 -vf "fieldmatch,crop=716:556:2:10,split[a][b],[a]bm3d=sigma=20:block=8:bstep=2:group=1:estim=basic[a],[b][a]bm3d=sigma=20:block=8:bstep=2:group=16:estim=final:ref=1,scale=764:556:flags=lanczos,setsar=1:1" -an -sn -dn output_denoised.mkv

a 1,4GB file.

That said, I want to achieve this with using source.mkv as input, but after the whole filter chain and without VMAF-reference to source.mkv:

ab-av1 crf-search --svt tune=3 --svt enable-overlays=1 --svt enable-dlf=2 --svt enable-tf=0 --min-crf 5 --min-samples 3 --thorough --preset 0 -i output_denoised.mkv
- crf 30 VMAF 89.43 (4%)
- crf 17 VMAF 94.95 (9%)
- crf 5 VMAF 99.03 (39%)
  00:11:14 ############################################################################################################################### (sampling crf 16, eta 0s)
Encode with: ab-av1 encode -i output_denoised.mkv --crf 16 --preset 0 --svt tune=3 --svt enable-overlays=1 --svt enable-dlf=2 --svt enable-tf=0

crf 16 VMAF 95.35 predicted video stream size 132.17 MiB (10%) taking 21 minutes

[alexheretic]

vmaf comparing the input after vfilter is how it behaves now. The vfilter is applied to each sample during vmaf comparison with the encoded sample (which had the same vfilter applied during encoding). This is why vmaf works at all here considering you are cropping.

So the crf-search is looking only at the quality of the encode, not the affect of vfilters as, in general, they cannot be compared (overridden by setting --reference-vfilter when you know better)

But in your case i think the default --vfilter behaviour is actually what you want.

[mr44er]

You're right. I was confused and the results gave me more confusion.

May I suggest the following changes to the docs and could you confirm whether I understood it correctly this time? :)

--vfilter <VFILTER>
          Ffmpeg video filter applied to the input before encoding. E.g. --vfilter "scale=1280:-1,fps=24".
          
          See https://ffmpeg.org/ffmpeg-filters.html#Video-Filters
          
          For VMAF calculations this is also applied to the reference video sample file(s), meaning VMAF scores represent the quality of
          original+vfilters (=what is produced *after* the filter chain) compared to the encoded result. To override this behaviour and 
          achieve the opposite set --reference-vfilter.


--reference-vfilter <REFERENCE_VFILTER>
          Ffmpeg video filter applied to the VMAF reference before analysis. E.g. --reference-vfilter "scale=1280:-1,fps=24".
          
          Overrides --vfilter which would otherwise be used.
          For VMAF calculations this would measure how much a filter/chain would score (=or "damage", example2 for denoising
          with sane defaults --reference-vfilter "hqdn3d") compared directly to the original. 

[alexheretic]

Thanks, I've reworded some of that in #223.

I'm not sure about the "achieve the opposite" wording, as --reference-vfilter really allows anything to be used there to process a source before VMAF. It is up to the user. I don't quite understand the --reference-vfilter "hqdn3d" example you propose.

[mr44er]

I'm not sure about the "achieve the opposite" wording

Yeah, that's likely more confusing...

I don't quite understand the --reference-vfilter "hqdn3d" example you propose.

hqdn3d within defaults isn't a big destroyer quality-wise. In my experience in combination with VMAF calculation this spreads mostly within the range +4/-4 VMAF points vs. +4/-4 CRF and is mostly known for what it does, if guys used handbrake in the past, it was or still is the main denoiser. "I wanted to find a known filter that doesn't hurt much or prevents finding a CRF" :)
The dedot-Filter is more unknown, effect mostly too small to see it even on the results.