Training with CSIQ-VQA database with OptimizeWarning #2
Comments
|
Hi Hugh
The optimization warning comes up when there's no good fit available for
the mapping. This can happen when the predictions are very different from
the scores, which I suppose is happening here. I've not tested ChipQA on
the CSIQ database but I would expect the SRCC to be around 0.7 since
BRISQUE and TLVQM get an SRCC of around that level (Table IV in
https://arxiv.org/pdf/2206.14713.pdf).
In line 67 of cleaner_svr.py, I'm making the assumption that the first
element of the '_' separated list of the filenames in the dataframe
corresponds to the content. If this is not the case, you have to define a
'content' column in the dataframe yourself. Also the code removes
distortions marked as 'p', which in our original database corresponded to
proprietary content. That might be one reason why performance is poor.
I will update cleaner_svr.py to fix these things, but you may have to make
more changes to the SVR code to make it run correctly for CSIQ, depending
on how you created your score CSV.
Regards,
Josh
…On Tue, Feb 14, 2023 at 3:16 AM IRON-HUGHY ***@***.***> wrote:
Hi Joshua,
I tried to use chipQA to train and test its performance on a CSIQ-VQA
database, which is a 480p resolution database.
1 - I run chipqa_yuv.py get the feature file
2 - I run python cleaner_svr.py --score_file /path/to/score.csv
--feature_folder feature_folder --train_and_test
I got the all_pred = [33.22243675 49.20781647 50.0144947 41.65980157
41.57749611 40.87659356, 61.43243688 58.08024139 38.79844636 58.73480296
46.18201971 36.00328378, 36.96726173 42.92148637 53.04071207 63.78199282
53.71268326 31.00923555, 43.38495816 44.25350569 43.77634616 44.97062795
44.97958702 44.92278459, 49.21896745 50.34141739 48.01144443 50.7113197
53.70022118 53.09089083, 36.81422243 42.11125906 47.5120631 48.23488525
52.38174794 59.13939887, 43.95188126 40.72350856 39.07306079 39.36255459
38.9502675 37.87678828, 36.04681885 38.40384619 41.65701426 40.27273715
42.31758531 43.51008817, 33.85725812 39.69753213 48.37801187 43.89804409
44.25391719 44.2835234 ]
while all_dmos = [30.96926305 55.12598144 68.47434569 63.19511855
44.82338636 39.05790258, 61.77650141 43.23640628 27.29835247 55.44590755
39.26934316 28.35209766, 29.65262045 41.59899252 55.96629109 71.88924176
34.53450825 17.19991484, 26.41774852 46.88645613 79.15402107 50.65216125
45.56089672 36.8136669, 64.98568555 49.49478196 29.56212976 68.14515278
49.32783463 39.70465263, 29.9807714 40.99972896 58.42660889 85.19789617
53.10270256 24.20572819, 31.1199481 51.57753506 64.18712587 61.9394302
52.56382728 39.07539394, 61.0901873 42.19819949 27.14502109 73.33284815
42.48086625 34.19254771, 26.21587958 40.64126874 52.39742685 70.05419936
42.42780915 26.91805556]
I would get "lib\site-packages\scipy\optimize\minpack.py:834:
OptimizeWarning: Covariance of the parameters could not be estimated
category=OptimizeWarning)" several times during this step. And the results
of PLCC and SROCC only (0.33308124176152726, 0.3385172479512102).
Can you tell me how to solve this problem and get the right result.
Thanks in advance,
Hugh
—
Reply to this email directly, view it on GitHub
<#2>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADT5VDF66JNF2JZBNQ65EY3WXNEOJANCNFSM6AAAAAAU3JPVUE>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
|
Hi Hugh I've fixed cleaner_svr.py. Please check and see if it works now. Regards |
Hi Josh, and no warning appeared after the adjustment.But there does not seem to be a fitting operation. In the end, I tried to ignore these warnings, and the end result was (PLCC,SROCC )only (0.43037323037323044, 0.4201873570797587).It could be this database data is no good fit available for the mapping. Regards, |
|
Hi Hugh
Ok. In any case, the fitting will not be necessary for NR algorithms
because the SVR is mapping to the score space. Fitting is more relevant for
FR algorithms which output some fidelity measure which needs to be mapped
to scores.
Also, these results are usually reported as median results for 100
content-separated train-test splits. In addition, using a linear SVR kernel
is known to sometimes give better performance than RBF kernels for these
features.
If you've done all these things then I suppose this database is just a
challenging one for ChipQA.
Regards,
Josh
…On Wed, Feb 15, 2023 at 2:12 AM IRON-HUGHY ***@***.***> wrote:
Hi Hugh The optimization warning comes up when there's no good fit
available for the mapping. This can happen when the predictions are very
different from the scores, which I suppose is happening here. I've not
tested ChipQA on the CSIQ database but I would expect the SRCC to be around
0.7 since BRISQUE and TLVQM get an SRCC of around that level (Table IV in
https://arxiv.org/pdf/2206.14713.pdf). In line 67 of cleaner_svr.py, I'm
making the assumption that the first element of the '_' separated list of
the filenames in the dataframe corresponds to the content. If this is not
the case, you have to define a 'content' column in the dataframe yourself.
Also the code removes distortions marked as 'p', which in our original
database corresponded to proprietary content. That might be one reason why
performance is poor. I will update cleaner_svr.py to fix these things, but
you may have to make more changes to the SVR code to make it run correctly
for CSIQ, depending on how you created your score CSV. Regards, Josh
… <#m_6900202025710888295_m_-6479947538803208170_>
On Tue, Feb 14, 2023 at 3:16 AM IRON-HUGHY *@*.*> wrote: Hi Joshua, I
tried to use chipQA to train and test its performance on a CSIQ-VQA
database, which is a 480p resolution database. 1 - I run chipqa_yuv.py get
the feature file 2 - I run python cleaner_svr.py --score_file
/path/to/score.csv --feature_folder feature_folder --train_and_test I got
the all_pred = [33.22243675 49.20781647 50.0144947 41.65980157 41.57749611
40.87659356, 61.43243688 58.08024139 38.79844636 58.73480296 46.18201971
36.00328378, 36.96726173 42.92148637 53.04071207 63.78199282 53.71268326
31.00923555, 43.38495816 44.25350569 43.77634616 44.97062795 44.97958702
44.92278459, 49.21896745 50.34141739 48.01144443 50.7113197 53.70022118
53.09089083, 36.81422243 42.11125906 47.5120631 48.23488525 52.38174794
59.13939887, 43.95188126 40.72350856 39.07306079 39.36255459 38.9502675
37.87678828, 36.04681885 38.40384619 41.65701426 40.27273715 42.31758531
43.51008817, 33.85725812 39.69753213 48.37801187 43.89804409 44.25391719
44.2835234 ] while all_dmos = [30.96926305 55.12598144 68.47434569
63.19511855 44.82338636 39.05790258, 61.77650141 43.23640628 27.29835247
55.44590755 39.26934316 28.35209766, 29.65262045 41.59899252 55.96629109
71.88924176 34.53450825 17.19991484, 26.41774852 46.88645613 79.15402107
50.65216125 45.56089672 36.8136669, 64.98568555 49.49478196 29.56212976
68.14515278 49.32783463 39.70465263, 29.9807714 40.99972896 58.42660889
85.19789617 53.10270256 24.20572819, 31.1199481 51.57753506 64.18712587
61.9394302 52.56382728 39.07539394, 61.0901873 42.19819949 27.14502109
73.33284815 42.48086625 34.19254771, 26.21587958 40.64126874 52.39742685
70.05419936 42.42780915 26.91805556] I would get
"lib\site-packages\scipy\optimize\minpack.py:834: OptimizeWarning:
Covariance of the parameters could not be estimated
category=OptimizeWarning)" several times during this step. And the results
of PLCC and SROCC only (0.33308124176152726, 0.3385172479512102). Can you
tell me how to solve this problem and get the right result. Thanks in
advance, Hugh — Reply to this email directly, view it on GitHub <#2
<#2>>, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADT5VDF66JNF2JZBNQ65EY3WXNEOJANCNFSM6AAAAAAU3JPVUE
<https://github.com/notifications/unsubscribe-auth/ADT5VDF66JNF2JZBNQ65EY3WXNEOJANCNFSM6AAAAAAU3JPVUE>
. You are receiving this because you are subscribed to this thread.Message
ID: @.*>
Hi Josh,
Thank you very much for your reply, but it doesn't appear to be the
problem with the.csv file, because the video file in the CSIQ database is
named very similar to the one in livestream, and the content of the video
is before the first '_' separator.I adjusted the sigmoid entry in curve_fit
() to make it less easy to overflow encountered in exp.
def sigmoids(inx):
if inx >= 0: #
return 1.0 / (1 + np.exp(-inx))
else:
return np.exp(inx) / (1 + np.exp(inx))
preds_fitted fitted with this adjustment,
def sigmoid(x):
x_ravel = x.ravel() # numpy
length = len(x_ravel)
y = []
for index in range(length):
if x_ravel[index] >= 0:
y.append(1.0 / (1 + np.exp(-x_ravel[index])))
else:
y.append(np.exp(x_ravel[index]) / (np.exp(x_ravel[index]) + 1))
return np.array(y).reshape(x.shape)
and no warning appeared after the adjustment.But there does not seem to be
a fitting operation. In the end, I tried to ignore these warnings, and the
end result was (PLCC,SROCC )only (0.43037323037323044,
0.4201873570797587).It could be this database data is no good fit available
for the mapping.
Regards,
Hugh
—
Reply to this email directly, view it on GitHub
<#2 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADT5VDCFU4IU3GO375BOBEDWXSFV3ANCNFSM6AAAAAAU3JPVUE>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
Hi Josh Sorry for the late reply. I have used a linear SVR kernel and tried 1000 times content-separated train-test splits. The result of PLCC and SROCC did improve, and now the results have become(0.522230860287501, 0.5335907335907336). Regards, |
Hi Joshua,
I tried to use chipQA to train and test its performance on a CSIQ-VQA database, which is a 480p resolution database.
1 - I run chipqa_yuv.py get the feature file
2 - I run python cleaner_svr.py --score_file /path/to/score.csv --feature_folder feature_folder --train_and_test
I got the all_pred = [33.22243675 49.20781647 50.0144947 41.65980157 41.57749611 40.87659356, 61.43243688 58.08024139 38.79844636 58.73480296 46.18201971 36.00328378, 36.96726173 42.92148637 53.04071207 63.78199282 53.71268326 31.00923555, 43.38495816 44.25350569 43.77634616 44.97062795 44.97958702 44.92278459, 49.21896745 50.34141739 48.01144443 50.7113197 53.70022118 53.09089083, 36.81422243 42.11125906 47.5120631 48.23488525 52.38174794 59.13939887, 43.95188126 40.72350856 39.07306079 39.36255459 38.9502675 37.87678828, 36.04681885 38.40384619 41.65701426 40.27273715 42.31758531 43.51008817, 33.85725812 39.69753213 48.37801187 43.89804409 44.25391719 44.2835234 ]
while all_dmos = [30.96926305 55.12598144 68.47434569 63.19511855 44.82338636 39.05790258, 61.77650141 43.23640628 27.29835247 55.44590755 39.26934316 28.35209766, 29.65262045 41.59899252 55.96629109 71.88924176 34.53450825 17.19991484, 26.41774852 46.88645613 79.15402107 50.65216125 45.56089672 36.8136669, 64.98568555 49.49478196 29.56212976 68.14515278 49.32783463 39.70465263, 29.9807714 40.99972896 58.42660889 85.19789617 53.10270256 24.20572819, 31.1199481 51.57753506 64.18712587 61.9394302 52.56382728 39.07539394, 61.0901873 42.19819949 27.14502109 73.33284815 42.48086625 34.19254771, 26.21587958 40.64126874 52.39742685 70.05419936 42.42780915 26.91805556]
I would get "lib\site-packages\scipy\optimize\minpack.py:834: OptimizeWarning: Covariance of the parameters could not be estimated
category=OptimizeWarning)" several times during this step. And the results of PLCC and SROCC only (0.33308124176152726, 0.3385172479512102).
Can you tell me how to solve this problem and get the right result.
Thanks in advance,
Hugh
The text was updated successfully, but these errors were encountered: