When is a vulnerability not a vulnerability?

[nigellh]

I raised an issue #2907 where DT appeared to list a vulnerability that did not appear to be valid. I am still unsure if it should not still be open, but it raises a wider question of when is a vulnerability not a vulnerability.

DT has four standard sources and a commercial one.

NVD
Sonatype
Github Advisories
Google Advisories
Snyke (Commercial)

Which of these is correct or is it the case that if one lists a vulnerability that none of the others have, DT lists it as an active one. Why is the case that one lists a CVE and none of the others do (and not just the above reporting tools). Yes I have seen this. Is it based on a 'majority' or some other way of knowing if it needs to be addressed or not.

What are we supposed to do when DT lists a vulnerability that is disputed. Do we treat this as an active/pending/not a vulnerability.

All of this impacts those who trust DT to provide accurate results that we can work with. I have had a number of conversations with groups in the company and no-one is quite sure. Not specific to DT, but any tool and whether it uses one or multiple reporting tools.

As I was investigating DT vs Mend vs the various vulnerability repos such as the above, I am amazed at the differences they report. A vulnerability might appear in one of them, some of them or all of them. It might be disputed and there are certainly vulnerabilities being reported by some where the version found is no longer shows as having the vulnerability in. (What I would term a false positive)

If someone uses DT or even just the NVD and it does not list a vulnerability, but Snyk does, they could easily be putting their product/company at risk if it is a valid vulnerability.

I have no issue with multiple reporting structures as long as they provide 'sound' information you can act on.

To give you an idea of the differences of opinion, in a meeting where this was being discussed, one person was suggesting that you do not want to have a tool that brings in all of the repos and another said that you want a tool that can target the technology the product is written in. My answer is, most companies write in multiple technologies so it needs to cover a wide range of repos to ensure you get most if not all of them.

Is a tool that reports 95% of the vulnerabilities accurately good enough, or one that lists 100% of the critical/high ones, but 80% of the low/medium ones good enough.

My head aches and I wish I had more time to look into this so thought I would ask those who have far more experience than I do.

[nscuro]

Thanks for raising this, I think that's an issue most of us in the industry have been battling with in form or the other.

Quoting my comment from #2907 for completeness sake:

[...] in general there will always be cases where vulnerability intelligence services disagree with each other. They may report the same most of the time, but sometimes there will be differences. In fact, in some cases you may be paying services to provide info that others do not (e.g. Snyk).

The more sources you enable, the higher the chance of this occurring.

Now, addressing some of your questions:

Why is the case that one lists a CVE and none of the others do (and not just the above reporting tools).

Differences in data quality and / or freshness. What you describe can happen in multiple scenarios:

• The source reporting the issue is just plain wrong (false positive)
• The source reporting the issue is the only one with accurate data
  • Perhaps "official" data from NVD is bad, but GitHub Advisories corrected it
• The source reporting the issue is the first who saw it
  • One of the reasons you'd pay a commercial service is to get notified sooner

Thus, I can't foresee DT (or any other tool) ignoring certain findings, if they were only reported by 1 out of 5 sources. What we could do, is provide a "confidence" scoring, which would be higher or lower, depending on how many of the enabled sources reported the finding. That at least would make this information available to users.

What are we supposed to do when DT lists a vulnerability that is disputed. Do we treat this as an active/pending/not a vulnerability.

Technically, DT could detect this and either not report disputed vulns at all, or make an automated analysis decision, and / or suppress them. I'd argue the "what are we supposed to do" comes down to your risk appetite. Will you trust the NVD that a CVE is indeed not a vulnerability, or do you prefer to double-check yourself, just in case?

If someone uses DT or even just the NVD and it does not list a vulnerability, but Snyk does, they could easily be putting their product/company at risk if it is a valid vulnerability.

Not sure I get this point, could you elaborate?

[nigellh]

Hi Niklas

We have DT set up to use

NVD
Sonatype
Github Advisories
Google Advisories

But not Snyk as that is commercial. Snyk is reporting vulnerabilities that the others are not. Some critical and high. If we just rely on DT for reporting as it is currently set up, we would be missing vulnerabilities that could cause us a problem. We only became aware of this when comparing the Mend output vs the DT output for the same code and wondered at the differences.

You have to know about vulnerabilities to be able to address them. At the moment, just relying on DT as we have it set up will mean that we are missing some as Mend is picking up the ones that DT is missing and that is likely from Snyk (All the missing ones are in there). So either we pay a commercial license for Snyk and get it running in DT, or we are at risk of having vulnerabilities in products.

(PS Just an example. We do not just rely on DT, but it gets the point across! We are looking at DT as an option for some things.)

[nigellh]

I should say that this issue is not DT specific, but thought it a good place to have it discussed due to the expertise here!!!