近期关于供应链安全与漏洞数据库的开源项目

[will-ww]

Google 近期陆续发布了一些开源项目，关于供应链与漏洞数据库的，是些比较好的数据源，可以利用起来。

1、osv.dev

开源漏洞数据库，是个不错的漏洞数据源。

仓库地址：https://github.com/google/osv.dev

2、OSV-Scanner​

Google 推出免费工具 OSV-Scanner，供开源开发人员可以更简单地存取和项目相关的漏洞资讯。该项目所使用到的数据即为上面的 osv 项目提供。参考信息：这里

仓库地址：https://github.com/google/osv-scanner

3、https://deps.dev/

Open Source Insights 是一项由 Google 开发和托管的实验性服务，旨在帮助开发人员更好地了解开源软件包的结构、构造和安全性。该服务检查每个包，构建其依赖关系及其属性的完整、详细的图表，并将结果提供给任何可以从中受益的人。目标是为开发人员提供一张图片，说明他们的软件是如何组合在一起的，它是如何随着依赖关系的变化而变化的，以及可能产生的后果。

该服务扫描数百万个开源包，构建它们的依赖关系图，并使用有关所有权、许可证、受欢迎程度和其他元数据的信息对其进行注释。它会定期更新数据以保持信息的最新性和相关性，同时允许开发人员回顾并了解事情随着时间的推移发生了怎样的变化。

该项目非开源，但上面的数据应该可以很好的自动化抓取，是个不错的参考。

[bifenglin]

首先这个数据集我调研了一下，根据 https://security.googleblog.com/2021/02/launching-osv-better-vulnerability.html 的介绍说是现在已经收集了上千个漏洞从380多个不同语言类型的项目中。所以现在这些数据量应该不大，所以实现较容易。

然后我调研了一下 https://osv.dev/ 的三个API，一个API是根据commit来获取信息的（这些commit所在的项目必须在380个项目里面），然后返回的结果如下：

{
    "vulns":[
        {
            "id":"OSV-2020-484",
            "summary":"Heap-buffer-overflow in AAT::KerxSubTableFormat4\u003cAAT::KerxSubTableHeader\u003e::driver_context_t::transition",
            "details":"OSS-Fuzz report: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=12532\n\n```\nCrash type: Heap-buffer-overflow READ 4\nCrash state:\nAAT::KerxSubTableFormat4\u003cAAT::KerxSubTableHeader\u003e::driver_context_t::transition\nvoid AAT::StateTableDriver\u003cAAT::ExtendedTypes, AAT::KerxSubTableFormat4\u003cAAT::Ker\nAAT::KerxSubTableFormat4\u003cAAT::KerxSubTableHeader\u003e::apply\n```\n",
            "modified":"2022-04-13T03:04:32.842142Z",
            "published":"2020-07-01T00:00:12.297418Z",
            "references":[
                {
                    "type":"REPORT",
                    "url":"https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=12532"
                }
            ],
            "affected":[
                {
                    "package":{
                        "name":"harfbuzz",
                        "ecosystem":"OSS-Fuzz",
                        "purl":"pkg:generic/harfbuzz"
                    },
                    "ranges":[
                        {
                            "type":"GIT",
                            "repo":"https://github.com/harfbuzz/harfbuzz.git",
                            "events":[
                                {
                                    "introduced":"4009a05ca7de21fff2176621597cd0cd01e9d80e"
                                },
                                {
                                    "fixed":"cc8e9a436fa408a1c63f4b9afb7643cea76a079c"
                                }
                            ]
                        }
                    ],
                    "versions":[
                        "2.2.0",
                        "2.3.0"
                    ],
                    "ecosystem_specific":{
                        "severity":"MEDIUM"
                    },
                    "database_specific":{
                        "source":"https://github.com/google/oss-fuzz-vulns/blob/main/vulns/harfbuzz/OSV-2020-484.yaml"
                    }
                }
            ],
            "schema_version":"1.3.0"
        }
    ]
}

从以上信息来看对我们来说用处不大，第二个API是根据包名（项目名）来查看信息，结果如下：

    "vulns":[
        {
            "id":"GHSA-462w-v97r-4m45",
            "summary":"High severity vulnerability that affects Jinja2",
            "details":"In Pallets Jinja before 2.10.1, str.format_map allows a sandbox escape.",
            "aliases":[
                "CVE-2019-10906"
            ],
            "modified":"2022-12-14T05:09:46.401758Z",
            "published":"2019-04-10T14:30:24Z",
            "database_specific":{
                "cwe_ids":[

                ],
                "severity":"HIGH",
                "github_reviewed":true
            },
            "references":[
                {
                    "type":"ADVISORY",
                    "url":"https://nvd.nist.gov/vuln/detail/CVE-2019-10906"
                },
                {
                    "type":"WEB",
                    "url":"https://access.redhat.com/errata/RHSA-2019:1152"
                },
                {
                    "type":"WEB",
                    "url":"https://access.redhat.com/errata/RHSA-2019:1237"
                },
                {
                    "type":"WEB",
                    "url":"https://access.redhat.com/errata/RHSA-2019:1329"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://github.com/advisories/GHSA-462w-v97r-4m45"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/09fc842ff444cd43d9d4c510756fec625ef8eb1175f14fd21de2605f@%3Cdevnull.infra.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/2b52b9c8b9d6366a4f1b407a8bde6af28d9fc73fdb3b37695fd0d9ac@%3Cdevnull.infra.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/320441dccbd9a545320f5f07306d711d4bbd31ba43dc9eebcfc602df@%3Cdevnull.infra.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/46c055e173b52d599c648a98199972dbd6a89d2b4c4647b0500f2284@%3Cdevnull.infra.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/57673a78c4d5c870d3f21465c7e2946b9f8285c7c57e54c2ae552f02@%3Ccommits.airflow.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/7f39f01392d320dfb48e4901db68daeece62fd60ef20955966739993@%3Ccommits.airflow.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/b2380d147b508bbcb90d2cad443c159e63e12555966ab4f320ee22da@%3Ccommits.airflow.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/f0c4a03418bcfe70c539c5dbaf99c04c98da13bfa1d3266f08564316@%3Ccommits.airflow.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.fedoraproject.org/archives/list/[email protected]/message/DSW3QZMFVVR7YE3UT4YRQA272TYAL5AF/"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.fedoraproject.org/archives/list/[email protected]/message/QCDYIS254EJMBNWOG4S5QY6AOTOR4TZU/"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.fedoraproject.org/archives/list/[email protected]/message/TS7IVZAJBWOHNRDMFJDIZVFCMRP6YIUQ/"
                },
                {
                    "type":"WEB",
                    "url":"https://palletsprojects.com/blog/jinja-2-10-1-released"
                },
                {
                    "type":"WEB",
                    "url":"https://usn.ubuntu.com/4011-1/"
                },
                {
                    "type":"WEB",
                    "url":"https://usn.ubuntu.com/4011-2/"
                },
                {
                    "type":"WEB",
                    "url":"http://lists.opensuse.org/opensuse-security-announce/2019-05/msg00030.html"
                },
                {
                    "type":"WEB",
                    "url":"http://lists.opensuse.org/opensuse-security-announce/2019-06/msg00064.html"
                }
            ],
            "affected":[
                {
                    "package":{
                        "name":"jinja2",
                        "ecosystem":"PyPI",
                        "purl":"pkg:pypi/jinja2"
                    },
                    "ranges":[
                        {
                            "type":"ECOSYSTEM",
                            "events":[
                                {
                                    "introduced":"0"
                                },
                                {
                                    "fixed":"2.10.1"
                                }
                            ]
                        }
                    ],
                    "versions":[
                        "2.0",
                        "2.0rc1",
                        "2.1",
                        "2.1.1",
                        "2.10",
                        "2.2",
                        "2.2.1",
                        "2.3",
                        "2.3.1",
                        "2.4",
                        "2.4.1",
                        "2.5",
                        "2.5.1",
                        "2.5.2",
                        "2.5.3",
                        "2.5.4",
                        "2.5.5",
                        "2.6",
                        "2.7",
                        "2.7.1",
                        "2.7.2",
                        "2.7.3",
                        "2.8",
                        "2.8.1",
                        "2.9",
                        "2.9.1",
                        "2.9.2",
                        "2.9.3",
                        "2.9.4",
                        "2.9.5",
                        "2.9.6"
                    ],
                    "database_specific":{
                        "source":"https://github.com/github/advisory-database/blob/main/advisories/github-reviewed/2019/04/GHSA-462w-v97r-4m45/GHSA-462w-v97r-4m45.json"
                    }
                }
            ],
            "schema_version":"1.3.0",
            "severity":[
                {
                    "type":"CVSS_V3",
                    "score":"CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:N/A:N"
                }
            ]
        },
        {
            "id":"GHSA-8r7q-cvjq-x353",
            "summary":"Incorrect Privilege Assignment in Jinja2",
            "details":"The default configuration for bccache.FileSystemBytecodeCache in Jinja2 before 2.7.2 does not properly create temporary files, which allows local users to gain privileges via a crafted .cache file with a name starting with __jinja2_ in /tmp.",
            "aliases":[
                "CVE-2014-1402"
            ],
            "modified":"2022-12-14T04:42:41.418128Z",
            "published":"2022-05-14T04:04:14Z",
            "database_specific":{
                "cwe_ids":[
                    "CWE-266"
                ],
                "severity":"MODERATE",
                "github_reviewed":true
            },
            "references":[
                {
                    "type":"ADVISORY",
                    "url":"https://nvd.nist.gov/vuln/detail/CVE-2014-1402"
                },
                {
                    "type":"WEB",
                    "url":"https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=734747"
                },
                {
                    "type":"WEB",
                    "url":"https://bugzilla.redhat.com/show_bug.cgi?id=1051421"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://github.com/advisories/GHSA-8r7q-cvjq-x353"
                },
                {
                    "type":"WEB",
                    "url":"https://github.com/pypa/advisory-database/tree/main/vulns/jinja2/PYSEC-2014-8.yaml"
                },
                {
                    "type":"WEB",
                    "url":"https://oss.oracle.com/pipermail/el-errata/2014-June/004192.html"
                },
                {
                    "type":"WEB",
                    "url":"http://advisories.mageia.org/MGASA-2014-0028.html"
                },
                {
                    "type":"WEB",
                    "url":"http://jinja.pocoo.org/docs/changelog/"
                },
                {
                    "type":"WEB",
                    "url":"http://openwall.com/lists/oss-security/2014/01/10/2"
                },
                {
                    "type":"WEB",
                    "url":"http://openwall.com/lists/oss-security/2014/01/10/3"
                },
                {
                    "type":"WEB",
                    "url":"http://rhn.redhat.com/errata/RHSA-2014-0747.html"
                },
                {
                    "type":"WEB",
                    "url":"http://rhn.redhat.com/errata/RHSA-2014-0748.html"
                },
                {
                    "type":"WEB",
                    "url":"http://secunia.com/advisories/56287"
                },
                {
                    "type":"WEB",
                    "url":"http://secunia.com/advisories/58783"
                },
                {
                    "type":"WEB",
                    "url":"http://secunia.com/advisories/58918"
                },
                {
                    "type":"WEB",
                    "url":"http://secunia.com/advisories/59017"
                },
                {
                    "type":"WEB",
                    "url":"http://secunia.com/advisories/60738"
                },
                {
                    "type":"WEB",
                    "url":"http://secunia.com/advisories/60770"
                },
                {
                    "type":"WEB",
                    "url":"http://www.gentoo.org/security/en/glsa/glsa-201408-13.xml"
                },
                {
                    "type":"WEB",
                    "url":"http://www.mandriva.com/security/advisories?name=MDVSA-2014:096"
                }
            ],
            "affected":[
                {
                    "package":{
                        "name":"jinja2",
                        "ecosystem":"PyPI",
                        "purl":"pkg:pypi/jinja2"
                    },
                    "ranges":[
                        {
                            "type":"ECOSYSTEM",
                            "events":[
                                {
                                    "introduced":"0"
                                },
                                {
                                    "fixed":"2.7.2"
                                }
                            ]
                        }
                    ],
                    "versions":[
                        "2.0",
                        "2.0rc1",
                        "2.1",
                        "2.1.1",
                        "2.2",
                        "2.2.1",
                        "2.3",
                        "2.3.1",
                        "2.4",
                        "2.4.1",
                        "2.5",
                        "2.5.1",
                        "2.5.2",
                        "2.5.3",
                        "2.5.4",
                        "2.5.5",
                        "2.6",
                        "2.7",
                        "2.7.1"
                    ],
                    "database_specific":{
                        "source":"https://github.com/github/advisory-database/blob/main/advisories/github-reviewed/2022/05/GHSA-8r7q-cvjq-x353/GHSA-8r7q-cvjq-x353.json"
                    }
                }
            ],
            "schema_version":"1.3.0"
        },
        {
            "id":"GHSA-g3rq-g295-4j3m",
            "summary":"Regular Expression Denial of Service (ReDoS) in Jinja2",
            "details":"This affects the package jinja2 from 0.0.0 and before 2.11.3. The ReDOS vulnerability of the regex is mainly due to the sub-pattern [a-zA-Z0-9._-]+.[a-zA-Z0-9._-]+ This issue can be mitigated by Markdown to format user content instead of the urlize filter, or by implementing request timeouts and limiting process memory.",
            "aliases":[
                "CVE-2020-28493"
            ],
            "modified":"2022-12-14T04:46:39.750547Z",
            "published":"2021-03-19T21:28:05Z",
            "database_specific":{
                "cwe_ids":[
                    "CWE-400"
                ],
                "severity":"MODERATE",
                "github_reviewed":true
            },
            "references":[
                {
                    "type":"ADVISORY",
                    "url":"https://nvd.nist.gov/vuln/detail/CVE-2020-28493"
                },
                {
                    "type":"WEB",
                    "url":"https://github.com/pallets/jinja/pull/1343"
                },
                {
                    "type":"PACKAGE",
                    "url":"https://github.com/pallets/jinja"
                },
                {
                    "type":"WEB",
                    "url":"https://github.com/pallets/jinja/blob/ab81fd9c277900c85da0c322a2ff9d68a235b2e6/src/jinja2/utils.py%23L20"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.fedoraproject.org/archives/list/[email protected]/message/PVAKCOO7VBVUBM3Q6CBBTPBFNP5NDXF4/"
                },
                {
                    "type":"WEB",
                    "url":"https://security.gentoo.org/glsa/202107-19"
                },
                {
                    "type":"WEB",
                    "url":"https://snyk.io/vuln/SNYK-PYTHON-JINJA2-1012994"
                }
            ],
            "affected":[
                {
                    "package":{
                        "name":"jinja2",
                        "ecosystem":"PyPI",
                        "purl":"pkg:pypi/jinja2"
                    },
                    "ranges":[
                        {
                            "type":"ECOSYSTEM",
                            "events":[
                                {
                                    "introduced":"0"
                                },
                                {
                                    "fixed":"2.11.3"
                                }
                            ]
                        }
                    ],
                    "versions":[
                        "2.0",
                        "2.0rc1",
                        "2.1",
                        "2.1.1",
                        "2.10",
                        "2.10.1",
                        "2.10.2",
                        "2.10.3",
                        "2.11.0",
                        "2.11.1",
                        "2.11.2",
                        "2.2",
                        "2.2.1",
                        "2.3",
                        "2.3.1",
                        "2.4",
                        "2.4.1",
                        "2.5",
                        "2.5.1",
                        "2.5.2",
                        "2.5.3",
                        "2.5.4",
                        "2.5.5",
                        "2.6",
                        "2.7",
                        "2.7.1",
                        "2.7.2",
                        "2.7.3",
                        "2.8",
                        "2.8.1",
                        "2.9",
                        "2.9.1",
                        "2.9.2",
                        "2.9.3",
                        "2.9.4",
                        "2.9.5",
                        "2.9.6"
                    ],
                    "database_specific":{
                        "source":"https://github.com/github/advisory-database/blob/main/advisories/github-reviewed/2021/03/GHSA-g3rq-g295-4j3m/GHSA-g3rq-g295-4j3m.json"
                    }
                }
            ],
            "schema_version":"1.3.0",
            "severity":[
                {
                    "type":"CVSS_V3",
                    "score":"CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L"
                }
            ]
        },
        {
            "id":"GHSA-hj2j-77xm-mc5v",
            "summary":"High severity vulnerability that affects Jinja2",
            "details":"In Pallets Jinja before 2.8.1, str.format allows a sandbox escape.",
            "aliases":[
                "CVE-2016-10745"
            ],
            "modified":"2022-12-14T04:56:03.128921Z",
            "published":"2019-04-10T14:30:13Z",
            "database_specific":{
                "cwe_ids":[
                    "CWE-134"
                ],
                "severity":"HIGH",
                "github_reviewed":true
            },
            "references":[
                {
                    "type":"ADVISORY",
                    "url":"https://nvd.nist.gov/vuln/detail/CVE-2016-10745"
                },
                {
                    "type":"WEB",
                    "url":"https://github.com/pallets/jinja/commit/9b53045c34e61013dc8f09b7e52a555fa16bed16"
                },
                {
                    "type":"WEB",
                    "url":"https://access.redhat.com/errata/RHSA-2019:1022"
                },
                {
                    "type":"WEB",
                    "url":"https://access.redhat.com/errata/RHSA-2019:1237"
                },
                {
                    "type":"WEB",
                    "url":"https://access.redhat.com/errata/RHSA-2019:1260"
                },
                {
                    "type":"WEB",
                    "url":"https://access.redhat.com/errata/RHSA-2019:3964"
                },
                {
                    "type":"WEB",
                    "url":"https://access.redhat.com/errata/RHSA-2019:4062"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://github.com/advisories/GHSA-hj2j-77xm-mc5v"
                },
                {
                    "type":"PACKAGE",
                    "url":"https://github.com/pallets/jinja"
                },
                {
                    "type":"WEB",
                    "url":"https://palletsprojects.com/blog/jinja-281-released/"
                },
                {
                    "type":"WEB",
                    "url":"https://usn.ubuntu.com/4011-1/"
                },
                {
                    "type":"WEB",
                    "url":"https://usn.ubuntu.com/4011-2/"
                },
                {
                    "type":"WEB",
                    "url":"http://lists.opensuse.org/opensuse-security-announce/2019-05/msg00030.html"
                },
                {
                    "type":"WEB",
                    "url":"http://lists.opensuse.org/opensuse-security-announce/2019-06/msg00064.html"
                }
            ],
            "affected":[
                {
                    "package":{
                        "name":"jinja2",
                        "ecosystem":"PyPI",
                        "purl":"pkg:pypi/jinja2"
                    },
                    "ranges":[
                        {
                            "type":"ECOSYSTEM",
                            "events":[
                                {
                                    "introduced":"0"
                                },
                                {
                                    "fixed":"2.8.1"
                                }
                            ]
                        }
                    ],
                    "versions":[
                        "2.0",
                        "2.0rc1",
                        "2.1",
                        "2.1.1",
                        "2.2",
                        "2.2.1",
                        "2.3",
                        "2.3.1",
                        "2.4",
                        "2.4.1",
                        "2.5",
                        "2.5.1",
                        "2.5.2",
                        "2.5.3",
                        "2.5.4",
                        "2.5.5",
                        "2.6",
                        "2.7",
                        "2.7.1",
                        "2.7.2",
                        "2.7.3",
                        "2.8"
                    ],
                    "database_specific":{
                        "source":"https://github.com/github/advisory-database/blob/main/advisories/github-reviewed/2019/04/GHSA-hj2j-77xm-mc5v/GHSA-hj2j-77xm-mc5v.json"
                    }
                }
            ],
            "schema_version":"1.3.0",
            "severity":[
                {
                    "type":"CVSS_V3",
                    "score":"CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:N/A:N"
                }
            ]
        },
        {
            "id":"PYSEC-2014-8",
            "details":"The default configuration for bccache.FileSystemBytecodeCache in Jinja2 before 2.7.2 does not properly create temporary files, which allows local users to gain privileges via a crafted .cache file with a name starting with __jinja2_ in /tmp.",
            "aliases":[
                "CVE-2014-1402",
                "GHSA-8r7q-cvjq-x353"
            ],
            "modified":"2021-07-05T00:01:22.043149Z",
            "published":"2014-05-19T14:55:00Z",
            "references":[
                {
                    "type":"WEB",
                    "url":"https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=734747"
                },
                {
                    "type":"ADVISORY",
                    "url":"http://advisories.mageia.org/MGASA-2014-0028.html"
                },
                {
                    "type":"WEB",
                    "url":"http://jinja.pocoo.org/docs/changelog/"
                },
                {
                    "type":"WEB",
                    "url":"http://openwall.com/lists/oss-security/2014/01/10/3"
                },
                {
                    "type":"REPORT",
                    "url":"https://bugzilla.redhat.com/show_bug.cgi?id=1051421"
                },
                {
                    "type":"ADVISORY",
                    "url":"http://www.mandriva.com/security/advisories?name=MDVSA-2014:096"
                },
                {
                    "type":"WEB",
                    "url":"http://openwall.com/lists/oss-security/2014/01/10/2"
                },
                {
                    "type":"ADVISORY",
                    "url":"http://secunia.com/advisories/59017"
                },
                {
                    "type":"ADVISORY",
                    "url":"http://secunia.com/advisories/58918"
                },
                {
                    "type":"ADVISORY",
                    "url":"http://secunia.com/advisories/60770"
                },
                {
                    "type":"ADVISORY",
                    "url":"http://secunia.com/advisories/60738"
                },
                {
                    "type":"ADVISORY",
                    "url":"http://secunia.com/advisories/56287"
                },
                {
                    "type":"WEB",
                    "url":"http://www.gentoo.org/security/en/glsa/glsa-201408-13.xml"
                },
                {
                    "type":"WEB",
                    "url":"https://oss.oracle.com/pipermail/el-errata/2014-June/004192.html"
                },
                {
                    "type":"ADVISORY",
                    "url":"http://secunia.com/advisories/58783"
                },
                {
                    "type":"ADVISORY",
                    "url":"http://rhn.redhat.com/errata/RHSA-2014-0748.html"
                },
                {
                    "type":"ADVISORY",
                    "url":"http://rhn.redhat.com/errata/RHSA-2014-0747.html"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://github.com/advisories/GHSA-8r7q-cvjq-x353"
                }
            ],
            "affected":[
                {
                    "package":{
                        "name":"jinja2",
                        "ecosystem":"PyPI",
                        "purl":"pkg:pypi/jinja2"
                    },
                    "ranges":[
                        {
                            "type":"ECOSYSTEM",
                            "events":[
                                {
                                    "introduced":"0"
                                },
                                {
                                    "fixed":"2.7.2"
                                }
                            ]
                        }
                    ],
                    "versions":[
                        "2.0",
                        "2.0rc1",
                        "2.1",
                        "2.1.1",
                        "2.2",
                        "2.2.1",
                        "2.3",
                        "2.3.1",
                        "2.4",
                        "2.4.1",
                        "2.5",
                        "2.5.1",
                        "2.5.2",
                        "2.5.3",
                        "2.5.4",
                        "2.5.5",
                        "2.6",
                        "2.7",
                        "2.7.1"
                    ],
                    "database_specific":{
                        "source":"https://github.com/pypa/advisory-database/blob/main/vulns/jinja2/PYSEC-2014-8.yaml"
                    }
                }
            ],
            "schema_version":"1.3.0"
        },
        {
            "id":"PYSEC-2014-82",
            "details":"FileSystemBytecodeCache in Jinja2 2.7.2 does not properly create temporary directories, which allows local users to gain privileges by pre-creating a temporary directory with a user's uid.  NOTE: this vulnerability exists because of an incomplete fix for CVE-2014-1402.",
            "aliases":[
                "CVE-2014-0012"
            ],
            "modified":"2021-08-27T03:22:05.027573Z",
            "published":"2014-05-19T14:55:00Z",
            "references":[
                {
                    "type":"REPORT",
                    "url":"https://bugzilla.redhat.com/show_bug.cgi?id=1051421"
                },
                {
                    "type":"WEB",
                    "url":"https://github.com/mitsuhiko/jinja2/pull/292"
                },
                {
                    "type":"FIX",
                    "url":"https://github.com/mitsuhiko/jinja2/commit/acb672b6a179567632e032f547582f30fa2f4aa7"
                },
                {
                    "type":"WEB",
                    "url":"http://seclists.org/oss-sec/2014/q1/73"
                },
                {
                    "type":"WEB",
                    "url":"https://github.com/mitsuhiko/jinja2/pull/296"
                },
                {
                    "type":"ADVISORY",
                    "url":"http://secunia.com/advisories/60738"
                },
                {
                    "type":"WEB",
                    "url":"http://www.gentoo.org/security/en/glsa/glsa-201408-13.xml"
                },
                {
                    "type":"ADVISORY",
                    "url":"http://secunia.com/advisories/56328"
                }
            ],
            "affected":[
                {
                    "package":{
                        "name":"jinja2",
                        "ecosystem":"PyPI",
                        "purl":"pkg:pypi/jinja2"
                    },
                    "ranges":[
                        {
                            "type":"GIT",
                            "repo":"https://github.com/mitsuhiko/jinja2",
                            "events":[
                                {
                                    "introduced":"0"
                                },
                                {
                                    "fixed":"acb672b6a179567632e032f547582f30fa2f4aa7"
                                }
                            ]
                        },
                        {
                            "type":"ECOSYSTEM",
                            "events":[
                                {
                                    "introduced":"0"
                                },
                                {
                                    "fixed":"2.7.3"
                                }
                            ]
                        }
                    ],
                    "versions":[
                        "2.0",
                        "2.0rc1",
                        "2.1",
                        "2.1.1",
                        "2.2",
                        "2.2.1",
                        "2.3",
                        "2.3.1",
                        "2.4",
                        "2.4.1",
                        "2.5",
                        "2.5.1",
                        "2.5.2",
                        "2.5.3",
                        "2.5.4",
                        "2.5.5",
                        "2.6",
                        "2.7",
                        "2.7.1",
                        "2.7.2"
                    ],
                    "database_specific":{
                        "source":"https://github.com/pypa/advisory-database/blob/main/vulns/jinja2/PYSEC-2014-82.yaml"
                    }
                }
            ],
            "schema_version":"1.3.0"
        },
        {
            "id":"PYSEC-2019-217",
            "details":"In Pallets Jinja before 2.10.1, str.format_map allows a sandbox escape.",
            "aliases":[
                "CVE-2019-10906",
                "GHSA-462w-v97r-4m45"
            ],
            "modified":"2021-11-22T04:57:52.862665Z",
            "published":"2019-04-07T00:29:00Z",
            "references":[
                {
                    "type":"ARTICLE",
                    "url":"https://palletsprojects.com/blog/jinja-2-10-1-released"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/b2380d147b508bbcb90d2cad443c159e63e12555966ab4f320ee22da@%3Ccommits.airflow.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/46c055e173b52d599c648a98199972dbd6a89d2b4c4647b0500f2284@%3Cdevnull.infra.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/f0c4a03418bcfe70c539c5dbaf99c04c98da13bfa1d3266f08564316@%3Ccommits.airflow.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/7f39f01392d320dfb48e4901db68daeece62fd60ef20955966739993@%3Ccommits.airflow.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/57673a78c4d5c870d3f21465c7e2946b9f8285c7c57e54c2ae552f02@%3Ccommits.airflow.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/320441dccbd9a545320f5f07306d711d4bbd31ba43dc9eebcfc602df@%3Cdevnull.infra.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/2b52b9c8b9d6366a4f1b407a8bde6af28d9fc73fdb3b37695fd0d9ac@%3Cdevnull.infra.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.apache.org/thread.html/09fc842ff444cd43d9d4c510756fec625ef8eb1175f14fd21de2605f@%3Cdevnull.infra.apache.org%3E"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.fedoraproject.org/archives/list/[email protected]/message/QCDYIS254EJMBNWOG4S5QY6AOTOR4TZU/"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.fedoraproject.org/archives/list/[email protected]/message/DSW3QZMFVVR7YE3UT4YRQA272TYAL5AF/"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.fedoraproject.org/archives/list/[email protected]/message/TS7IVZAJBWOHNRDMFJDIZVFCMRP6YIUQ/"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://access.redhat.com/errata/RHSA-2019:1152"
                },
                {
                    "type":"WEB",
                    "url":"http://lists.opensuse.org/opensuse-security-announce/2019-05/msg00030.html"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://access.redhat.com/errata/RHSA-2019:1237"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://access.redhat.com/errata/RHSA-2019:1329"
                },
                {
                    "type":"WEB",
                    "url":"https://usn.ubuntu.com/4011-1/"
                },
                {
                    "type":"WEB",
                    "url":"https://usn.ubuntu.com/4011-2/"
                },
                {
                    "type":"WEB",
                    "url":"http://lists.opensuse.org/opensuse-security-announce/2019-06/msg00064.html"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://github.com/advisories/GHSA-462w-v97r-4m45"
                }
            ],
            "affected":[
                {
                    "package":{
                        "name":"jinja2",
                        "ecosystem":"PyPI",
                        "purl":"pkg:pypi/jinja2"
                    },
                    "ranges":[
                        {
                            "type":"ECOSYSTEM",
                            "events":[
                                {
                                    "introduced":"0"
                                },
                                {
                                    "fixed":"2.10.1"
                                }
                            ]
                        }
                    ],
                    "versions":[
                        "2.0",
                        "2.0rc1",
                        "2.1",
                        "2.1.1",
                        "2.10",
                        "2.2",
                        "2.2.1",
                        "2.3",
                        "2.3.1",
                        "2.4",
                        "2.4.1",
                        "2.5",
                        "2.5.1",
                        "2.5.2",
                        "2.5.3",
                        "2.5.4",
                        "2.5.5",
                        "2.6",
                        "2.7",
                        "2.7.1",
                        "2.7.2",
                        "2.7.3",
                        "2.8",
                        "2.8.1",
                        "2.9",
                        "2.9.1",
                        "2.9.2",
                        "2.9.3",
                        "2.9.4",
                        "2.9.5",
                        "2.9.6"
                    ],
                    "database_specific":{
                        "source":"https://github.com/pypa/advisory-database/blob/main/vulns/jinja2/PYSEC-2019-217.yaml"
                    }
                }
            ],
            "schema_version":"1.3.0"
        },
        {
            "id":"PYSEC-2019-220",
            "details":"In Pallets Jinja before 2.8.1, str.format allows a sandbox escape.",
            "aliases":[
                "CVE-2016-10745",
                "GHSA-hj2j-77xm-mc5v"
            ],
            "modified":"2021-11-22T04:57:52.929678Z",
            "published":"2019-04-08T13:29:00Z",
            "references":[
                {
                    "type":"ARTICLE",
                    "url":"https://palletsprojects.com/blog/jinja-281-released/"
                },
                {
                    "type":"FIX",
                    "url":"https://github.com/pallets/jinja/commit/9b53045c34e61013dc8f09b7e52a555fa16bed16"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://access.redhat.com/errata/RHSA-2019:1022"
                },
                {
                    "type":"WEB",
                    "url":"http://lists.opensuse.org/opensuse-security-announce/2019-05/msg00030.html"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://access.redhat.com/errata/RHSA-2019:1237"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://access.redhat.com/errata/RHSA-2019:1260"
                },
                {
                    "type":"WEB",
                    "url":"https://usn.ubuntu.com/4011-1/"
                },
                {
                    "type":"WEB",
                    "url":"https://usn.ubuntu.com/4011-2/"
                },
                {
                    "type":"WEB",
                    "url":"http://lists.opensuse.org/opensuse-security-announce/2019-06/msg00064.html"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://access.redhat.com/errata/RHSA-2019:3964"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://access.redhat.com/errata/RHSA-2019:4062"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://github.com/advisories/GHSA-hj2j-77xm-mc5v"
                }
            ],
            "affected":[
                {
                    "package":{
                        "name":"jinja2",
                        "ecosystem":"PyPI",
                        "purl":"pkg:pypi/jinja2"
                    },
                    "ranges":[
                        {
                            "type":"GIT",
                            "repo":"https://github.com/pallets/jinja",
                            "events":[
                                {
                                    "introduced":"0"
                                },
                                {
                                    "fixed":"9b53045c34e61013dc8f09b7e52a555fa16bed16"
                                }
                            ]
                        },
                        {
                            "type":"ECOSYSTEM",
                            "events":[
                                {
                                    "introduced":"0"
                                },
                                {
                                    "fixed":"2.8.1"
                                }
                            ]
                        }
                    ],
                    "versions":[
                        "2.0",
                        "2.0rc1",
                        "2.1",
                        "2.1.1",
                        "2.2",
                        "2.2.1",
                        "2.3",
                        "2.3.1",
                        "2.4",
                        "2.4.1",
                        "2.5",
                        "2.5.1",
                        "2.5.2",
                        "2.5.3",
                        "2.5.4",
                        "2.5.5",
                        "2.6",
                        "2.7",
                        "2.7.1",
                        "2.7.2",
                        "2.7.3",
                        "2.8"
                    ],
                    "database_specific":{
                        "source":"https://github.com/pypa/advisory-database/blob/main/vulns/jinja2/PYSEC-2019-220.yaml"
                    }
                }
            ],
            "schema_version":"1.3.0"
        },
        {
            "id":"PYSEC-2021-66",
            "details":"This affects the package jinja2 from 0.0.0 and before 2.11.3. The ReDoS vulnerability is mainly due to the `_punctuation_re regex` operator and its use of multiple wildcards. The last wildcard is the most exploitable as it searches for trailing punctuation. This issue can be mitigated by Markdown to format user content instead of the urlize filter, or by implementing request timeouts and limiting process memory.",
            "aliases":[
                "CVE-2020-28493",
                "SNYK-PYTHON-JINJA2-1012994",
                "GHSA-g3rq-g295-4j3m"
            ],
            "modified":"2021-03-22T16:34:00Z",
            "published":"2021-02-01T20:15:00Z",
            "references":[
                {
                    "type":"WEB",
                    "url":"https://github.com/pallets/jinja/blob/ab81fd9c277900c85da0c322a2ff9d68a235b2e6/src/jinja2/utils.py%23L20"
                },
                {
                    "type":"WEB",
                    "url":"https://github.com/pallets/jinja/pull/1343"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://snyk.io/vuln/SNYK-PYTHON-JINJA2-1012994"
                },
                {
                    "type":"WEB",
                    "url":"https://lists.fedoraproject.org/archives/list/[email protected]/message/PVAKCOO7VBVUBM3Q6CBBTPBFNP5NDXF4/"
                },
                {
                    "type":"ADVISORY",
                    "url":"https://github.com/advisories/GHSA-g3rq-g295-4j3m"
                }
            ],
            "affected":[
                {
                    "package":{
                        "name":"jinja2",
                        "ecosystem":"PyPI",
                        "purl":"pkg:pypi/jinja2"
                    },
                    "ranges":[
                        {
                            "type":"ECOSYSTEM",
                            "events":[
                                {
                                    "introduced":"0"
                                },
                                {
                                    "fixed":"2.11.3"
                                }
                            ]
                        }
                    ],
                    "versions":[
                        "2.0rc1",
                        "2.0",
                        "2.1",
                        "2.1.1",
                        "2.2",
                        "2.2.1",
                        "2.3",
                        "2.3.1",
                        "2.4",
                        "2.4.1",
                        "2.5",
                        "2.5.1",
                        "2.5.2",
                        "2.5.3",
                        "2.5.4",
                        "2.5.5",
                        "2.6",
                        "2.7",
                        "2.7.1",
                        "2.7.2",
                        "2.7.3",
                        "2.8",
                        "2.8.1",
                        "2.9",
                        "2.9.1",
                        "2.9.2",
                        "2.9.3",
                        "2.9.4",
                        "2.9.5",
                        "2.9.6",
                        "2.10",
                        "2.10.1",
                        "2.10.2",
                        "2.10.3",
                        "2.11.0",
                        "2.11.1",
                        "2.11.2"
                    ],
                    "database_specific":{
                        "source":"https://github.com/pypa/advisory-database/blob/main/vulns/jinja2/PYSEC-2021-66.yaml"
                    }
                }
            ],
            "schema_version":"1.3.0"
        }
    ]
}

这个API是我们主要用的API里面有一些依赖关系等等。第三个API是根据OSV_ID查询，因为我们不知道osv_id，所以用处不大。然后这380多个项目的列表在 https://github.com/google/oss-fuzz/tree/master/projects 中，他们的原理是用Google的漏洞扫描工具进行自动化扫描然后存储在数据库中，其中Google的漏洞扫描工具是什么样的原理需要有感兴趣的同学进行了解。

这些数据可以让我们了解每个项目中的漏洞有什么，如何自动化的识别漏洞，目前没有看到依赖网络相关数据，这部分不清楚是否有相关的同学了解。

[bifenglin]

在爬取漏洞信息的时候遇见了问题：
根据API https://osv.dev/docs/#tag/api/operation/OSV_QueryAffectedBatch 的内容要求需求2个参数：

1. 指定版本号或者commit hash号
2. 包名（如果给了commit hash号就不需要写包名）

然后根据包名获取github的commit hash号，尝试了很多commit hash号 都没有数据库都没有这个commit的数据。
那么需要知道每个项目的版本号，不知道有什么好的方法？

@frank-zsy

[TieWay59]

我想补充一下，根据 OSV.dev 社区维护者的一个评论，我观察到他们网站是有开源的 data dump 的，所以可以尝试看看能不能绕过爬数据的这个步骤。

google/osv.dev#448 (comment)

[will-ww]

我想补充一下，根据 OSV.dev 社区维护者的一个评论，我观察到他们网站是有开源的 data dump 的，所以可以尝试看看能不能绕过爬数据的这个步骤。

google/osv.dev#448 (comment)

是的，可以关注~ @bifenglin

[TieWay59]

更新一下现状：

官方数据dump，这些来源被汇总并持续导出到 OSV 维护的一个 GCS（Google 云存储）桶中：gs://osv-vulnerabilities。这个桶包含了每个生态系统的单独条目，格式为 gs://osv-vulnerabilities//.json，以及 zip 文件，包含了每个生态系统的所有漏洞信息，位于 gs://osv-vulnerabilities//all.zip。例如，对于 PyPI（Python 包索引）的漏洞可以通过 https://osv-vulnerabilities.storage.googleapis.com/PyPI/all.zip 的外部链接下载。

所有当前的列表可以在 https://osv-vulnerabilities.storage.googleapis.com/ecosystems.txt 找到。得到的数据是一系列json文档。

我需要解析这些json文档导入到我们 rm-uf6f53dpu5lac389m8o.mysql.rds.aliyuncs.com 的 open-tag 关系数据库中。

手工解析导出的json文档还有一定难度，我决定深入阅读一下 OSV 仓库的代码，可能会看到数据导入导出的信息，避免重复造轮子。

[will-ww]

有一定难度，我决定深入阅读一下 OSV 仓库的代码，可能会看到数据导入导出的信息，避免重复造轮

Many thanks~

[TieWay59]

更新：

根据 osv 仓库中 model 的代码¹，可以看出他们的数据库客户端采用的是谷歌云存储的 ndb 客户端库 ²，也就是说他们数据原本就是按照json文档的形式存储在 No-SQL 的数据库里的。所以我们要解析出SQL数据库只能手动去操作了。

他们数据的 schema 参考这个仓库的文档： https://github.com/ossf/osv-schema/blob/main/docs/schema.md

Footnotes

1. https://github.com/google/osv.dev/blob/98e5f9798b251cfc86e9ad97f72c8ebb38bc1a24/osv/models.py#L21-L58 ↩

2. ndb 是一个用于 Google Cloud Datastore 的客户端库。Google Cloud Datastore 是一个 NoSQL 文档数据库，它可以自动扩缩容，提供高性能和易用的应用开发体验。它的存储模型是基于实体（entity）和键（key）的，每个实体都有一个唯一的键，以及一组属性（property）和值（value）。 ↩

[bifenglin]

感谢 @TieWay59 同学的工作，解决了dump全量数据的问题，根据分析得到数据量较少，为了以后展示与查询，使用结构化数据库，建立结构化表来进行存储，后面再同步到clickhouse、maxcompute数据库，采用数据库为mysql数据库，建立的表为：

• vulnerability
• vulnerability_affected
• vulnerability_affected_range
• vulnerability_affected_version
• vulnerability_reference

相关表的说明可参照 https://github.com/ossf/osv-schema/blob/main/docs/schema.md ，其中aliases 字段cwe_ids字段属于数组，目前设计将这两个字段放入vulnerability表当成这个对象的两个属性，用,隔开存储，原因是根据大量数据可以看出这两个字段大多数都是一条记录，避免了一定的表关联复杂度。