Skip to content

Commit

Permalink
doc: 更新制品分析文档 #2757
Browse files Browse the repository at this point in the history
  • Loading branch information
@cnlkl
cnlkl authored Nov 18, 2024
1 parent 1b264d0 commit a997b6e
Show file tree
Hide file tree
Showing 4 changed files with 161 additions and 69 deletions.
25 changes: 18 additions & 7 deletions docs/analyst/overview.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

制品分析功能主要由`analyst`和`analysis-executor`两个服务构成

`analyst`服务负责管理扫描器、扫描任务、扫描报告存取
`analyst`服务负责管理扫描器、任务执行集群、扫描任务、扫描报告存取

`analysis-executor`是实际执行扫描任务的服务，通过`analyst`服务创建的任务最终都将由`analysis-executor`执行，
执行完后再将扫描结果上报到`analyst`服务
Expand Down Expand Up @@ -43,13 +43,12 @@

子扫描任务创建后会保存在数据库的任务队列中，如果有其他任务队列实现也会被加入到对应的队列中

1. 子任务刚创建时处于CREATED状态
2. 子任务被主动拉取时处于PULLED状态
3. 子任务加入扫描任务队列后处于ENQUEUED状态
4. 扫描执行器开始执行任务后子任务处于EXECUTING状态
5. 扫描结束上报结果后子任务从数据库的队列中移除
1. 子任务刚创建时处于CREATED状态，如果任务数超过项目任务配额将处于BLOCKED状态
2. 子任务被主动拉取时处于PULLED状态，此时可能尚未下发到执行集群
3. 扫描执行器开始执行任务后子任务后会上报状态，此时更新子任务状态为EXECUTING
4. 扫描结束上报结果后子任务从数据库的队列中移除

会定时查询数据库中的子扫描任务队列，将CREATED或者处于PULLED、ENQUEUED、EXECUTING这三个状态很久的任务重新提交执行，
会定时查询数据库中的子扫描任务队列，将CREATED或者处于PULLED、EXECUTING这两个状态过久的任务重新提交执行，
一个子扫描任务最多执行次数有限制，超过限制后会被从数据库中的扫描任务队列移除，不再重试

## 扫描结果
Expand All @@ -59,3 +58,15 @@
类似漏洞数量、敏感信息数量这种统计类型数据会存储到通用的扫描结果表中

特定类型扫描器特有的扫描结果会根据不同的扫描器实现进行存取，比如目前实现的arrowhead扫描器结果存储在单独的数据库中

## 相关Node元数据

扫描过程中会将扫描任务状态更新到制品元数据中，key为`scanStatus`，value可选值如下

INIT：等待扫描
RUNNING: 扫描中
STOP：扫描中止
UN_QUALITY：未设置质量规则
QUALITY_PASS：质量规则通过
FAILED：扫描异常
QUALITY_UNPASS：质量规则未通过
53 changes: 20 additions & 33 deletions docs/apidoc/scanner/report.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,14 @@
"https://msrc-blog.microsoft.com/2021/12/11/microsofts-response-to-cve-2021-44228-apache-log4j2/",
"https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-apache-log4j-qRuKNEbd"
],
"path": "/97eef8b72e121347074c8b3062b010170187a6fa7375555fd1ed68540adaea1f.jar"
"versionsPaths": [
{
"version": "2.14.1",
"paths": [
"/97eef8b72e121347074c8b3062b010170187a6fa7375555fd1ed68540adaea1f.jar"
]
}
]
}
],
"page": 1,
Expand All @@ -275,18 +282,18 @@

data字段说明

| 字段 | 类型 | 说明 | Description |
|-------------------|--------|--------|-------------------|
| vulId | string | 漏洞id | vul id |
| severity | string | 漏洞等级 | vul severity |
| pkgName | string | 所属依赖 | dependency |
| installedVersion | array | 使用的版本 | installed version |
| title | string | 漏洞标题 | vul title |
| vulnerabilityName | string | 漏洞名 | vul name |
| description | string | 漏洞描述 | description |
| officialSolution | string | 官方解决方案 | official solution |
| reference | array | 关联引用 | reference |
| path | string | 漏洞文件路径 | vul path |
| 字段 | 类型 | 说明 | Description |
|-------------------|--------|-----------------|-------------------|
| vulId | string | 漏洞id | vul id |
| severity | string | 漏洞等级 | vul severity |
| pkgName | string | 所属依赖 | dependency |
| installedVersion | array | 使用的版本 | installed version |
| title | string | 漏洞标题 | vul title |
| vulnerabilityName | string | 漏洞名 | vul name |
| description | string | 漏洞描述 | description |
| officialSolution | string | 官方解决方案 | official solution |
| reference | array | 关联引用 | reference |
| versionsPaths | array | 存在漏洞的制品各个版本所在路径 | vul path |

响应体参考[分页接口响应格式](../common/common.md?id=统一分页接口响应格式)

Expand All @@ -313,23 +320,3 @@ data字段说明
- 响应体

响应体参考[获取子任务扫描报告详情](./report.md?id=获取子任务扫描报告详情)

## 获取属于方案的子任务信息

- API: GET /analyst/api/scan/artifact/count/{projectId}/{subScanTaskId}
- API 名称: get_plan_subtask_report_detail
- 功能说明：
- 中文：获取属于方案的扫描子任务信息
- English：get scan plan subtask
- 请求体 此接口请求体为空

- 请求字段说明

| 字段 | 类型 | 是否必须 | 默认值 | 说明 | Description |
|---------------|--------|------|-----|-------|-------------|
| projectId | string | 是 | 无 | 项目id | project id |
| subScanTaskId | string | 是 | 无 | 子任务id | project id |

- 响应体

响应体参考[获取扫描子任务](./scan.md?id=获取扫描子任务)
66 changes: 55 additions & 11 deletions docs/apidoc/scanner/scan.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,61 @@

扫描结果预览字段参考[获取扫描报告预览](./report.md?id=获取扫描报告预览)

## 创建跨项目扫描任务

- API: POST /analyst/api/scan/global
- API 名称: global scan
- 功能说明：
- 中文：跨项目扫描
- English：global scan
- 请求体

```json
{
"scanner": "scanner",
"rule": {
"relation": "AND",
"rules": [
{
"field": "repoName",
"value": "maven-local",
"operation": "EQ"
},
{
"field": "fullPath",
"value": "/",
"operation": "PREFIX"
}
]
},
"projectMetadata": [
{
"key": "bg",
"value": "test"
}
],
"metadata": [
{
"key": "buildNumber",
"value": "32"
}
]
}
```

- 请求字段说明

| 字段 | 类型 | 是否必须 | 默认值 | 说明 | Description |
|-----------------|---------|------|-------|------------------------------------------------------------|--------------------|
| scanner | string | 否 | 无 | 要获取的报告使用的扫描器名称，扫描器名称在扫描器注册到制品库后确定，需要联系制品库管理员确认 | scanner name |
| force | boolean | 否 | false | 是否强制扫描，为true时即使文件已扫描过也会再次执行扫描 | force scan |
| rule | object | 否 | 无 | 要扫描的文件匹配规则，参考[自定义搜索接口公共说明](../common/search.md?id=自定义搜索协议) | file match rule |
| projectMetadata | array | 否 | 无 | 指定项目元数据用于筛选需要扫描的项目 | scan task metadata |
| metadata | array | 否 | 无 | 为扫描任务附加元数据，用于标识扫描任务 | scan task metadata |

- 响应体

响应体参考[创建扫描任务](./scan.md?id=创建扫描任务)响应体

## 通过流水线创建扫描任务

Expand Down Expand Up @@ -226,17 +281,6 @@

- 响应体

```json
{
"code": 0,
"message": null,
"data": {},
"traceId": ""
}
```

- data字段说明

响应体参考[创建扫描任务](./scan.md?id=创建扫描任务)响应体

## 停止扫描
Expand Down
86 changes: 68 additions & 18 deletions docs/apidoc/scanner/scanner.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@
{
"name": "arrowhead",
"image": "example.com/example/scanner:1.0",
"dockerRegistryUsername": "xxx",
"dockerRegistryPassword": "xxx",
"cmd": "scan",
"version": "1.0",
"args": [
Expand All @@ -32,26 +34,44 @@
"maxScanDurationPerMb": 6000,
"supportFileNameExt": ["tar", "apk", "ipa", "jar"],
"supportPackageTypes": ["DOCKER", "GENERIC", "MAVEN"],
"supportScanTypes": ["SECURITY", "LICENSE"]
"supportScanTypes": ["SECURITY", "LICENSE"],
"supportDispatchers": ["k8s-1", "k8s-2"],
"limitMem": 34359738368,
"requestMem": 17179869184,
"requestStorage": 17179869184,
"limitStorage": 137438953472,
"requestCpu": 4.0,
"limitCpu": 16.0,
"unsupportedArtifactNameRegex": [".*\\.jar"]
}
```

- 请求字段说明

| 字段 | 类型 | 是否必须 | 默认值 | 说明 | Description |
|----------------------|---------|------|-------|------------------------------------------------|------------------------------|
| name | string | 是 | 无 | 扫描器名 | scanner name |
| image | string | 是 | 无 | 扫描器镜像 | scanner image |
| cmd | string | 是 | 无 | 扫描器启动命令，扫描器镜像不需要设置entrypoint，而是制品库启动扫描器时候设置cmd | scanner cmd |
| version | string | 是 | 无 | 扫描器版本 | scanner version |
| type | string | 是 | 无 | 扫描器类型，固定为standard | scanner type |
| description | string | 是 | 无 | 扫描器描述 | scanner description |
| rootPath | string | 是 | 无 | 扫描器工作根目录 | scanner work dir |
| cleanWorkDir | boolean | 否 | true | 扫描结束后是否清理目录 | clean work dir after scan |
| maxScanDurationPerMb | number | 否 | 6000 | 每MB文件最大允许的扫描时间 | max scan duration per mb |
| supportFileNameExt | array | 否 | empty | 支持扫描的文件名后缀 | support file name extensions |
| supportPackageTypes | array | 否 | empty | 支持扫描的包类型 | support package types |
| supportScanTypes | array | 否 | empty | 支持扫描的类型 | support scan types |
| 字段 | 类型 | 是否必须 | 默认值 | 说明 | Description |
|------------------------------|---------|------|--------------|------------------------------------------------|--------------------------------------|
| name | string | 是 | 无 | 扫描器名 | scanner name |
| image | string | 是 | 无 | 扫描器镜像 | scanner image |
| dockerRegistryUsername | string | 否 | 无 | 扫描器镜像所在仓库用户名 | scanner image |
| dockerRegistryPassword | string | 否 | 无 | 扫描器镜像所在仓库密码 | scanner image |
| cmd | string | 是 | 无 | 扫描器启动命令，扫描器镜像不需要设置entrypoint，而是制品库启动扫描器时候设置cmd | scanner cmd |
| version | string | 是 | 无 | 扫描器版本 | scanner version |
| type | string | 是 | 无 | 扫描器类型，固定为standard | scanner type |
| description | string | 是 | 无 | 扫描器描述 | scanner description |
| rootPath | string | 是 | 无 | 扫描器工作根目录 | scanner work dir |
| cleanWorkDir | boolean | 否 | true | 扫描结束后是否清理目录 | clean work dir after scan |
| maxScanDurationPerMb | number | 否 | 6000 | 每MB文件最大允许的扫描时间 | max scan duration per mb |
| supportFileNameExt | array | 否 | empty | 支持扫描的文件名后缀 | support file name extensions |
| supportPackageTypes | array | 否 | empty | 支持扫描的包类型 | support package types |
| supportScanTypes | array | 否 | empty | 支持扫描的类型 | support scan types |
| supportDispatchers | array | 否 | empty | 支持运行的扫描执行集群 | support execution cluster dispatcher |
| limitMem | number | 否 | 34359738368 | 扫描容器limit mem | limit mem |
| requestMem | number | 否 | 17179869184 | 扫描容器request mem | request mem |
| requestStorage | number | 否 | 17179869184 | 扫描容器request ephemeralStorage | request ephemeral storage |
| limitStorage | number | 否 | 137438953472 | 扫描容器limit ephemeralStorage | limit ephemeral storage |
| requestCpu | number | 否 | 4.0 | 扫描容器request cpu | request cpu |
| limitCpu | number | 否 | 16.0 | 扫描容器limit cpu | limit cpu |
| unsupportedArtifactNameRegex | array | 否 | empty | 不支持的制品名称正则列表 | unsupported artifact name regex |

- 响应体

Expand All @@ -62,6 +82,8 @@
"data": {
"name": "arrowhead",
"image": "example.com/example/scanner:1.0",
"dockerRegistryUsername": "xxx",
"dockerRegistryPassword": "xxx",
"cmd": "scan",
"version": "1.0",
"args": [
Expand All @@ -79,7 +101,15 @@
"maxScanDurationPerMb": 6000,
"supportFileNameExt": ["tar", "apk", "ipa", "jar"],
"supportPackageTypes": ["DOCKER", "GENERIC", "MAVEN"],
"supportScanTypes": ["SECURITY", "LICENSE"]
"supportScanTypes": ["SECURITY", "LICENSE"],
"supportDispatchers": ["k8s-1", "k8s-2"],
"limitMem": 34359738368,
"requestMem": 17179869184,
"requestStorage": 17179869184,
"limitStorage": 137438953472,
"requestCpu": 4.0,
"limitCpu": 16.0,
"unsupportedArtifactNameRegex": [".*\\.jar"]
},
"traceId": ""
}
Expand Down Expand Up @@ -108,6 +138,8 @@
"data": {
"name": "arrowhead",
"image": "example.com/example/scanner:1.0",
"dockerRegistryUsername": "xxx",
"dockerRegistryPassword": "xxx",
"cmd": "scan",
"version": "1.0",
"args": [
Expand All @@ -125,7 +157,15 @@
"maxScanDurationPerMb": 6000,
"supportFileNameExt": ["tar", "apk", "ipa", "jar"],
"supportPackageTypes": ["DOCKER", "GENERIC", "MAVEN"],
"supportScanTypes": ["SECURITY", "LICENSE"]
"supportScanTypes": ["SECURITY", "LICENSE"],
"supportDispatchers": ["k8s-1", "k8s-2"],
"limitMem": 34359738368,
"requestMem": 17179869184,
"requestStorage": 17179869184,
"limitStorage": 137438953472,
"requestCpu": 4.0,
"limitCpu": 16.0,
"unsupportedArtifactNameRegex": [".*\\.jar"]
},
"traceId": ""
}
Expand Down Expand Up @@ -182,6 +222,8 @@
"data": {
"name": "arrowhead",
"image": "example.com/example/scanner:1.0",
"dockerRegistryUsername": "xxx",
"dockerRegistryPassword": "xxx",
"cmd": "scan",
"version": "1.0",
"args": [
Expand All @@ -199,7 +241,15 @@
"maxScanDurationPerMb": 6000,
"supportFileNameExt": ["tar", "apk", "ipa", "jar"],
"supportPackageTypes": ["DOCKER", "GENERIC", "MAVEN"],
"supportScanTypes": ["SECURITY", "LICENSE"]
"supportScanTypes": ["SECURITY", "LICENSE"],
"supportDispatchers": ["k8s-1", "k8s-2"],
"limitMem": 34359738368,
"requestMem": 17179869184,
"requestStorage": 17179869184,
"limitStorage": 137438953472,
"requestCpu": 4.0,
"limitCpu": 16.0,
"unsupportedArtifactNameRegex": [".*\\.jar"]
},
"traceId": ""
}
Expand Down

0 comments on commit a997b6e

Please sign in to comment.