Unable to import large SBOM into DT

[nigellh]

Hi,

I have a basic understanding of DT. I know how to get a containerised version version up and running. I have logged in, created an account, can create a project and have previously uploaded SBOMs to see what the vulnerability processing is like.

I have spent most of the day trying to get a 9.5MByte CycloneDX SBOM (validated against the schema) into DT. It has around 20,000 packages in it.

This is getting to be typical of some of the large products we work with where there can be 100-200 Docker Images.

I created the project and went to import the SBOM.

You can select the file, it says it has imported it and gives you a little green pop up message, but then nothing appears. I am aware that sometimes it can take a little while, but 15 hours is a little much!

Are their limitations to what DT can handle as an input?

I can see there have been discussions that large payloads might be a problem and https://docs.dependencytrack.org/usage/cicd/ refers to other methods to import them.

Specifically using

curl -X "PUT" "http://dtrack.example.com/api/v1/bom" \
     -H 'Content-Type: application/json' \
     -H 'X-API-Key: LPojpCDSsEd4V9Zi6qCWr4KsiF3Konze' \
     -d @payload.json

I have tried using this command, but used file:///etc as it is a local file.

I get no error messages and it just appears to write out to the screen. (Not used curl before so that could be normal)

I am also completely unaware of where to get the "X-API-Key"? Is it the key in the example and that is valid for all DT instances? (e.g. yes it is the one that I used.) Searched the DT Interface to see if there was anything obvious under settings so I could have missed it.

This command does seem odd in that it does not specify the project it is to go into.

From that page also tried

curl -X "PUT" "http://dtrack.example.com/api/v1/bom" \
     -H 'Content-Type: application/json' \
     -H 'X-API-Key: LPojpCDSsEd4V9Zi6qCWr4KsiF3Konze' \
     -d $'{
  "project": "f90934f5-cb88-47ce-81cb-db06fc67d4b4",
  "bom": "PD94bWwgdm..."
  }'

No errors and just seemed to write the SBOM to the screen. At least it specifies the project, just not sure about the bom reference at the end or, again, the X-API-Key.

I have searched DT Docs and internet for suggestions to get this to work, but getting stuck. So, very much out of my comfort zone, but always willing to try. Just requesting a little hand-holding please.

Any guidance appreciated!

[nigellh]

Do you know where I can get access to the logs when running from a container? Might show the problem in there.

[nscuro]

Are their limitations to what DT can handle as an input?

This will depend on the resources of the machine DT runs on. What hardware are you running on?

This is getting to be typical of some of the large products we work with where there can be 100-200 Docker Images.

Are you saying those 100-200 Docker images are all in the same SBOM?

You can select the file, it says it has imported it and gives you a little green pop up message, but then nothing appears. I am aware that sometimes it can take a little while, but 15 hours is a little much!

Uploaded BOMs are processed asynchronously in the background. As you can imagine, keeping the connection open while processing would be fairly inefficient. The idea is that the BOM upload response will contain a token. That token can then be used to query the API for the current processing status of the BOM. Mind you, the frontend does not do this. Instead, you'd refresh the page (or the tables / lists you're interested in).

Do you know where I can get access to the logs when running from a container? Might show the problem in there.

Logs are written to both standard output and /data/.dependendy-track/dependency-track.log within the container. If you used Docker Compose for the setup, docker compose logs -f dtrack-apiserver will be the easiest option.

[nigellh]

Broke the scan in to 3. One set of results is small and the other two large.

The small one imports in the standard way so confirms the basic process is working.

The 4.5Mbyte SBOM would not import into the same one as the small one, but would into its own project.

The 6.5Mbyte SBOM would not even import into its own project.

So a little bit of guess work is that this could be size limited in some way?

Just not sure where to look.

[nigellh]

Hi @nscuro

The system is a VM that has 16Gbytes RAM and plenty of disk space. It is running Ubuntu 20.04 and it is pretty much the only thing on there. I spun it up for these. Not been a problem in the past, but these are getting larger. Plenty of disk space left. Cannot remember if it is defined with 4 or 8 CPUs.

In this particular case, no there are not 100-200 docker images in this one, but there are in some and this is as big as one of those. There are around 20,000 unique open source packages listed in there and hence the 9.5Mbytes of SBOM size.

Our team do a lot of processing of products for the company and they are getting larger. Due to the need to create a single SBOM for a product of this sort of size, it will be large. That is the requirement.

Thanks for the logs tip, i will try it.

[nigellh]

Sigh, all sorted. Once I got to the logs it hinted to a problem with the SBOM. We had validated the SBOM using a SBOM validation tool and it had a blank name field. Not unusual to parse a package file and it has crud in there. This was one of those. Removed the record and it all imported.

Important thing was accessing the logs, thank you