Extraction is bottlenecked / doesn't make full usage of multiple cores

[frankier]

When performing an extraction on a machine with 40 cores and the following settings:

  "extractor": {
    "threadPoolSize": 64,
    "taskQueueSize": 80,
    "shotQueueSize": 80,
    "outputLocation": "out"
  },

Most of the resources as not used. This is using the JSON writer as advised for HPC extraction in the other issue.

About 70% of the time it's as if single threaded

About 30% of the time it uses slightly more but not close to the full 40 cores

Since this is essentially an embarrassingly parallel problem (each video is independent), we should be able to do improve upon this which would help for performing extraction on large collections.

An easy way of fixing this would be to have "external" parallelism where many extraction processes are started with Snakemake or similar (I have been using this together with slurm like so https://github.com/frankier/singslurm ) and each extraction process would only deal with a single file before finishing, which would also allow extraction across multiple nodes. The resulting JSONs could then be merged later on. This is already almost possible I think by templating the file name into the the extraction job config but in this case I think the paths in the metadata JSONs will need to be fixed up. Or is this what the skip and limit parameters are intended for? Should I template the job file to start many jobs to deal with e.g. 10 videos at a time this way? If so I suppose these numbers include all files, or just mp4 files? If this is the recommended mechanism for HPC extractions, we should definitely document it.

[lucaro]

How many feature modules are you using during the extraction? If you are only using a small number, the decoding and 'house-keeping' overhead might be larger than the extraction workload, so you might not get a high resource utilization independent of the queue size. Since cineast does not have the possibility to work on multiple videos simultaneously, starting multiple instances in parallel would be your easiest option here. You should be able to set the output directory for the JSON output in the job file, so avoiding conflicts when writing files should also be possible.

[frankier]

I am just using the example configuration:

  "retriever": {
    "threadPoolSize": 2,
    "maxResults": 1000,
    "resultsPerModule": 1500,
    "features": {
      "semantic" : [
        {"feature": "ConceptMasksAde20k", "weight" : 1.0}
      ],
      "globalcolor": [
        {"feature": "AverageColor",  "weight": 2.3},
        {"feature": "MedianColor", "weight": 1.2},
        {"feature": "AverageFuzzyHist", "weight": 0.7},
        {"feature": "HueHistogram", "weight": 0.7},
        {"feature": "MedianFuzzyHist",   "weight": 1.3 }
      ],
      "localcolor": [
        {"feature": "AverageColorARP44", "weight": 0.5},
        {"feature": "AverageColorARP44Normalized", "weight": 0.5},
        {"feature": "SubDivMedianFuzzyColor", "weight": 0.85},
        {"feature": "AverageColorGrid8", "weight": 1.8},
        {"feature": "AverageColorGrid8Normalized", "weight": 1.8},
        {"feature": "CLD", "weight": 1.3},
        {"feature": "CLDNormalized", "weight": 1.3},
        {"feature": "MedianColorGrid8", "weight": 1.7},
        {"feature": "AverageColorRaster", "weight": 1.0}
      ],
      "edge": [
        {"feature": "EHD", "weight": 0.7},
        {"feature": "DominantEdgeGrid16", "weight": 1.4},
        {"feature": "DominantEdgeGrid8", "weight": 1.4}
      ],
      "motion": [
        {"feature": "SubDivMotionHistogram3", "weight": 0.5},
        {"feature": "SubDivMotionHistogram5", "weight": 0.5},
        {"feature": "SubDivMotionHistogramBackground3", "weight": 0.5},
        {"feature": "SubDivMotionHistogramBackground5", "weight": 0.5}
      ],
      "quantized": [

      ],
      "localfeatures": [
        {"feature": "HOGMirflickr25K512", "weight": 1.0}
      ],
      "localfeatures_fast": [
        {"feature": "HOGMirflickr25K256", "weight": 1.0}
      ],
      "audiofingerprint": [
        {"feature": "HPCP12Shingle", "weight": 1.0}
      ],
      "tags": [
        {"feature": "SegmentTags", "weight": 1.0}
      ],
      "ocr": [
        {"feature": "OCRSearch", "weight": 1.0}
      ],
      "asr": [
        {"feature":  "AudioTranscriptionSearch", "weight":  1.0}
      ],
      "scenecaption": [
        {"feature":  "DescriptionTextSearch", "weight":  1.0}
      ]
    }
  },

I will move on to arranging my own process level parallelism for now.

[frankier]

I noticed that the REST API provides the possibility of submitting extraction jobs. It seems there was also a project https://github.com/vitrivr/cthulhu which I assume used this API. Could you please tell me a bit about the status of this?

[lucaro]

Cthulhu and the extraction API were two independent paths to a somewhat related goal, neither of which gained a lot of traction. The idea behind both was to have a means for continuous extraction, where you would have one or several vitrivr instances or machines where instances could be run sitting somewhere, waiting for input and an external mechanism which would provide this input. In the case of Cthulhu, it was meant more as an orchestration tool to distribute the extraction of large amounts of data in a cluster setting. It doesn't make use of this extraction API, since that API didn't exist at the time. The primary problem here was that, while the approach worked reasonably well in a grid-like setting, where you have a bunch of machines which are always there and you could use some of their resources, it didn't work all that well in a job-cluster setting. The extraction API came later, to enable continuous extension of an indexed collection of media items. We used it for some demos, where an external tool would crawl the web for some recent images matching some predefined properties and feed whatever it found to a vitrivr instance, running on the same machine. Other than such demos, we never did a lot with it, outside of such demo applications.

[frankier]

Okay. Thanks for the background.

Edited to add: this seems like it might be a good profiler for picking up problems with lock contentions: https://github.com/jvm-profiling-tools/async-profiler

I've had good experiences with profilers that work this way (halt using a signal at random and then inspect memory dumps to recover information about program state) for Python -- they give quite accurate/actionable data.

[frankier]

I didn't benchmark, but I suspect part of it might be these locks:

cineast/cineast-core/src/main/java/org/vitrivr/cineast/core/data/segments/VideoSegment.java

Line 204 in c5d1c6d

synchronized (getAvgLock) {

cineast/cineast-core/src/main/java/org/vitrivr/cineast/core/data/segments/VideoSegment.java

Line 216 in c5d1c6d

synchronized (getMedianLock) {

cineast/cineast-core/src/main/java/org/vitrivr/cineast/core/data/segments/VideoSegment.java

Line 282 in c5d1c6d

synchronized (getMostRepresentativeLock) {

Lots of features use one of these computed properties of the segment and they might all have to wait for these results. I don't have an easy fix, but often when there's this type of data dependency structure pipelining can help.

[lucaro]

These locks are there to prevent having to perform this computation multiple times, so the alternative of waiting would be redundant computation in this case.