Compute gets stuck

[antolinos]

I've launched the POST /compute but it seems to get stuck as the progressPercent has not changed for the last few hours:

requested	"2022-05-19T07:54:43.405306"
started	"2022-05-19T07:54:48.430000"
ended	null
progressPercent	0.5
progressDescription	"Computing weights"
inProgress	true

I restarted the docker and launch the compute but nothing changed. the script that I ran is:

import sys
import time
import logging
import argparse
import requests

#########################
# LOGGING CONFIGURATION
##########################
LOG = logging.getLogger('Scoring')
LOG.setLevel(logging.INFO)

handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter(
    '%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
LOG.addHandler(handler)


LOG.info("Start")
##########################
# INPUT PARAMETERS
##########################
parser = argparse.ArgumentParser()
requiredNamed = parser.add_argument_group('required arguments')

requiredNamed.add_argument(
    '--scoring', help='Scoring server. Example: http://dau-dm-01:9000', required=True)


args = parser.parse_args()
pss_base_url = args.scoring 
pss_compute_url = '/'.join([pss_base_url,'compute'])

LOG.info(f"Computing")
requests.post(pss_compute_url,  timeout=0.1)
LOG.info(f"Computing launched")

Any idea about what should I try? Thanks!

[nitrosx]

@antolinos : my apologies for not answering before.
Did you manage to run the computation to completion?

[antolinos]

@antolinos : my apologies for not answering before. Did you manage to run the computation to completion?

I tried several times but it did not work. I workaround-ed the problem by only working with documents. I have ignored datasets for the moment that is not ideal but at least is not blocking the developments

[nitrosx]

Can you tell me how many dataset items were present in the database?

[antolinos]

Can you tell me how many dataset items were present in the database?

Around 250k

[nitrosx]

My best guess is that it will take at least 20 min to finish the computation

[antolinos]

In my case, it did not finish. When I wrote the issue it spent few hours and then I waited for a whole day. Percentage was always 0.5
I can not touch the scoring database as I am working on that but I could try the process again to double check

[nitrosx]

That's too long!!!
Will you be able to provide an example for your items?
I check the code and I will think about how to replicate the issue on my machine.