Quota related issues

[ihsaan-ullah]

TODOS:

• Reset User Quota from bytes to GB
  Solved by User quota is updated to GB from Bytes #1749

• Find the use of GiB/MiB/KiB in storage analytics and other places and replace with GB/MB/KB
  Solved by: File Sizes cleanup #1752

• Size of files is formatted at different places with different functions, use only one size formatter everywhere
  Solved by: File Sizes cleanup #1752

• Check why the submission size in used quota doubles when submission finishes
  Maybe because of submission file saved in the prediction output

• Find and fix the NAN file size issue

For more details check the comments section of this PR: #1738

[ObadaS]

Concerning the Quota units, the unit in the Django Admin Interface is in bits (or bytes not sure), making it hard to know how much quota is actually assigned at a quick glance

[Didayolo]

NaN values for size on production server for old files (older than ~30 days)

• Resource tab "Size" values are NaN KB #1603

[ihsaan-ullah]

check format_file_size function in src/static/riot/submissions/resource_submissions.tag

Edit:

No need to check this for NA size. The problem is explained below in comments

[ihsaan-ullah]

@ObadaS I read somewhere that minio files meta-data can expire and file_size is a metadata that may be affected if there is an expiry role. If this is true then this maybe the cause of NAN size

Edit:

This was not the problem. The problem is explained in the comments below

[ObadaS]

Interesting, I will check that tomorrow.
However, there is no NaN size problem on the codabench-test website, which is connected to another MinIO that might have a different configuration.

Codalab.lisn.upsaclay.fr also shows dataset size and does not seem to be impacted by the bug, even though they are both connected to the same MinIO, which could point to either a bucket problem on MinIO or a problem in the code of the plateform

[ihsaan-ullah]

The only problem right now is reproducibility. If we can reproduce the bug locally then solving it will not be a problem.

I am pretty much convinced that there is no problem in the code because we see sizes for some files but I will investigate the code a bit more to be sure

[ihsaan-ullah]

We found that the file sizes are emptied by reset_computed_storage_analytics function. Rerunning storage analytics should fix the NA sizes

[Didayolo]

We found that the file sizes are emptied by reset_computed_storage_analytics function. Rerunning storage analytics should fix the NA sizes

I started the storage analytics on production.

What starts the reset_computed_storage_analytics in the first place?

[ihsaan-ullah]

I confirmed using this code that accessing file size from minio takes longer than accessing the file_size from DB. We can run it on codabench-test to be sure.

Accessing from DB

import time
from datasets.models import Data

start_time = time.time()
sizes = sum(item.file_size for item in Data.objects.all())  
end_time = time.time()

execution_time = end_time - start_time
print(f"Execution Time: {execution_time:.4f} seconds")

Accessing from Minio

import time
from datasets.models import Data

start_time = time.time()
sizes = sum(item.data_file.size for item in Data.objects.all())  
end_time = time.time()

execution_time = end_time - start_time
print(f"Execution Time: {execution_time:.4f} seconds")

[Didayolo]

We need to separate the computing of the file sizes from the analytics task:

codabench/src/apps/analytics/tasks.py

Line 37 in 44c7d7f

def create_storage_analytics_snapshot():

    # Measure all files with unset size
    for dataset in Data.objects.filter(Q(file_size__isnull=True) | Q(file_size__lt=0)):
        try:
            dataset.file_size = Decimal(
                dataset.data_file.size / 1024
            )  # file_size is in KiB
        except Exception:
            dataset.file_size = Decimal(-1)
        finally:
            dataset.save()

[Didayolo]

New version of the script to compute file sizes:

from datasets.models import Data
from decimal import Decimal

print("Total objects: ", Data.objects.all().count())

datasets = Data.objects.all().order_by("id")
print("Processing now: ", datasets.count())

for dataset in datasets:
    if dataset.data_file and hasattr(dataset.data_file, 'size'):
        try:
            file_size = dataset.data_file.size
            if file_size <= 0 or file_size is None:
                file_size = Decimal(0)
            else:
                file_size = Decimal(file_size) / 1024
            dataset.file_size = file_size
            dataset.save()
        except Exception as e:
            print(f"Skipping dataset ID {dataset.id}: Invalid file size ({dataset.data_file.size}) - Error: {e}")
    else:
        print(f"File size problem, Data ID {dataset.id}")

[Didayolo]

The file sizes are back on production.

[ihsaan-ullah]

Some more files with code to fix sizes

submission_detail.txt
submission_prediction_result.txt
submission_scoring_result.txt
submission_detailed_result.txt

[Didayolo]

List of data files that failed:
Errors.txt