refactor: csv loader code implementation #4529
Conversation
AfaqShuaib09
force-pushed
the
afaq/prod-4238
branch
3 times, most recently
from
a2949c8 to
a32c714
Compare
course_discovery/apps/course_metadata/data_loaders/csv_loader.py
Outdated
Show resolved
Hide resolved
course_discovery/apps/course_metadata/data_loaders/csv_loader.py
Outdated
Show resolved
Hide resolved
AfaqShuaib09
force-pushed
the
afaq/prod-4238
branch
from
82c4fe7 to
563a4a3
Compare
course_discovery/apps/course_metadata/data_loaders/csv_loader.py
Outdated
Show resolved
Hide resolved
course_discovery/apps/course_metadata/data_loaders/csv_loader.py
Outdated
Show resolved
Hide resolved
AfaqShuaib09
force-pushed
the
afaq/prod-4238
branch
from
50e160b to
d797485
Compare
course_discovery/apps/course_metadata/data_loaders/csv_loader.py
Outdated
Show resolved
Hide resolved
| @@ -438,6 +453,16 @@ def _log_ingestion_error(self, error_code, message): | |||
| logger.error(message) | |||
| self._register_ingestion_error(error_code, message) | |||
|
|
|||
| @classmethod | |||
| def clear_caches(cls): | |||
There was a problem hiding this comment.
Why do we need to clear the caches?
There was a problem hiding this comment.
Because functools caches are global caches instead of instance-level caches, I believe it's better to clear the caches after a job has been completed. If we don't do that, we would need to explicitly clear them in testsuites.
There was a problem hiding this comment.
If the only usage is in the tests, maybe add a comment indicating that.
There was a problem hiding this comment.
It's kind of used for tests as well, it's better to clear all the method-level caches after the execution of the job therefore I called this at the end of ingest method
| @@ -438,6 +453,16 @@ def _log_ingestion_error(self, error_code, message): | |||
| logger.error(message) | |||
| self._register_ingestion_error(error_code, message) | |||
|
|
|||
| @classmethod | |||
| def clear_caches(cls): | |||
There was a problem hiding this comment.
If the only usage is in the tests, maybe add a comment indicating that.
AfaqShuaib09
force-pushed
the
afaq/prod-4238
branch
from
30ef3be to
ba6fbbf
Compare
AfaqShuaib09
force-pushed
the
afaq/prod-4238
branch
from
ba6fbbf to
fc7e7b4
Compare
PROD-4238
This PR refactors the CSV loader code to improve readability, making it cleaner and easier to follow.