marius_preprocess triggers program aborted

[VeritasYin]

Describe the bug
run marius_preprocess or import preprocess would trigger the following error.

free(): invalid pointer
Aborted

To Reproduce
Steps to reproduce the behavior:

1. Run the given example 'marius_preprocess output_dir/ --dataset fb15k'
   OR
2. 'from marius.tools import preprocess' in Python

Environment
gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)
Python 3.8.5

[JasonMoho]

Thanks for reporting, haven't seen this before. I'll try reproducing this tomorrow with the above environment.

Could you try working around this by using a different environment? The gcc version should be fine, but you can try python 3.7 or 3.9 and see if that fixes it.

I also haven't deployed on ubuntu20 before and usually work off ubuntu18. So if changing the python version doesn't work, you can try running with docker using this dockerfile. This should ensure that there are no environment issues.

[JasonMoho]

@VeritasYin
Noticed in #82 You got things running by downgrading to python 3.7. Did you change anything else in your environment? Or was that the only thing you needed to do?

[VeritasYin]

I first downgrade to Python 3.7 and then recompile the code from the scratch. This time, it works. But it occasically shows error when running under cpu config as "libgomp: Thread creation failed: Resource temporarily unavailable".

[JasonMoho]

Yeah must be some strange issue with python 3.8. I usually run on 3.6 or 3.7. We have a new version of the system releasing in February, I'll make sure to address this issue for that release.

As far as the libgomp issue. It's possible that too many threads are being created by OMP and pytorch. You could try limiting the number of OMP threads by setting OMP_NUM_THREADS manually. I think by default this will be set to the number of cores on your machine.

[VeritasYin]

@JasonMoho Thanks for your prompt response. Another issue regarding preprocess is that for homophily graphs such as ogbl datasets, the preprocess will not generate the file "rel_mapping.txt", which would trigger errors during postprocessing for exporting node embeddings.

[JasonMoho]

No problem! Appreciate that you are reporting these issues.

For our upcoming release we have completely revamped the preprocessing and postprocessing, and this mapping issue was fixed for that. Sorry about the current alpha state that things are in, it will be quite a bit more polished soon :)

[JasonMoho]

Closing this as version 0.0.2 has been merged and includes improved preprocessing.