Skip to content

oduwsdl/Extract-URLs

Folders and files

NameName
Last commit message
Last commit date
May 26, 2023
Dec 6, 2021
Jun 5, 2023
Jun 5, 2023
Sep 21, 2023
May 9, 2023
Jun 5, 2023
Jun 30, 2023
Jul 11, 2022
Mar 7, 2023
Aug 21, 2023
Jun 5, 2023
Sep 21, 2023
May 17, 2023
Mar 7, 2023
May 26, 2023
Feb 2, 2022
Sep 16, 2022
Mar 2, 2022
Mar 2, 2022
Apr 18, 2022
Mar 7, 2023
Nov 5, 2021
Mar 31, 2023
Feb 14, 2022
Apr 17, 2023
Feb 2, 2022
May 9, 2023
Jun 1, 2023
Nov 15, 2021
Apr 25, 2023
Mar 2, 2022
May 26, 2023
Jun 30, 2023
May 9, 2023
Apr 18, 2022

Repository files navigation

Extract-URLs

This project is currently set up to extract URLs from PDf publications in the arXiv corpus and run numerous data analysis programs to understand the use of URLs in the arXiv corpus.

Order of Programs

Some of the programs/scripts require the files from other programs/scripts. The correct order is shown below:

  • extract_urls.py (creates a JSON file of extracted URLs for each directory)
  • get_repo_urls.py (creates repo_urls.json (only files with a repo URL) and all_file_urls.json (all files))
  • data_processing/repo_repeats.sh (outputs the frequency of each URL for each repo)
  • data_processing/dir_urls.py (aggregates the number of URLs per month per repo and creates dir_urls.csv and file_count.csv)
  • data_processing/file_url_counts.py (creates file_url_counts.csv, ecdf_numbers.csv, and ccdf_numbers.csv)
  • data_processing/url_freq.sh (creates URL_frequency.csv)
  • data_processing/multiple_repos.sh (finds files that contain URLs to more than 1 repo and creates multiple_repos.csv)
  • data_processing/multiple_repos_table.py (creates a table of the file and which repos it links to)
  • data_processing/pdf_stats.py (calculates mean, median, and quartiles for the number of repo URLs in a PDF and creates pdf_stats.csv)
  • data_processing/repo_url_counts.py (creates repo_url_counts.csv which contains the URL count for each directory and repo)
  • data_processing/urls_per_file.py (creates urls_per_file.csv which contains the average number of URLs per mopnth for each repo)

About

No description, website, or topics provided.

Resources

Citation

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published