Skip to content
@commoncrawl

Common Crawl Foundation

Common Crawl provides an archive of webpages going back to 2007.

Pinned Loading

  1. cc-pyspark cc-pyspark Public

    Process Common Crawl data with Python and Spark

    Python 413 88

  2. cc-crawl-statistics cc-crawl-statistics Public

    Statistics of Common Crawl monthly archives mined from URL index files

    Python 167 11

  3. cc-index-table cc-index-table Public

    Index Common Crawl archives in tabular format

    Java 111 9

  4. cc-warc-examples cc-warc-examples Public

    Forked from Smerity/cc-warc-examples

    CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop

    Java 37 18

  5. cc-citations cc-citations Public

    Scientific articles using or citing Common Crawl data

    Jupyter Notebook 12 3

  6. cc-notebooks cc-notebooks Public

    Various Jupyter notebooks about Common Crawl data

    Jupyter Notebook 49 9

Repositories

Showing 10 of 64 repositories
  • cc-webgraph Public

    Tools to construct and process webgraphs from Common Crawl data

    commoncrawl/cc-webgraph’s past year of commit activity
    Java 84 Apache-2.0 5 2 (1 issue needs help) 0 Updated Jan 31, 2025
  • cc-crawl-statistics Public

    Statistics of Common Crawl monthly archives mined from URL index files

    commoncrawl/cc-crawl-statistics’s past year of commit activity
    Python 167 Apache-2.0 11 0 0 Updated Jan 31, 2025
  • cc-citations Public

    Scientific articles using or citing Common Crawl data

    commoncrawl/cc-citations’s past year of commit activity
    Jupyter Notebook 12 3 0 0 Updated Jan 30, 2025
  • cc-notebooks Public

    Various Jupyter notebooks about Common Crawl data

    commoncrawl/cc-notebooks’s past year of commit activity
    Jupyter Notebook 49 Apache-2.0 9 0 1 Updated Jan 28, 2025
  • web-languages Public

    Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ for the code

    commoncrawl/web-languages’s past year of commit activity
    33 37 0 0 Updated Jan 28, 2025
  • webarchive-indexing Public Forked from ikreymer/webarchive-indexing

    Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.

    commoncrawl/webarchive-indexing’s past year of commit activity
    Python 5 MIT 10 0 2 Updated Jan 27, 2025
  • cc-index-server Public Forked from ikreymer/cc-index-server

    Common Crawl Index Server

    commoncrawl/cc-index-server’s past year of commit activity
    HTML 65 25 6 1 Updated Jan 24, 2025
  • cc-downloader Public

    A polite and user-friendly downloader for Common Crawl data

    commoncrawl/cc-downloader’s past year of commit activity
    Rust 22 Apache-2.0 1 1 0 Updated Jan 21, 2025
  • uap-core Public Forked from ua-parser/uap-core

    The regex file necessary to build language ports of Browserscope's user agent parser.

    commoncrawl/uap-core’s past year of commit activity
    JavaScript 0 462 0 0 Updated Jan 17, 2025
  • web-languages-code Public

    The code used to generate templates for the web-languages repo https://github.com/commoncrawl/web-languages

    commoncrawl/web-languages-code’s past year of commit activity
    Python 1 Apache-2.0 1 0 0 Updated Jan 11, 2025
View all repositories