This repo takes several word lists and compiles them to a list of common words and a list of less common words.

To use the word lists

raw/wordnik.txt is an opensource wordlist from Wordnik. It contains ~200,000 entries, plus a few entries that were added as per user request.

raw/wiki.txt is the word frequency on Wikipedia, compiled by https://github.com/IlyaSemenov/wikipedia-word-frequency. It contains ~2.6 million entries, including non-English entries.

raw/gutenberg.txt is the 40,000 most common words on Project Gutenberg, as compiled in the April 16, 2006 list by Wiktionary.

raw/movies.txt is the 10,000 most common words from movies and TV, as compiled in the 2006 list by Wiktionary.

LDNOOBW.txt is the "list of dirty, naughty, obscene, and otherwise bad words" from LDNOOBW, minus multi-word phrases, minus words not in raw/wordnik.txt, minus words that were subjectively deemed to be non-offensive due to a non-slang or scientific meaning, plus variations of words.

Files were processed into files that contain words that also exist in the Wordnik list but not the LDNOOBW word list.

processed/wordnik.txt is raw/wordnik.txt minus words from LDNOOBW.txt. processed/wordnik_processor.py is the script that generated this file.

processed/gutenberg.txt is raw/gutenberg.txt minus words that do not exist in processed/wordnik.txt. processed/gutenberg_processor.py is the script that generated this file.

processed/movies.txt is raw/movies.txt minus words that do not exist in processed/wordnik.txt. processed/movies_processor.py is the script that generated this file.

processed/wiki.txt is raw/wiki.txt, minus words that words that do not exist in processed/wordnik.txt. processed/wiki_frequency_processor.py is the script that generated this file.

compiled/commonWords.txt consists of words that are on common to wordnik, wiki, and gutenberg AND words that are common to wordnik and movies. It omits words from LDNOOBW.txt. In addition, the list excludes the manually compiled compiles/notActuallyCommon.txt that were deemed subjectively not common and includes the manually compiled compiles/notActuallyUncommon.txt that were deemed subjectively not uncommon.

compiled/uncommonWords.json consists of all of the wordnik words that are not in compiled/commonWords.txt, minus words from LDNOOBW.txt.

These files were generated by compiled/compile.py.