Merge branch 'main' into aws_benchmarks

README.md

@@ -49,21 +49,21 @@ If you want to use this code, contribute to it, ask questions or propose ideas,
 
 [![](docs/figures/overview/discourse_screenshot.png)](https://thousandbrains.discourse.group/)
 
-If you would like to receive updates, follow us on [Bluesky](https://bluesky.social/thousandbrainsproject) or [Twitter](https://x.com/1000brainsproj) or [LinkedIn](https://www.linkedin.com/showcase/thousand-brains-project/).
+If you would like to receive updates, follow us on [Bluesky](https://bsky.app/profile/1000brainsproj.bsky.social) or [Twitter](https://x.com/1000brainsproj) or [LinkedIn](https://www.linkedin.com/company/thousand-brains-project/).
 
 If you have further questions or suggestions for collaborations, don't hesitate to contact us directly at **[email protected]**.
 
 # Citing the Project
 If you're writing a publication that references the Thousand Brains Project, please cite our TBP whitepaper:
 ```
 @misc{thousandbrainsproject2024,
-      title={The Thousand Brains Project: A New Paradigm for Sensorimotor Intelligence}, 
+      title={The Thousand Brains Project: A New Paradigm for Sensorimotor Intelligence},
       author={Viviane Clay and Niels Leadholm and Jeff Hawkins},
       year={2024},
       eprint={2412.18354},
       archivePrefix={arXiv},
       primaryClass={cs.AI},
-      url={https://arxiv.org/abs/2412.18354}, 
+      url={https://arxiv.org/abs/2412.18354},
 }
 ```
 

docs/contributing/contributing.md

@@ -57,7 +57,7 @@ We love seeing how you use Monty. If you created something interesting with it,
 - **Write a Blog Post**: Share your experience and insights by writing a blog post. Please share your post with the community on our [Discourse server](https://thousandbrains.discourse.group/).
 - **Publish a Paper**: If you use our Monty implementation or ideas from the Thousand Brains Theory in your next publication, we would like to feature you on our [TBP-based papers](../community/tbp-based-papers.md) list and increase the visibility of your research.
 - **Present at Community Events**: We host regular webinars and community meetups. If you are interested in presenting your project or research, please get in touch with us at [[email protected]](mailto:[email protected]).
-- **Social Media**: Share your creations on social media using the hashtag `#1000brainsproject`. Follow us on [X](https://x.com/1000brainsproj), [Bluesky](https://bsky.app/profile/1000brainsproj.bsky.social) or [LinkedIn](https://www.linkedin.com/showcase/thousand-brains-project/), and subscribe to our [YouTube Channel](https://www.youtube.com/@thousandbrainsproject), or our [email list](https://thousandbrains.org/#newsletter).
+- **Social Media**: Share your creations on social media using the hashtag `#1000brainsproject`. Follow us on [X](https://x.com/1000brainsproj), [Bluesky](https://bsky.app/profile/1000brainsproj.bsky.social) or [LinkedIn](https://www.linkedin.com/company/thousand-brains-project/), and subscribe to our [YouTube Channel](https://www.youtube.com/@thousandbrainsproject), or our [email list](https://thousandbrains.org/#newsletter).
 
 # Promote Monty
 

docs/overview/welcome-to-the-thousand-brains-project-documentation.md

@@ -46,18 +46,18 @@ We are excited to have you here!  Our intention for making the project open-sour
 | <a href="https://thousandbrains.org/" style="display: flex; align-items: center;"><img src="../figures/overview/website.png" alt="Website" height="15" style="opacity: 1; transition: opacity 0.2s; &:hover { opacity: 0.8; }" pointer-events="none">&nbsp;Official&nbsp;Website</a> | Learn about our mission, team and the science behind the project |
 | <a href="https://www.youtube.com/@thousandbrainsproject" style="display: flex; align-items: center;"><img src="../figures/overview/youtube.png" alt="YouTube" height="15" style="opacity: 1; transition: opacity 0.2s; &:hover { opacity: 0.8; }" pointer-events="none">&nbsp;YouTube&nbsp;Channel</a> | Watch tutorials, technical deep-dives and project updates |
 | <a href="https://bsky.app/profile/1000brainsproj.bsky.social" style="display: flex; align-items: center;"><img src="../figures/overview/bluesky.png" alt="Bluesky" height="15" style="opacity: 1; transition: opacity 0.2s; &:hover { opacity: 0.8; }" pointer-events="none">&nbsp;Bluesky</a> <a href="https://x.com/1000brainsproj" style="display: flex; align-items: center;"><img src="../figures/overview/twitter.png" alt="X/Twitter" height="15" style="opacity: 1; transition: opacity 0.2s; &:hover { opacity: 0.8; }" pointer-events="none">&nbsp;X/Twitter</a> | Get the latest news and announcements, and engage with our community |
-| <a href="https://www.linkedin.com/showcase/thousand-brains-project" style="display: flex; align-items: center;"><img src="../figures/overview/linkedin.png" alt="LinkedIn" height="15" style="opacity: 1; transition: opacity 0.2s; &:hover { opacity: 0.8; }" pointer-events="none">&nbsp;LinkedIn&nbsp;Page</a> | Get the latest news and announcements |
+| <a href="https://www.linkedin.com/company/thousand-brains-project" style="display: flex; align-items: center;"><img src="../figures/overview/linkedin.png" alt="LinkedIn" height="15" style="opacity: 1; transition: opacity 0.2s; &:hover { opacity: 0.8; }" pointer-events="none">&nbsp;LinkedIn&nbsp;Page</a> | Get the latest news and announcements |
 
 # Citing this Project
 If you're writing a publication that references the Thousand Brains Project, please cite our TBP whitepaper:
 ```
 @misc{thousandbrainsproject2024,
-      title={The Thousand Brains Project: A New Paradigm for Sensorimotor Intelligence}, 
+      title={The Thousand Brains Project: A New Paradigm for Sensorimotor Intelligence},
       author={Viviane Clay and Niels Leadholm and Jeff Hawkins},
       year={2024},
       eprint={2412.18354},
       archivePrefix={arXiv},
       primaryClass={cs.AI},
-      url={https://arxiv.org/abs/2412.18354}, 
+      url={https://arxiv.org/abs/2412.18354},
 }
 ```

tools/github_readme_sync/colors.py

@@ -37,6 +37,7 @@ def _supports_color():
     WHITE = Fore.WHITE
     CYAN = Fore.CYAN
     BLUE = Fore.BLUE
+    YELLOW = Fore.YELLOW
     RESET = Style.RESET_ALL
 else:
     # If color is not supported, use empty strings
@@ -46,6 +47,7 @@ def _supports_color():
     WHITE = ""
     CYAN = ""
     BLUE = ""
+    YELLOW = ""
     RESET = ""
 
 # No need to call any functions; the file automatically initializes color support.

tools/github_readme_sync/excluded_items.py

@@ -10,3 +10,9 @@
 
 IGNORE_DOCS = ["placeholder-example-doc", "some-existing-doc"]
 IGNORE_IMAGES = ["docs-only-example.png"]
+IGNORE_EXTERNAL_URLS = [
+    "openai.com",
+    "science.org",
+    "annualreviews.org",
+    "sciencedirect.com",
+]

tools/github_readme_sync/hierarchy.py

@@ -13,17 +13,25 @@
 import os
 import re
 import sys
+import timeit
 
 import requests
 
-from tools.github_readme_sync.colors import CYAN, GREEN, RED, RESET, WHITE
-from tools.github_readme_sync.excluded_items import IGNORE_DOCS, IGNORE_IMAGES
+from tools.github_readme_sync.colors import CYAN, GREEN, RED, RESET, WHITE, YELLOW
+from tools.github_readme_sync.excluded_items import (
+    IGNORE_DOCS,
+    IGNORE_EXTERNAL_URLS,
+    IGNORE_IMAGES,
+)
 
 HIERARCHY_FILE = "hierarchy.md"
 CATEGORY_PREFIX = "# "
 DOCUMENT_PREFIX = "- "
 INDENTATION_UNIT = "  "  # Single indentation level
 
+# URLs that are checked
+README_URL = "https://thousandbrainsproject.readme.io"
+
 
 def create_hierarchy_file(output_dir, hierarchy):
     with open(os.path.join(output_dir, HIERARCHY_FILE), "w") as f:
@@ -195,7 +203,7 @@ def check_external(folder, ignore_dirs, rdme):
             [os.path.join(root, file) for file in files if file.endswith(".md")]
         )
 
-    with concurrent.futures.ThreadPoolExecutor() as executor:
+    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
         future_to_file = {
             executor.submit(process_file, file_path, rdme, url_cache): file_path
             for file_path in md_files
@@ -259,22 +267,29 @@ def extract_links(content):
 
 
 def is_readme_url(url):
-    return url.startswith("https://thousandbrainsproject.readme.io")
+    return url.startswith(README_URL)
 
 
 def is_external_url(url):
     return url.startswith(("http://", "https://"))
 
 
 def check_readme_link(url, rdme):
-    if url == "https://thousandbrainsproject.readme.io/":
+    if url == f"{README_URL}/":
         return []
 
     try:
         doc_slug = url.split("/")[-1]
+        time = timeit.default_timer()
         response = rdme.get_doc_by_slug(doc_slug)
+        time = timeit.default_timer() - time
+        status_color = GREEN if response else RED
+        log_msg = f"{CYAN}{url} {status_color}"
+        log_msg += f"[{200 if response else 404}]{RESET}"
+        if time > 1:
+            log_msg += f" ({YELLOW}{time:.2f}s{RESET})"
+        logging.info(log_msg)
         if not response:
-            logging.debug(f"{WHITE}  {url} (Not found){RESET}")
             return [f"  broken link: {url} (Not found)"]
     except Exception as e:
         return [f"  {url}: {str(e)}"]
@@ -283,18 +298,55 @@ def check_readme_link(url, rdme):
 
 
 def check_external_link(url):
+    if any(ignored_url in url for ignored_url in IGNORE_EXTERNAL_URLS):
+        logging.info(f"{WHITE}{url} {GREEN}[IGNORED]{RESET}")
+        return []
+
     try:
-        headers = request_headers()
-        response = requests.get(url, timeout=5, headers=headers)
+        time = timeit.default_timer()
+        response = check_url(url)
+        time = timeit.default_timer() - time
+
+        status_color = GREEN if 200 <= response.status_code <= 299 else RED
+        log_msg = f"{WHITE}{url} {status_color}[{response.status_code}]{RESET}"
+        if time > 1:
+            log_msg += f" ({YELLOW}{time:.2f}s{RESET})"
+        logging.info(log_msg)
         if response.status_code < 200 or response.status_code > 299:
-            logging.debug(f"{WHITE}  {url} ({response.status_code}){RESET}")
             return [f"  broken link: {url} ({response.status_code})"]
     except requests.RequestException as e:
         return [f"  {url}: {str(e)}"]
 
     return []
 
 
+def check_url(url):
+    """Check if the URL exists.
+
+    The cache-control was just in-case.
+    The User Agent was needed to stop a 406 from
+    ycbbenchmarks.com. I think it will work with
+    any user-agent, but I figured a realistic one
+    was a bit more future proof.
+
+    Returns:
+        requests.Response: The response from the URL request.
+    """
+    headers = request_headers()
+
+    try:
+        response = requests.head(url, timeout=5, headers=headers)
+    except requests.RequestException:
+        # If HEAD fails, try GET instead
+        response = requests.get(url, timeout=5, headers=headers)
+    else:
+        # If HEAD succeeds but returns non-2xx, try GET
+        if not (200 <= response.status_code <= 299):
+            response = requests.get(url, timeout=5, headers=headers)
+
+    return response
+
+
 def request_headers():
     """Populate the headers for the request.
 
@@ -317,6 +369,7 @@ def request_headers():
 
 
 def report_errors(errors, total_links_checked):
+    logging.info("")
     if errors:
         for file_path, file_errors in errors.items():
             logging.error(f"{RED}{file_path}{RESET}")