New Doc: Debug Local DNS Cache

source/content/continuous-integration.md

@@ -57,3 +57,6 @@ At this time, Pantheon does not provide or support:
 - [Build Tools](/guides/build-tools)
 - [Local Development](/guides/local-development)
 - [Pantheon Multidev](/guides/multidev)
+
+### Debug CI pipeline failures
+If your continuous integration pipeline is suddenly failing across a large portfolio of sites due to permission denied errors for Git/SSH authentication, we recommend [debugging local DNS cache](/local-dns-cache) to solve. 

source/content/guides/git/08-troubleshooting.md

@@ -62,7 +62,7 @@ However, some Git GUI clients, including SourceTree, also support the use of
 
 1. Paste the URL into the **Source URL** field.
 
-   1. Remove `git clone` from the beginning of the URL. 
+   1. Remove `git clone` from the beginning of the URL.
 
    1. Remove the trailing space and `my-site` name from the end of the URL provided in the **Connection Info** section of your Pantheon Dashboard.
 
@@ -91,6 +91,9 @@ fatal: Could not read from remote repository.
 
 To clear this up, you may need to work with your network administrators to unblock this port. If this isn't an option, you may need to try a [Port 2222 Blocked Workaround](/guides/sftp/port-2222).
 
+### Debug CI pipeline failures
+If your continuous integration pipeline is suddenly failing across a large portfolio of sites due to permission denied errors for Git/SSH authentication, we recommend [debugging local DNS cache](/local-dns-cache) to solve.
+
 ## More Resources
 
 We recommend the following resources for further learning:

source/content/local-dns-cache.md

@@ -0,0 +1,217 @@
+---
+title: Debug Local DNS Cache
+description: Recommendations for addressing SSH / Git authentication failures due to stale local DNS.
+tags: [cdn]
+reviewed: "2024-09-13"
+contenttype: [doc]
+innav: [true]
+categories: [cache]
+---
+DNS caching issues are critical for Pantheon developers to understand and manage. Stale DNS entries can lead to SSH/Git authentication failures, potentially disrupting your workflow and deployments. This guide provides solutions to mitigate these issues, ensuring smooth operations across your Pantheon sites.
+
+Pantheon application containers are sometimes migrated during routine platform maintenance. Following an application container migration, it is possible for Local DNS cache to cause SSH / Git authentication failures resulting in errors like `Permission denied (publickey)` or the following error, as the result of an operation like `git clone`:
+
+```
+fatal: Could not read from remote repository.
+
+Please make sure you have the correct access rights
+and the repository exists.
+```
+
+Retrying after the failure will resolve the issue, which is fine enough in the context of a single site but not so easy to handle across hundreds of sites. Those with large site portfolios using continuous integration and automated deployments might see this issue surface as a large spike in failed deployments across many sites. The scripts below provide a flexible and automated approach to managing local DNS cache issues when working with Pantheon sites. By flushing stale DNS entries and retrying failed commands, you can ensure smoother SSH / Git operations and reduce the likelihood of deployment failures due to stale DNS entries.
+
+## Context and Use Cases
+These scripts are particularly valuable for:
+
+- Large-scale deployments managing multiple Pantheon sites
+- CI/CD pipelines where automated Git operations are frequent
+- Developers experiencing intermittent SSH/Git authentication failures
+
+## Prerequisites
+Before implementing these solutions, ensure you have:
+
+- Terminus installed and authenticated
+- Appropriate permissions to execute system commands (for DNS cache flushing)
+- Git installed and configured for your Pantheon sites
+
+## Option 1: Eliminate Local DNS Caching
+The first option is to assume the DNS cache is always stale and to flush caches prior to any interaction with a Pantheon codeserver. In the example below, flush_dns_cache is called prior to executing any git command.
+
+### Pros and Cons
+
+**Pros:**
+
+- Ensures fresh DNS resolution for every Git operation
+- Simplifies troubleshooting by eliminating DNS caching as a variable
+
+**Cons:**
+
+- May introduce slight performance overhead due to frequent cache flushing
+- Requires elevated permissions to flush DNS cache on some systems
+
+```
+# Function to flush DNS cache based on the operating system
+function flush_dns_cache() {
+    case "$(uname -s)" in
+        Darwin)
+            # macOS
+            echo "Flushing DNS cache on macOS..."
+            sudo dscacheutil -flushcache
+            sudo killall -HUP mDNSResponder
+            ;;
+        Linux)
+            # Linux
+            echo "Flushing DNS cache on Linux..."
+            sudo systemd-resolve --flush-caches
+            ;;
+        CYGWIN*|MINGW32*|MSYS*|MINGW*)
+            # Windows
+            echo "Flushing DNS cache on Windows..."
+            ipconfig /flushdns
+            ;;
+        *)
+            echo "Unsupported OS. DNS cache flush not performed."
+            ;;
+    esac
+}
+
+# Flush the DNS cache
+flush_dns_cache
+
+# Standard Pantheon git commands, eg
+git clone ssh://codeserver.dev.[id]@codeserver.dev.[id].drush.in:2222/~/repository.git -b master mysite
+```
+
+
+#### How to Use
+
+1. Create a Script File: Save the above script in a file, e.g., `flush_dns_and_clone.sh`.
+2. Make the Script Executable: Run `chmod +x flush_dns_and_clone.sh` to make the script executable.
+3. Run the Script: Execute the script by running `./flush_dns_and_clone.sh`.
+
+This script will flush the DNS cache based on your operating system and then proceed with the git clone command.
+
+## Option 2: Conditionally Flush Stale Local DNS and Retry Git Command Failures
+
+To resolve git command errors such as `Permission denied (publickey)`, it is recommended to automatically flush stale local DNS caches for Pantheon application containers and retry failed commands in order to prevent CI deployment errors following Pantheon platform maintenance tasks.
+
+### Pros and Cons
+
+**Pros:**
+
+- More efficient than Option 1, as it only flushes DNS cache when necessary
+- Automatically retries failed Git commands, reducing manual intervention
+
+**Cons:**
+
+- Slightly more complex implementation compared to Option 1
+- Requires setting up environment variables (`SITE_ID`, `ENV_ID`, `BRANCH`)
+- May increase execution time for Git operations due to potential retries
+- Depends on external commands (dig, terminus) which need to be available in the environment
+
+In the example below, any git clone or push command error results in calling the `check_dns_cache` function to flush stale local DNS entries, and then the respective git command is retried. See below for a more detailed explanation of the functions used in the script:
+
+**Detailed Function Explanations**
+
+1. `flush_dns_cache()`: This function detects the operating system and executes the appropriate DNS cache flushing command.
+
+2. `check_dns_cache()`: This function compares the cached IP with the actual IP of the Git host. If they differ, it triggers a DNS cache flush.
+
+3. `push_code()`: This function encapsulates Git clone and push operations with retry logic, calling `check_dns_cache()` on failures.
+
+```
+function flush_dns_cache() {
+    case "$(uname -s)" in
+        Darwin)
+            # macOS
+            echo "Flushing DNS cache on macOS..."
+            sudo dscacheutil -flushcache
+            sudo killall -HUP mDNSResponder
+            ;;
+        Linux)
+            # Linux
+            echo "Flushing DNS cache on Linux..."
+            sudo systemd-resolve --flush-caches
+            ;;
+        CYGWIN*|MINGW32*|MSYS*|MINGW*)
+            # Windows
+            echo "Flushing DNS cache on Windows..."
+            ipconfig /flushdns
+            ;;
+        *)
+            echo "Unsupported OS. DNS cache flush not performed."
+            ;;
+    esac
+}
+
+function check_dns_cache {
+  GIT_HOST=$(terminus connection:info ${SITE_ID}.${ENV_ID} --field=git_host)
+  CACHED_IP=$(dig +short A $GIT_HOST)
+  UNCACHED_IP=$(dig +short $GIT_HOST @8.8.8.8)
+  echo "Checking for stale DNS cache for $GIT_HOST"
+  # Check if CACHED_IP does not equal UNCACHED_IP
+  if [ "$CACHED_IP" != "$UNCACHED_IP" ]; then
+    echo "IP address has changed, flushing DNS cache"
+    flush_dns_cache
+  else
+    echo "IP address has not changed, no need to flush DNS cache"
+  fi
+}
+
+function push_code {
+  # Example: if git clone fails, check for stale DNS cache and retry
+  MAX_RETRIES=5
+  RETRY_COUNT=0
+  while [ $RETRY_COUNT -lt $MAX_RETRIES ]; do
+    git clone --branch $ENV_ID --depth 2 ${{ env.UPSTREAM_GIT_URL }}
+    if [ $? -eq 0 ]; then
+      break
+    else
+      RETRY_COUNT=$((RETRY_COUNT+1))
+      check_dns_cache && echo "Retrying git clone command..."
+    fi
+  done
+  # Continue existing code
+
+  # Example: if git push fails, check for stale DNS cache and retry
+  MAX_RETRIES=5
+  RETRY_COUNT=0
+  while [ $RETRY_COUNT -lt $MAX_RETRIES ]; do
+    git push pantheon $BRANCH -vv
+    if [ $? -eq 0 ]; then
+      break
+    else
+      RETRY_COUNT=$((RETRY_COUNT+1))
+      check_dns_cache && echo "Retrying git push command..."
+    fi
+  done
+}
+
+push_code
+```
+
+### How to Use
+
+1. Create a Script File: Save the above script in a file, e.g., `conditional_flush_and_retry.sh`.
+2. Make the Script Executable: Run `chmod +x conditional_flush_and_retry.sh` to make the script executable.
+3. Set Environment Variables: Ensure that `SITE_ID`, `ENV_ID`, and `BRANCH` environment variables are set appropriately.
+4. Run the Script: Execute the script by running `./conditional_flush_and_retry.sh`.
+
+## Troubleshooting
+- Issue: Permission denied when flushing DNS cache
+  Solution: Ensure your script has sudo privileges or run with elevated permissions
+
+- Issue: Git commands still failing after DNS flush
+  Solution: Verify your SSH keys are correctly set up in your Pantheon account
+
+## Best Practices
+1. Implement these scripts in your local development environment to catch issues early
+2. Use Option 2 to minimize unnecessary cache flushes
+3. Regularly update your local Git configurations to match Pantheon's latest recommendations
+
+## Performance Considerations
+
+For large site portfolios:
+
+- Consider implementing a cooldown period between retries to prevent overwhelming Pantheon's services
+- Monitor the frequency of DNS cache flushes and adjust the retry logic if necessary