Troubleshooting

Troubleshooting

Logs

Follow the server logs in case of errors:

• apache-tomcat-x/logs/catalina.out
• apache-tomcat-x/logs/asgard.log

If you cannot determine the solution, search the Asgard Google group for the error. If you don't find a solution, post your question there.

Increase JVM heap size for larger accounts

If your account contains a large number of cloud objects you might need to increase Asgard’s memory footprint. It’s also advisable to change the JVM garbage collection settings. To do make these changes, create a setenv.sh file in apache-tomcat-x/bin/ similar to the following code block, then restart Tomcat.

    if [ "$1" == "start" ]; then
        export JAVA_OPTS=" \
            -verbose:sizes \
            -Xmx4g -Xms4g \
            -Xmn2g \
            -XX:MaxPermSize=256m \
            -XX:+HeapDumpOnOutOfMemoryError \
            -XX:-UseGCOverheadLimit \
            -XX:+ExplicitGCInvokesConcurrent \
            -XX:+PrintGCDateStamps -XX:+PrintGCDetails \
            -XX:+PrintTenuringDistribution \
            -XX:+CMSClassUnloadingEnabled \
            -XX:+UseConcMarkSweepGC \
        "
    else
        export JAVA_OPTS=""
    fi

Problems getting data from Amazon

If Asgard is failing to start up after several minutes, check your asgard.log file. See if it shows a lot of exceptions filling caches, like this:

[2012-11-05 17:08:43,772] [background-process-4] com.netflix.asgard.CachedMap Exception filling cache us-east-1 Security Group
com.amazonaws.AmazonClientException: Unable to unmarshall response (ParseError at [row,col]:[5097,13]
Message: Read timed out)

If so, there might be a lot of metadata in your AWS accounts, combined with a flaky network connection between your Asgard instance and some AWS API endpoints. To work around this, you can start Asgard without waiting for all the caches to load. This means some of your cloud object lists will be empty for a while until their caches succeed in pulling Amazon data.

To skip the cache fill initialization process, start up Asgard with this JVM system property flag: -DskipCacheFill=true

In Tomcat, you can use skipCacheFill by adding a parameter to the JAVA_OPTS configuration in your apache-tomcat-x/bin/setenv.sh file

    if [ "$1" == "start" ]; then
        export JAVA_OPTS=" \
            -DskipCacheFill=true \
        "
    else
        export JAVA_OPTS=""
    fi

Get a thread dump from Tomcat

If you want to diagnose the state of a misbehaving server, run this command to dump the current threads to catalina.out:

kill -3 `ps -ef | grep java.*tomcat | grep -v grep | awk '{print$2}'` &

Kill Tomcat process if stuck

On Mac or Linux, if shutdown.sh does not successfully stop Tomcat, run this command to kill the Tomcat process:

kill -9 `ps -ef | grep java.*tomcat | grep -v grep | awk '{print$2}'` &

Known Issues

Asgard fails on Java 7

Users have reported this error when running Asgard on Java 7. java.lang.IncompatibleClassChangeError: the number of constructors during runtime and compile time for java.io.PrintWriter do not match. Expected 8 but got 9 Currently Asgard is only tested with Java 6.

Asgard hangs in OpenJDK

Some users have reported problems when running OpenJDK. There have been incompatibilities between Grails and OpenJDK, so switching your JAVA_HOME to Sun/Oracle JDK can fix some problems.