Skip to content

Latest commit

 

History

History
84 lines (56 loc) · 3.29 KB

TROUBLESHOOTING.md

File metadata and controls

84 lines (56 loc) · 3.29 KB
Raw

Troubleshooting

Out of Memory Issues

When debugging OutOfMemoryError, you can set the environment variable $PXF_OOM_DUMP_PATH in ${PXF_CONF}/conf/pxf-env.sh. This results in these flags being added during JVM startup:

-XX:+HeapDumpOnOutOfMemoryError -XX:+HeapDumpPath=$PXF_OOM_DUMP_PATH

Heap dump files are usually large (GBs), so make sure you have enough disk space at $PXF_OOM_DUMP_PATH in case of an OutOfMemoryError.

If $PXF_OOM_DUMP_PATH is a directory, a new java heap dump file will be generated at $PXF_OOM_DUMP_PATH/java_pid<PID>.hprof for each OutOfMemoryError, where <PID> is the process ID of the PXF server instance. If $PXF_OOM_DUMP_PATH is not a directory, a single Java heap dump file will be generated on the first OutOfMemoryError, but will not be overwritten on subsequent OutOfMemoryErrors, so rename files accordingly.

Generate a heap dump for PXF

jmap -dump:live,format=b,file=<your_location>/heap_dump "$(pgrep -f tomcat)"
  • Note: live will force a full garbage collection before dump

Collect JVM Statistics from PXF

The following command will collect Java Virtual Machine Statistics every 60 seconds.

 jstat -gcutil $(pgrep -f tomcat) 60000 > /tmp/jstat_pxf_1min.out &

Restrict PXF Threads

Alternatively you can reduce the maximum number of concurrent requests PXF can handle by setting PXF_MAX_THREADS in ${PXF_CONF}/conf/pxf-env.sh. This would result in reduced number of PXF worker threads and if there are more incoming requests than PXF threads the request would error out as opposed to overloading the PXF JVM's memory.

CLI

Cluster commands do not work from master

You may see the following error:

ERROR: pxf cluster commands should only be run from Greenplum master.

You will either need to change the hostname to match the gp_segment_configuration table entry for the master node or change the master hostname to match the gp_segment_configuration entry. Both locations need to match for the cluster command to succeed.

** Note: this error was fixed in PXF Version 5.3.0

Dataproc

Accessing Dataproc clusters from external network

Dataproc uses the internal IP addresses for the partition locations. We can ask dataproc to use the datanode hostnames instead. We need to set a property in hdfs-site.xml dfs.client.use.datanode.hostname=true.

  • dfs.client.use.datanode.hostname: By default HDFS clients connect to DataNodes using the IP address provided by the NameNode. Depending on the network configuration this IP address may be unreachable by the clients. The fix is letting clients perform their own DNS resolution of the DataNode hostname. The following setting enables this behavior.

    <property>
    <name>dfs.client.use.datanode.hostname</name>
    <value>true</value>
    <description>Whether clients should use datanode hostnames when
        connecting to datanodes.
    </description>
</property>

Dataproc with Kerberos

A kerberized dataproc cluster might start with permission checking turned off. This means that any user will be able to access any file in the HDFS cluster. To enable permission checking modify hdfs-site.xml

  <property>
      <name>dfs.permissions.enabled</name>
      <value>true</value>
  </property>