troubleshoot.html.md.erb

---
title: Troubleshooting and FAQs for On-Demand VMware RabbitMQ for Tanzu Application Service
owner: London Services Enablement
---

This topic provides operators with basic troubleshooting techniques and FAQs for
on-demand <%= vars.product_full %>.

## <a id="guid"></a> How to Retrieve a Service Instance GUID

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services/retrieve-guid' %>

## <a id="errors"></a> Troubleshoot Errors

Start here if you are responding to a specific error or error messages.

### <a id="x-product"></a> Common Services Errors

The following errors occur in multiple services:

+ [Failed Installation](#install-fail)
+ [Cannot Create or Delete Service Instances](#cannot-create-delete)
+ [Broker Request Timeouts](#timeouts)
+ [Instance Does Not Exist](#instance-not-exist)
+ [Cannot Bind to or Unbind from Service Instances](#cannot-bind)
+ [Cannot Connect to a Service Instance](#cannot-connect)
+ [Cannot Update a Service Instance](#upgrade-individual-instances)
+ [Upgrade All Service Instances Errand Fails](#upgrade-all-fails)
+ [Missing Logs and Metrics](#missing-logs)

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-err-install-fail' %>

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-err-cannot-create-delete' %>

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-err-timeouts' %>

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-err-instance-not-exist' %>

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-err-other-errors' %>

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-err-cannot-connect' %>

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-err-cannot-update' %>

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-err-upgrade-all-fails' %>

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-err-missing-logs' %>


### <a id="rabbitmQ"></a> <%= vars.product_short %>-Specific Errors

The following troubleshooting errors are specific to <%= vars.product_short %>:

<a name="failed-deployment"></a>
<table class="nice">
  <style>
    main table {
      table-layout: fixed;
    }
  </style>
  <col width="20%">
  <col width="80%">
  <tr>
  <th colspan="2" style="text-align: left;">
    <br>
    Failed Deployment on Upgrade or after Apply Changes
    <br><br>
  </th>
  </tr>
  <tr>
    <th>Symptom</th>
    <td>
      Your deployment fails after editing the <strong>Assign AZs and Networks</strong>
      pane in the <%= vars.product_short %> tile.
    </td>
  </tr>
  <tr>
    <th>Cause</th>
    <td>
      This error might occur if you change the
      IP addresses assigned to the <code>RabbitMQ Server</code> job. <%= vars.product_short %> requires
      that you do not change these IP addresses after they are assigned.
      This includes changes made to your current installation or during an upgrade.
    </td>
  </tr>
  <tr>
    <th style="vertical-align: top; padding-top: 1em;">Solution</th>
    <td>
      To diagnose and solve this issue, see
      <a href="./changing-ips.html">Changing Network or IP Addresses Results in a Failed Deployment</a>
    </td>
  </tr>
</table>

<a name="manual-queues"></a>
<table class="nice">
  <style>
    main table {
      table-layout: fixed;
    }
  </style>
  <col width="20%">
  <col width="80%">
  <tr>
  <th colspan="2" style="text-align: left;">
    <br>
    Pre-Stop Script Times Out When Waiting for Queue Synchronization
    <br><br>
  </th>
  </tr>
  <tr>
    <th>Symptom</th>
    <td>
      A pre-stop script times out with the error message:
      <pre class="terminal">Timed out waiting for mirror queue critical node to sync after 3600 seconds</pre>
    </td>
  </tr>
  <tr>
    <th>Cause</th>
    <td>
      You have not manually synced your queues,
      but you selected the <strong>Wait for Queue Synchronization</strong> checkbox and have
      mirrored queues with the default policy setting <code>ha-sync-mode: manual</code>.
    </td>
  </tr>
  <tr>
    <th style="vertical-align: top; padding-top: 1em;">Solution</th>
    <td>
      To manually sync your queues, run <code>rabbitmqctl sync_queue</code>.
      To set <code>ha-sync-mode</code> to <code>automatic</code> instead, see <a href="./policies-pp.html#viewing">Setting or Changing the Policy</a>.
    </td>
  </tr>
</table>

## <a id="components"></a> Troubleshoot Components

Guidance on checking for and fixing issues in on-demand service components.

### <a id="bosh"></a> BOSH Problems

#### <a id="large-queue"></a>Large BOSH Queue

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-comp-large-queue' %>

### <a id="bosh-config"></a> Configuration

#### <a id="bosh-instance-fail"></a> Service Instances in Failing State

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-comp-bosh-instance-fail' %>

### <a id="auth"></a> Authentication

#### <a id="uaa-change"></a> UAA Changes

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-comp-uaa-change' %>

### <a id="network"></a> Networking

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-comp-network' %>

#### <a id="broker-to-instances"></a> Validate Service Broker Connectivity to Service Instances

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-comp-broker-to-instances' %>

#### <a id="app-to-instances"></a> Validate App Access to Service Instance

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-comp-app-to-instances' %>

### <a id="quotas"></a> Quotas

#### <a id="plan-quotas"></a> Plan Quota Issues

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-comp-plan-quotas' %>

#### <a id="global-quotas"></a> Global Quota Issues

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-comp-global-quotas' %>

### <a id="failing-jobs"></a> Failing Jobs and Unhealthy Instances

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-comp-failing-jobs' %>

## <a id="techniques"></a> Techniques for Troubleshooting

This section contains instructions on interacting with the on-demand service
broker and on-demand service instance BOSH deployments, and on performing general
maintenance and housekeeping tasks.

### <a id="parse-error"></a> Parse a Cloud Foundry (CF) Error Message

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-tech-parse-error' %>

### <a id="bosh-cf-access"></a> Access Broker and Instance Logs and VMs

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-tech-bosh-cf-access' %>

#### <a id="access-broker"></a> Access Broker Logs and VMs

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-tech-access-broker' %>

#### <a id="access-instance"></a> Access Service Instance Logs and VMs

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-tech-access-instance' %>

### <a id="broker-errands"></a> Run Service Broker Errands to Manage Brokers and Instances

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-tech-broker-errands' %>

#### <a id="register-broker"></a> Register Broker

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-tech-register-broker' %>

#### <a id="deregister-broker"></a> Deregister Broker

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-tech-deregister-broker' %>

#### <a id="upgrade-all"></a> Upgrade All Service Instances

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-tech-upgrade-all' %>

#### <a id="delete-all"></a> Delete All Service Instances

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-tech-delete-all' %>

#### <a id="detect-orphans"></a> Detect Orphaned Service Instances

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-tech-detect-orphans' %>

### <a id="instance-creds"></a> Get Admin Credentials for a Service Instance

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-tech-instance-creds' %>

### <a id="reinstall"></a> Reinstall a Tile

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-tech-reinstall' %>

### <a id="view-resources"></a> View Resource Saturation and Scaling

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-tech-view-resources' %>

### <a id="id-instance-owner"></a> Identify Apps using a Service Instance

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-tech-id-instance-owner' %>

### <a id="monitor-quota"></a> Monitor the Quota Saturation and Service Instance Count

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-tech-monitor-quota' %>

### <a id="drop-restore-traffic"></a> Drop and Restore AMQP or AMQPS Traffic to a <%= vars.product_short %> Service Instance

While debugging a <%= vars.product_short %> service instance, you can prevent apps from
sending and receiving messages, for example, to decrease the server load.
You can use <code>drop-amqp-traffic</code> and <code>restore-amqp-traffic</code> scripts,
which run the necessary <code>iptables</code> commands to achieve that.<br><br>

To stop and then restore traffic to a <%= vars.product_short %> service instance, do the following:<br>
<ol>
    <li> To stop all AMQP or AMQPS traffic to a <%= vars.product_short %> service instance, enter the following command:
    <pre>
    bosh -d service-instance_GUID ssh rabbitmq-server
    "echo y | sudo /var/vcap/packages/rabbitmq-admin/bin/drop-amqp-traffic"
    </pre>
    </li>
    <li> After performing the troubleshooting steps, restore the traffic.
         To do this, enter the following command:
    <pre>
    bosh -d service-instance_GUID ssh rabbitmq-server
    "echo y | sudo /var/vcap/packages/rabbitmq-admin/bin/restore-amqp-traffic"
    </pre>
    </li>
</ol>

Alternatively, you can run these scripts on individual nodes:

<ol>
    <li><code>bosh ssh</code> to a rabbitmq-server instance.</li>
    <li><code>sudo -s</code> to gain root privileges.</li>
    <li>Execute <code>drop-amqp-traffic</code> to drop all AMQP or AMQPS traffic to this
        instance, or <code>restore-amqp-traffic</code> to start accepting traffic again.</li>
</ol>

### <a id="service-gateway"></a> Troubleshoot Service-Gateway Access

Service-gateway access leverages the <%= vars.app_runtime_abbr %> TCP Router.
Each service instance and protocol for which service-gateway access is enabled corresponds
to a route in the TCP Router.

When debugging problems with service-gateway access:

1. See [Troubleshooting TCP Routing](https://docs.pivotal.io/application-service/adminguide/troubleshooting_tcp_routing.html)
  in the <%= vars.app_runtime_abbr %> documentation.


## <a id="faq"></a> Frequently Asked Questions

### <a id="check-deploy"></a> What should I check before deploying a new version of the tile?

Ensure that all nodes in the cluster are healthy from the RabbitMQ Management UI,
or health metrics exposed through Firehose.
You cannot rely solely on the BOSH <code>instances</code> output as that reflects the state
of the Erlang VM used by RabbitMQ and not the RabbitMQ app.


### <a id="stopstart"></a> What is the correct way to stop and start <%= vars.product_short %>?

Only BOSH commands should be used by the operator to interact with the RabbitMQ app.
<br><br>
For example:
<br><br>
<code>bosh stop rabbitmq-server</code> and <code>bosh start rabbitmq-server</code>.
<br><br>
There are BOSH job lifecycle hooks which are only fired when rabbitmq-server is
stopped through BOSH.
You can also stop individual instances by running the stop command and specifying
<code>JOB [index]</code>.

<p class="note"><strong>Note:</strong> Do not use <code>monit stop rabbitmq-server</code> as this does not call the drain scripts.
</p>

### <a id="boshstop"></a> What happens when I run bosh stop rabbitmq-server?

BOSH starts the shutdown sequence from the bootstrap instance.
<br><br>
We start by telling the RabbitMQ app to shutdown and then shutdown the
Erlang VM within which it is running. If this succeeds, we run the following
checks to ensure that the RabbitMQ app and Erlang VM have stopped:
<ol>
    <li>
        If <code>/var/vcap/sys/run/rabbitmq-server/pid</code> exists, check that
        the PID inside this file does not point to a running Erlang VM process.
        Notice that we are tracking the Erlang PID and not the RabbitMQ PID.
    </li>
    <li>Check that <code>rabbitmqctl</code> does not return an Erlang VM PID.</li>
</ol>
Once this completes on the bootstrap instance, BOSH continues the same
sequence on the next instance. All remaining rabbitmq-server instances
stop one by one.

### <a id="serverfailprocess"></a> What happens when bosh stop rabbitmq-server fails?

If the BOSH <code>stop</code> fails, you will likely get an error saying that the drain
script failed with:

<pre class="terminal">result: 1 of 1 drain scripts failed. Failed Jobs: rabbitmq-server.</pre>

### <a id="serverfailactions"></a> What do I do when bosh stop rabbitmq-server fails?

The drain script logs to <code>/var/vcap/sys/log/rabbitmq-server/drain.log</code>. If you
have a remote syslog configured, this appears as the <code>rmq_server_drain</code>
program.
<br><br>
First, BOSH <code>ssh</code> into the failing rabbitmq-server instance and start the
rabbitmq-server job by running <code>monit start rabbitmq-server</code>. You will not be
able to start the job with BOSH <code>start</code> as this always runs the drain script first
and will fail as the drain script is failing.
<br><br>
Once rabbitmq-server job is running (confirm this with <code>monit status</code>), run <code>DEBUG=1</code>
<code>/var/vcap/jobs/rabbitmq-server/bin/drain</code>. This tells you exactly why it is
failing.

### <a id="backup"></a> How can I manually back up the state of the RabbitMQ cluster?

It is possible to back up the state of a RabbitMQ cluster for both the on-demand
and pre-provisioned services using the RabbitMQ Management API.
Backups include virtual hosts, exchanges, queues, and users.

#### Back up Manually

<ol>
    <li>Log in to the RabbitMQ Management UI as the admin user you created.
      For instructions about how to do so with or without OAuth enabled, see
      <a href="./managing.html">Using the RabbitMQ Management UI</a>.</li>
    <li>Select <strong>export definitions</strong> from the main page.</li>
</ol>

#### Back up and Restore with a Script

Use the API to run scripts with code similar to the following:

<ul>
    <li>
        <strong>For the backup:</strong>
        <pre>
curl -u "$USERNAME:$PASSWORD" "http://$RABBIT-ADDRESS:15672/api/definitions"
-o "$BACKUP-FOLDER/rabbit-backup.json"
        </pre>
    </li>
    <li>
        <strong>For the restore:</strong>
        <pre>
curl -u "$USERNAME:$PASSWORD" "http://$RABBIT-ADDRESS:15672/api/definitions"
-X POST -H "Content-Type: application/json" -d
"@$BACKUP-FOLDER/rabbit-backup.json"
        </pre>
    </li>
</ul>

### <a id="preupgrade"></a> What pre-upgrade checks should I do?

Before doing any upgrade of RabbitMQ, VMware recommends checking the following:
<ol>
    <li>
      In <%= vars.ops_manager %> check that the status of all of the instances is healthy.
    </li>
    <li>
      Log in to the RabbitMQ Management UI. For instructions about how to do so with or without
      OAuth enabled, see <a href="./managing.html">Using the RabbitMQ Management UI</a>.
    </li>
    <li>
      Check that no alarms have been triggered and that all nodes display as green, showing they
      are healthy.
    </li>
    <li>
        Check that the system is not close to hitting either the memory or disk alarm.
        Do this by looking at what has been consumed by each node in the RabbitMQ Management UI.
    </li>
</ol>

## <a id="kb"></a> Knowledge Base (Community)

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-kb' %>

## <a id="support"></a> File a Support Ticket

<%# The below partial is in https://github.com/pivotal-cf/docs-partials %>

<%= partial vars.path_to_partials + '/services-tshoot/tshoot-support' %>