Transient error StatusCode.UNAVAILABLE encountered while exporting logs

[gilbertobr]

I have an Opentelemetry that is running on Kubernetes on port 80 or 443.
https://artifacthub.io/packages/helm/opentelemetry-helm/opentelemetry-collector

But when I try to send a log via grpc using the python sdk (https://github.com/open-telemetry/opentelemetry-python), I am getting this warning, and the log is not sent:

WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.UNAVAILABLE encountered while exporting logs, retrying in 1s.
WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.UNAVAILABLE encountered while exporting logs, retrying in 2s.
WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.UNAVAILABLE encountered while exporting logs, retrying in 4s.
WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.UNAVAILABLE encountered while exporting logs, retrying in 8s.
WARNING:opentelemetry.exporter.otlp.proto.grpc.exporter:Transient error StatusCode.UNAVAILABLE encountered while exporting logs, retrying in 16s.

When I send the LOG via POSTMAN, I can send it normally.
Have you been through this?

script test:

import logging

from opentelemetry import trace
from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import (
    OTLPLogExporter,
)
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor,
    ConsoleSpanExporter,
)

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(ConsoleSpanExporter())
)

logging.basicConfig(level=logging.DEBUG)

logger_provider = LoggerProvider(
    resource=Resource.create(
        {
            "service.name": "shoppingcart",
            "service.instance.id": "instance-12",
        }
    ),
)
set_logger_provider(logger_provider)

exporter = OTLPLogExporter(endpoint="grpc.otel-collector.example.com:80", insecure=True, timeout=20)
logger_provider.add_log_record_processor(BatchLogRecordProcessor(exporter))
handler = LoggingHandler(level=logging.NOTSET, logger_provider=logger_provider)

# Attach OTLP handler to root logger
logging.getLogger().addHandler(handler)

# Log directly
logging.info("Jackdaws love my big sphinx of quartz.")

# Create different namespaced loggers
logger1 = logging.getLogger("myapp.area1")
logger2 = logging.getLogger("myapp.area2")

logger1.debug("Quick zephyrs blow, vexing daft Jim.")
logger1.info("How quickly daft jumping zebras vex.")
logger2.warning("Jail zesty vixen who grabbed pay from quack.")
logger2.error("The five boxing wizards jump quickly.")


# Trace context correlation
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("foo"):
    # Do something
    logger2.error("Hyderabad, we have a major problem.")

logger_provider.shutdown()

pip show

backoff                                2.2.1
grpcio                                 1.53.0
grpcio-tools                           1.53.0
opentelemetry-api                      1.17.0
opentelemetry-exporter-otlp            1.17.0
opentelemetry-exporter-otlp-proto-grpc 1.17.0
opentelemetry-exporter-otlp-proto-http 1.17.0
opentelemetry-instrumentation          0.38b0
opentelemetry-instrumentation-grpc     0.38b0
opentelemetry-proto                    1.17.0
opentelemetry-sdk                      1.17.0
opentelemetry-semantic-conventions     0.38b0

[srikanthccv]

But when I try to send a log via grpc using the python sdk (https://github.com/open-telemetry/opentelemetry-python), I am getting this warning, and the log is not sent

When I send the LOG via POSTMAN, I can send it normally.
Have you been through this?

When you send from POSTMAN you are most likely sending using HTTP. The example shared uses gRPC. Please make sure you use the gRPC port.

[gilbertobr]

Exactly, for those who need to do GRPC tests via POSTMAN, follow below how to configure.

https://blog.postman.com/postman-now-supports-grpc/

[srikanthccv]

Can I assume your question is resolved and close this?

[gilbertobr]

I believe the problem is not that.
And yes in the route that the sdk is making the request.
But I'm still investigating.

I noticed that there is a closed PR in the opentelemetry-collector project, but not resolved.
The problem has followed with other people.

[srikanthccv]

The example uses the gRPC exporter, which will not work with the HTTP endpoint. Either use the HTTP exporter or give a valid gRPC endpoint. This is not an issue with the SDK/Exporter but rather config/ network. Since this is not a bug and there is no action item, I am converting this to a GitHub discussion.

[prashanth-nani]

Another reason that this could happen is due to the memory_limiter processor configuration in the OpentelemetryCollector config. We were hitting the soft limit and the collector rejected the data with the WARNING: Transient error StatusCode.UNAVAILABLE encountered while exporting logs. Once the spike_limit_mib and limit_mib values are adjusted, the warnings are gone. The soft limit is calculated as limit_mib - spike_limit_mib. Make sure that you are not hitting the limit.