Recurring job invocation failing with unexpected status code

[WhitWaldo]

Reported by Ajay on Discord.

Expected Behavior

I expect to see the job be invoked on the recurring basis it was scheduled for with no errors. Using .NET 9 for testing.

Actual Behavior

The MapDaprScheduledJobHandler function only got called once, then it wasn't called again. We saw the following provided log.

jobs: time="2025-02-22T19:49:42.0069463+05:30" level=error msg="failed to invoke schedule app job: unexpected status code returned from app while processing triggered job myJob. status code returned: 2" app_id=jobs instance=dinmHXH4NZ3 scope=dapr.runtime.scheduler.cluster type=log ver=1.15.0-rc.14
jobs: == APP == fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
jobs: == APP == An unhandled exception has occurred while executing the request.
jobs: == APP == System.Threading.Tasks.TaskCanceledException: A task was canceled.
jobs: == APP == at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
jobs: == APP == at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc)
jobs: == APP == at Dapr.Jobs.Extensions.EndpointRouteBuilderExtensions.<>cDisplayClass0_0.<b0>d.MoveNext()
jobs: == APP == --- End of stack trace from previous location ---
jobs: == APP == at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)
jobs: time="2025-02-22T19:49:43.0021227+05:30" level=error msg="unexpected status code returned from app while processing triggered job myJob. status code returned: 2" app_id=jobs instance=dinmHXH4NZ3 scope=dapr.runtime.scheduler.cluster type=log ver=1.15.0-rc.14

Steps to Reproduce the Problem

#pragma warning disable CS0618 // Type or member is obsolete
using System.Text;
using Dapr.Jobs;
using Dapr.Jobs.Extensions;
using Dapr.Jobs.Models;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddDaprJobsClient();
builder.Logging.ClearProviders();
builder.Logging.AddConsole();

var app = builder.Build();

//Set a handler to deal with incoming jobs
var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(15));
app.MapDaprScheduledJobHandler(async (string jobName, ReadOnlyMemory<byte> jobPayload, ILogger? logger, CancellationToken cancellationToken) =>
{
    logger?.LogInformation("Received trigger invocation for job '{jobName}'", jobName);

    var deserializedPayload = Encoding.UTF8.GetString(jobPayload.Span);
    logger?.LogInformation("Received invocation for the job '{jobName}' with payload '{deserializedPayload}'",
        jobName, deserializedPayload);
    await Task.Delay(TimeSpan.FromSeconds(3), cancellationToken);

    return Task.CompletedTask;
}, cancellationTokenSource.Token);

using var scope = app.Services.CreateScope();
var logger = scope.ServiceProvider.GetRequiredService<ILogger<Program>>();
var daprJobsClient = scope.ServiceProvider.GetRequiredService<DaprJobsClient>();

CronExpressionBuilder cronExpressionBuilder = new CronExpressionBuilder();
cronExpressionBuilder.Every(EveryCronPeriod.Second, 10);

logger.LogInformation("Scheduling one-time job 'myJob' to execute 10 seconds from now");
await daprJobsClient.ScheduleJobAsync("myJob", DaprJobSchedule.FromCronExpression(cronExpressionBuilder),
    Encoding.UTF8.GetBytes("This is a test"));
logger.LogInformation("Scheduled one-time job 'myJob'");

app.Run();

#pragma warning restore CS0618 // Type or member is obsolete

Release Note

RELEASE NOTE: FIX Bug preventing recurring jobs from invoking callback function.

[siri-varma]

@WhitWaldo I investigated this issue. Looks like the cancellation token here is causing the issue

dotnet-sdk/src/Dapr.Jobs/Extensions/EndpointRouteBuilderExtensions.cs

Line 53 in 52f0851

payload = await streamContent.ReadAsByteArrayAsync(cancellationToken);

. When I removed the cancellation token it works fine.

[siri-varma]

After further investigation, I think the code is fine. (Used .Net 9 and C# 13)

The problem might be with the snippet in this issue. We are supplying this cancellation token

var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(15));

to the MapDaprScheduledJobHandler. Because the TimeSpan is set to 15 seconds, the token expires causing the task to be cancelled.

On a different node, can a cancellation token apply to MapDaprScheduledJobHandler at all ? Because the cancellation token is at the function level and endpoint mapping is per rest call.