So I have an on-prem API that writes to an Azure Service Bus queue which has been running without any issues for several months. The appropriate firewall entries are in place and the required ports are open. However we had a network outage recently that prevented our API from writing to the queue, but the ServiceBusSender SendMessageAsync did not throw an exception or fail in any way. We eventually found an exception in App Insights: "System.Net.WebSockets.WebSocketException (0x80004005): Unable to connect to the remote server".

We have a pretty standard Controller => Service => ServiceBusService structure:

AccountController:

public ActionResult<SuccessOkResponse> VerifyRedemption
(
    [FromQuery, Required] string username,
    [FromBody, Required] RedemptionVerificationRequest request
)
{
    try
    {
        Service.VerifyRedemption(username, request, HttpContext.User.Identity?.Name);
        return Ok(new SuccessOkResponse { Success = true });
    }
    catch (Exception exc)
    {
        ProblemDetails problemDetails;
        var exceptionType = exc.GetType();
        // etc.
    }
}

AccountService:

public void VerifyRedemption
(
    string username,
    RedemptionVerificationRequest request,
    string apiConsumerName
)
{
    // Validation code etc.
    // ...

    if (AppSettings.ServiceBusRepayReinvest.ServiceBusEnabled)
    {
        // Write to service bus queue
        ServiceBusService.SendMessage(redemptionRequest);
    }

    // update db etc.
    // ...

}

ServiceBusService:

public async Task SendMessage(HoldingRedemptionRequest payload)
{
    var clientOptions = new ServiceBusClientOptions()
    {
        TransportType = ServiceBusTransportType.AmqpWebSockets
    };
    var connectionString = _appSettings.ServiceBusRepayReinvest.ServiceBusRepayReinvestConnectionString;
    queueName = _appSettings.ServiceBusRepayReinvest.ServiceBusRepayReinvestQueue;
    await using var client = new ServiceBusClient(connectionString, clientOptions);
    ServiceBusSender sender = client.CreateSender(queueName);

    try
    {
        string messagePayload = JsonSerializer.Serialize(payload.RequestId);
        ServiceBusMessage message = new ServiceBusMessage(messagePayload)
        {
            CorrelationId = payload.SSCorrelationId
        };
        await sender.SendMessageAsync(message).ConfigureAwait(false);
        Logger?.LogInformation($"A message ({payload}) has been published to the queue.");
    }
    finally
    {
        // Calling DisposeAsync on client types is required to ensure that network
        // resources and other unmanaged objects are properly cleaned up.
        await sender.DisposeAsync();
        await client.DisposeAsync();
    }
}

To replicate the issue I have disabled the queue in Azure, and when I run the code locally in VS the code runs the await sender.SendMessageAsync(message).ConfigureAwait(false) line as normal with no exception thrown, and completes the rest of the code. I can see an exception in App Insights, but I would expect the code to fail if the queue is disabled. Is there something wrong with how the sender.SendMessageAsync is being called, in terms of async await, or anything else that might be causing this?
Thanks in advance.

So I have an on-prem API that writes to an Azure Service Bus queue which has been running without any issues for several months. The appropriate firewall entries are in place and the required ports are open. However we had a network outage recently that prevented our API from writing to the queue, but the ServiceBusSender SendMessageAsync did not throw an exception or fail in any way. We eventually found an exception in App Insights: "System.Net.WebSockets.WebSocketException (0x80004005): Unable to connect to the remote server".

We have a pretty standard Controller => Service => ServiceBusService structure:

AccountController:

public ActionResult<SuccessOkResponse> VerifyRedemption
(
    [FromQuery, Required] string username,
    [FromBody, Required] RedemptionVerificationRequest request
)
{
    try
    {
        Service.VerifyRedemption(username, request, HttpContext.User.Identity?.Name);
        return Ok(new SuccessOkResponse { Success = true });
    }
    catch (Exception exc)
    {
        ProblemDetails problemDetails;
        var exceptionType = exc.GetType();
        // etc.
    }
}

AccountService:

public void VerifyRedemption
(
    string username,
    RedemptionVerificationRequest request,
    string apiConsumerName
)
{
    // Validation code etc.
    // ...

    if (AppSettings.ServiceBusRepayReinvest.ServiceBusEnabled)
    {
        // Write to service bus queue
        ServiceBusService.SendMessage(redemptionRequest);
    }

    // update db etc.
    // ...

}

ServiceBusService:

public async Task SendMessage(HoldingRedemptionRequest payload)
{
    var clientOptions = new ServiceBusClientOptions()
    {
        TransportType = ServiceBusTransportType.AmqpWebSockets
    };
    var connectionString = _appSettings.ServiceBusRepayReinvest.ServiceBusRepayReinvestConnectionString;
    queueName = _appSettings.ServiceBusRepayReinvest.ServiceBusRepayReinvestQueue;
    await using var client = new ServiceBusClient(connectionString, clientOptions);
    ServiceBusSender sender = client.CreateSender(queueName);

    try
    {
        string messagePayload = JsonSerializer.Serialize(payload.RequestId);
        ServiceBusMessage message = new ServiceBusMessage(messagePayload)
        {
            CorrelationId = payload.SSCorrelationId
        };
        await sender.SendMessageAsync(message).ConfigureAwait(false);
        Logger?.LogInformation($"A message ({payload}) has been published to the queue.");
    }
    finally
    {
        // Calling DisposeAsync on client types is required to ensure that network
        // resources and other unmanaged objects are properly cleaned up.
        await sender.DisposeAsync();
        await client.DisposeAsync();
    }
}

To replicate the issue I have disabled the queue in Azure, and when I run the code locally in VS the code runs the await sender.SendMessageAsync(message).ConfigureAwait(false) line as normal with no exception thrown, and completes the rest of the code. I can see an exception in App Insights, but I would expect the code to fail if the queue is disabled. Is there something wrong with how the sender.SendMessageAsync is being called, in terms of async await, or anything else that might be causing this?
Thanks in advance.

• azure
• asp-core
• async-await
• azureservicebus

Share Improve this question Follow asked Nov 17, 2024 at 13:48 some_randomersome_randomer 33311 gold badge55 silver badges1515 bronze badges 2

• In addition to Sean's observations below, you're also not following documented lifetime guidance for ServiceBusClient and senders. You really don't want to create ephemeral instances to use for a single operation and throw them away. This won't scale well and will cause elevated resource use under load. You'll want to either use the DI extensions to register and inject tour client/senders or create and mange your own singleton instance. see: learn.microsoft/dotnet/azure/sdk/… – Jesse Squire Commented Nov 17, 2024 at 18:49
• Thanks @JesseSquire good point. The feature that uses this won't be operating under heavy load. At the moment the queue operation might be used around 30 to 40 times per day, and ultimately we don't ever see it being called more than 100 times per day. But I will look at going the DI route. – some_randomer Commented Nov 17, 2024 at 20:21

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 2

The reason you didn’t get exceptions in your code is because you had the sending done in fire-and-fet mode. The call to the async method would return faster than the message sending task would take to complete, the controller would complete its work and disposal would take place while sending is still taking place. Depending on various factors, the sending operation would randomly succeed and or fail.

This is a typical problem with asynchronous method invocation over synchronous code path. ServiceBusService.SendMessage() needs to be awaited when called from VerifyRedemption(). And the latter needs to be awaited when invoked from the AccountController controller. Make the controller’s VerifyRedemption to be asynchronous and await the calls to fix this.