I have a weird scenario. I have a list of groups. For each group, I need to start a task for all the contents, but I need to make sure the last value in each group is run last. Simplified function below:

public async Task DoAllTheWork(IEnumerable<IGrouping<string, string>> groups)
{
    var allTasks = new List<Task>();
    foreach (var group in groups)
    {
        var tasksInGroup = new List<Task>();
        var values = group.ToList();

        // start tasks for all except last value
        foreach (var value in values.SkipLast(1))
        {
            var theTask = SomeAsyncFunction(value);

            allTasks.Add(theTask);
            tasksInGroup.Add(theTask);
        }

        // start a task for last value once all previous tasks are done
        var lastTask = Task.Run(async () =>
        {
            await Task.WhenAll(tasksInGroup);
            SomeAsyncFunction(values.Last());
        });

        allTasks.Add(lastTask);
    }

    await Task.WhenAll(allTasks);
}

This works fine. The issue is that all the tasks are run at once, which may end up being a problem if there are too many groups or the groups have too many values.

I'd like to limit the # of concurrent tasks being run. The easiest way I can think of is that if allTasks contained a list of tasks that haven't actually been started yet. Then I could just do something like:

Parallel.ForEach(allTasks, new ParallelOptions {MaxDegreeOfParallelism = 10},
aTask =>
{
     aTask.Start();
});

Is there a way to do something like that?

I have a weird scenario. I have a list of groups. For each group, I need to start a task for all the contents, but I need to make sure the last value in each group is run last. Simplified function below:

public async Task DoAllTheWork(IEnumerable<IGrouping<string, string>> groups)
{
    var allTasks = new List<Task>();
    foreach (var group in groups)
    {
        var tasksInGroup = new List<Task>();
        var values = group.ToList();

        // start tasks for all except last value
        foreach (var value in values.SkipLast(1))
        {
            var theTask = SomeAsyncFunction(value);

            allTasks.Add(theTask);
            tasksInGroup.Add(theTask);
        }

        // start a task for last value once all previous tasks are done
        var lastTask = Task.Run(async () =>
        {
            await Task.WhenAll(tasksInGroup);
            SomeAsyncFunction(values.Last());
        });

        allTasks.Add(lastTask);
    }

    await Task.WhenAll(allTasks);
}

This works fine. The issue is that all the tasks are run at once, which may end up being a problem if there are too many groups or the groups have too many values.

I'd like to limit the # of concurrent tasks being run. The easiest way I can think of is that if allTasks contained a list of tasks that haven't actually been started yet. Then I could just do something like:

Parallel.ForEach(allTasks, new ParallelOptions {MaxDegreeOfParallelism = 10},
aTask =>
{
     aTask.Start();
});

Is there a way to do something like that?

• c#
• asynchronous
• parallel-processing

Share Improve this question Follow asked 7 hours ago pseudodevpseudodev 13311 silver badge88 bronze badges 2

• most if not all Async functions give you a "hot" task that is already running. – Ivan Petrov Commented 6 hours ago
• 2 Throw in a SemaphoreSlim to await / release. Either inside SomeAsyncFunction or a lamda... – Jeremy Lakeman Commented 4 hours ago

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

You've envisioned a wrong solution for the problem. This approach won't do what you think:

Parallel.ForEach(allTasks, new ParallelOptions {MaxDegreeOfParallelism = 10},
aTask =>
{
     aTask.Start();
});

This will limit the concurrency of the starting of the tasks only. Starting a task with Start is practically instantaneous, since all it does is to schedule the task's execution to the default TaskScheduler, which is the ThreadPool. So the SomeAsyncFunction isn't going to be throttled. The correct solution is to use the Parallel.ForEach, or even better the Parallel.ForEachAsync, to run the SomeAsyncFunction directly, without explicit task management. The Parallel class uses tasks internally, so that you don't have to do yourself (that's why the overall technology is called Task Parallel Library).

public async Task DoAllTheWork(IEnumerable<IGrouping<string, string>> groups)
{
    IEnumerable<string> firstValues = groups.SelectMany(g => g.SkipLast(1));
    IEnumerable<string> lastValues = groups.Select(g => g.Last());

    ParallelOptions options = new()
    {
        MaxDegreeOfParallelism = 10
    };

    await Parallel.ForEachAsync(firstValues, options, (value, _) =>
    {
        await SomeAsyncFunction(value);
    }).ConfigureAwait(false);

    await Parallel.ForEachAsync(lastValues, options, (value, _) =>
    {
        await SomeAsyncFunction(value);
    }).ConfigureAwait(false);
}

This solution is not perfect because you'll lose some concurrency while the firstValues are finishing and before the lastValues are started, but the effect of this imperfection should be small. Implementing a perfect solution, where the MaxDegreeOfParallelism is enforced correctly from start to end, is not trivial.