In both .NET 4 and .NET 4.5, PLINQ supports enumerables with up to Int32.MaxValue elements. Beyond that limit, PLINQ will throw an overflow exception. LINQ to Objects itself has this limitation with certain query operators (such as the indexed Select operator which counts the elements processed), but PLINQ has it with more.<\/p>\n
This limitation impacts so few scenarios that it’s relatively benign and I rarely hear about it. That said, it does come up now and again, and I was in fact asked about it earlier this week. So, to help in case anyone else runs into this…<\/p>\n
There is a relatively straightforward workaround you can apply if you do run up against this limit: ensure the enumerable passed to PLINQ has no more than Int32.MaxValue. That answer might sound flippant and impractical, but in reality it’s often easily accomplished by batching a longer enumerable into multiple shorter enumerables, where none of the shorter enumerables exceed the limit.<\/p>\n
Consider a basic query like the following:<\/p>\n
\nIEnumerable<Output> outputs =
from input in inputs.AsParallel()
where Filter(input)
select Map(input);<\/font><\/p>\n<\/blockquote>\nIf the inputs<\/em> enumerable has more than Int32.MaxValue elements, this will result in an overflow. To workaround this, imagine if we had a Batch<\/em> extension method for IEnumerable<T> that would partition the IEnumerable<T> into multiple sequential IEnumerable<T> instances, each of which had no more than Int32.MaxValue elements. With that, we could rewrite this as follows:<\/p>\n
\nIEnumerable<Output> outputs = inputs.Batch(Int32.MaxValue).SelectMany(batch =>
from input in batch.AsParallel()
where Filter(input)
select Map(input)
);<\/font><\/p>\n<\/blockquote>\nThe original query remains intact, except that instead of operating on the original inputs<\/em> enumerable, it’s operating on the new batch<\/em> enumerable, which is a subset provided by the Batch method. We’ll now be parallelizing our processing of each sequential batch, moving on to the next batch to be processed in parallel once we finish the previous one.<\/p>\n
Such a Batch<\/em> method doesn’t actually exist built-in to .NET 4.5, but writing one is relatively straightforward. Here’s one implementation which you could customize as your needs demand:<\/p>\n
\nstatic class MyLinqExtensions
{
public static IEnumerable<IEnumerable<T>> Batch<T>(
this IEnumerable<T> source, int batchSize)
{
using (var enumerator = source.GetEnumerator())
while (enumerator.MoveNext())
yield return YieldBatchElements(enumerator, batchSize – 1);
} <\/p>\nprivate static IEnumerable<T> YieldBatchElements<T>(
IEnumerator<T> source, int batchSize)
{
yield return source.Current;
for (int i = 0; i < batchSize && source.MoveNext(); i++)
yield return source.Current;
}
}<\/font><\/p>\n<\/blockquote>\n","protected":false},"excerpt":{"rendered":"In both .NET 4 and .NET 4.5, PLINQ supports enumerables with up to Int32.MaxValue elements. Beyond that limit, PLINQ will throw an overflow exception. LINQ to Objects itself has this limitation with certain query operators (such as the indexed Select operator which counts the elements processed), but PLINQ has it with more. This limitation impacts […]<\/p>\n","protected":false},"author":360,"featured_media":58792,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[7908],"tags":[7907,7925,7909,7910],"class_list":["post-55991","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-pfxteam","tag-net-4","tag-net-4-5","tag-parallel-extensions","tag-plinq"],"acf":[],"blog_post_summary":"
In both .NET 4 and .NET 4.5, PLINQ supports enumerables with up to Int32.MaxValue elements. Beyond that limit, PLINQ will throw an overflow exception. LINQ to Objects itself has this limitation with certain query operators (such as the indexed Select operator which counts the elements processed), but PLINQ has it with more. This limitation impacts […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/55991","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/users\/360"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/comments?post=55991"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/55991\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media\/58792"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media?parent=55991"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/categories?post=55991"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/tags?post=55991"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}