Reduce complexity of QueryLayer after composition

[bkoelman]

Attempts to address the slow query performance reported in #1731.

The reported issue involves a type hierarchy that includes two nested includes for each type in the hierarchy. The number of rows in the type hierarchy is high (various kinds of products), whereas the two nested includes (unit group and unit) are small. Because pagination is applied at every level by default, the SQL query becomes very complex. This can be fixed by disabling pagination in the two nested includes via resource definitions.

This PR simplifies the query using the following changes:

• If pagination is turned off at some depth in the inclusion chain, do not add sort-by-ID at that depth when no sorting is provided.
• If filter/sort/page/fields is not specified at some depth in the inclusion chain, remove the explicit selector that selects everything.

This reduces the query for /PriceGroup?include=products.unitGroup.units from:

Expression tree: [Microsoft.EntityFrameworkCore.Query.EntityQueryRootExpression]
    .AsNoTrackingWithIdentityResolution()
    .Include("Products.UnitGroup.Units")
    .OrderBy(priceGroup => priceGroup.Id)
    .Take(value)
    .Select(
        priceGroup => new PriceGroup
        {
            Id = priceGroup.Id,
            CreatedAt = priceGroup.CreatedAt,
            CreatedById = priceGroup.CreatedById,
            Description = priceGroup.Description,
            IsDeleted = priceGroup.IsDeleted,
            Name = priceGroup.Name,
            UpdatedAt = priceGroup.UpdatedAt,
            UpdatedById = priceGroup.UpdatedById,
            Products = priceGroup.Products
                .OrderBy(commonProduct => commonProduct.Id)
                .Take(value)
                .Select(
                    commonProduct => (commonProduct.GetType() == value)
                        ? (CommonProduct)new ProductAddon
                        {
                            Id = commonProduct.Id,
                            AllowOrder = commonProduct.AllowOrder,
                            CopyProductId = commonProduct.CopyProductId,
                            CreatedAt = commonProduct.CreatedAt,
                            CreatedById = commonProduct.CreatedById,
                            Discriminator = commonProduct.Discriminator,
                            FullDescription = commonProduct.FullDescription,
                            IsEnabled = commonProduct.IsEnabled,
                            IsNew = commonProduct.IsNew,
                            Name = commonProduct.Name,
                            PriceGroupId = commonProduct.PriceGroupId,
                            ScreenshotLinkEnabled = commonProduct.ScreenshotLinkEnabled,
                            ShortDescription = commonProduct.ShortDescription,
                            SupportOptions = commonProduct.SupportOptions,
                            UnitGroupId = commonProduct.UnitGroupId,
                            UpdatedAt = commonProduct.UpdatedAt,
                            UpdatedById = commonProduct.UpdatedById,
                            IsTaxable = commonProduct.IsTaxable,
                            ProductGroupId = commonProduct.ProductGroupId,
                            UniqueCode = commonProduct.UniqueCode,
                            AllowsCustomEndDate = ((StandaloneProduct)commonProduct).AllowsCustomEndDate,
                            TrialDuration = ((StandaloneProduct)commonProduct).TrialDuration,
                            TrialNo = ((StandaloneProduct)commonProduct).TrialNo,
                            UnitGroup = (commonProduct.UnitGroup == null)
                                ? (UnitGroup)null
                                : new UnitGroup
                                {
                                    Id = commonProduct.UnitGroup.Id,
                                    CreatedAt = commonProduct.UnitGroup.CreatedAt,
                                    CreatedById = commonProduct.UnitGroup.CreatedById,
                                    Description = commonProduct.UnitGroup.Description,
                                    IsActive = commonProduct.UnitGroup.IsActive,
                                    Name = commonProduct.UnitGroup.Name,
                                    UpdatedAt = commonProduct.UnitGroup.UpdatedAt,
                                    UpdatedById = commonProduct.UnitGroup.UpdatedById,
                                    Units = commonProduct.UnitGroup.Units
                                        .OrderBy(unit => unit.Id)
                                        .ToHashSet()
                                }
                        }
                        : (commonProduct.GetType() == value)
                            ? (CommonProduct)new Product
                            {
                                Id = commonProduct.Id,
                                AllowOrder = commonProduct.AllowOrder,
                                CopyProductId = commonProduct.CopyProductId,
                                CreatedAt = commonProduct.CreatedAt,
                                CreatedById = commonProduct.CreatedById,
                                Discriminator = commonProduct.Discriminator,
                                FullDescription = commonProduct.FullDescription,
                                IsEnabled = commonProduct.IsEnabled,
                                IsNew = commonProduct.IsNew,
                                Name = commonProduct.Name,
                                PriceGroupId = commonProduct.PriceGroupId,
                                ScreenshotLinkEnabled = commonProduct.ScreenshotLinkEnabled,
                                ShortDescription = commonProduct.ShortDescription,
                                SupportOptions = commonProduct.SupportOptions,
                                UnitGroupId = commonProduct.UnitGroupId,
                                UpdatedAt = commonProduct.UpdatedAt,
                                UpdatedById = commonProduct.UpdatedById,
                                IsTaxable = commonProduct.IsTaxable,
                                ProductGroupId = commonProduct.ProductGroupId,
                                UniqueCode = commonProduct.UniqueCode,
                                AllowsCustomEndDate = ((StandaloneProduct)commonProduct).AllowsCustomEndDate,
                                TrialDuration = ((StandaloneProduct)commonProduct).TrialDuration,
                                TrialNo = ((StandaloneProduct)commonProduct).TrialNo,
                                SupportsMonthlyBillingFrequency = ((Product)commonProduct).SupportsMonthlyBillingFrequency,
                                UnitGroup = (commonProduct.UnitGroup == null)
                                    ? (UnitGroup)null
                                    : new UnitGroup
                                    {
                                        Id = commonProduct.UnitGroup.Id,
                                        CreatedAt = commonProduct.UnitGroup.CreatedAt,
                                        CreatedById = commonProduct.UnitGroup.CreatedById,
                                        Description = commonProduct.UnitGroup.Description,
                                        IsActive = commonProduct.UnitGroup.IsActive,
                                        Name = commonProduct.UnitGroup.Name,
                                        UpdatedAt = commonProduct.UnitGroup.UpdatedAt,
                                        UpdatedById = commonProduct.UnitGroup.UpdatedById,
                                        Units = commonProduct.UnitGroup.Units
                                            .OrderBy(unit => unit.Id)
                                            .ToHashSet()
                                    }
                            }
                            : (commonProduct.GetType() == value)
                                ? (CommonProduct)new ProductBundle
                                {
                                    Id = commonProduct.Id,
                                    AllowOrder = commonProduct.AllowOrder,
                                    CopyProductId = commonProduct.CopyProductId,
                                    CreatedAt = commonProduct.CreatedAt,
                                    CreatedById = commonProduct.CreatedById,
                                    Discriminator = commonProduct.Discriminator,
                                    FullDescription = commonProduct.FullDescription,
                                    IsEnabled = commonProduct.IsEnabled,
                                    IsNew = commonProduct.IsNew,
                                    Name = commonProduct.Name,
                                    PriceGroupId = commonProduct.PriceGroupId,
                                    ScreenshotLinkEnabled = commonProduct.ScreenshotLinkEnabled,
                                    ShortDescription = commonProduct.ShortDescription,
                                    SupportOptions = commonProduct.SupportOptions,
                                    UnitGroupId = commonProduct.UnitGroupId,
                                    UpdatedAt = commonProduct.UpdatedAt,
                                    UpdatedById = commonProduct.UpdatedById,
                                    IsTaxable = commonProduct.IsTaxable,
                                    ProductGroupId = commonProduct.ProductGroupId,
                                    UniqueCode = commonProduct.UniqueCode,
                                    UnitGroup = (commonProduct.UnitGroup == null)
                                        ? (UnitGroup)null
                                        : new UnitGroup
                                        {
                                            Id = commonProduct.UnitGroup.Id,
                                            CreatedAt = commonProduct.UnitGroup.CreatedAt,
                                            CreatedById = commonProduct.UnitGroup.CreatedById,
                                            Description = commonProduct.UnitGroup.Description,
                                            IsActive = commonProduct.UnitGroup.IsActive,
                                            Name = commonProduct.UnitGroup.Name,
                                            UpdatedAt = commonProduct.UnitGroup.UpdatedAt,
                                            UpdatedById = commonProduct.UnitGroup.UpdatedById,
                                            Units = commonProduct.UnitGroup.Units
                                                .OrderBy(unit => unit.Id)
                                                .ToHashSet()
                                        }
                                }
                                : commonProduct)
                .ToHashSet()
        })

to:

Expression tree: [Microsoft.EntityFrameworkCore.Query.EntityQueryRootExpression]
    .AsNoTrackingWithIdentityResolution()
    .Include("Products.UnitGroup.Units")
    .OrderBy(priceGroup => priceGroup.Id)
    .Take(value)
    .Select(
        priceGroup => new PriceGroup
        {
            Id = priceGroup.Id,
            CreatedAt = priceGroup.CreatedAt,
            CreatedById = priceGroup.CreatedById,
            Description = priceGroup.Description,
            IsDeleted = priceGroup.IsDeleted,
            Name = priceGroup.Name,
            UpdatedAt = priceGroup.UpdatedAt,
            UpdatedById = priceGroup.UpdatedById,
            Products = priceGroup.Products
                .OrderBy(commonProduct => commonProduct.Id)
                .Take(value)
                .ToHashSet()
        })

However, this doesn't entirely solve all problems. But so far, I think it's the best we can offer. Because it's not possible to express a selector at a base level.

For example, using the following model:

abstract class Abstract;
class ConcreteBase : Abstract;
sealed class Derived1 : ConcreteBase;
sealed class Derived2 : ConcreteBase;

sealed class Root
{
    [HasMany] ISet<Abstract> Items { get; set; }
}

The following query is produced without any parameters (so all pagination turned off):

var query = source
    .Include(root => root.Items);

But when a parameter is specified somewhere in the type hierarchy (for example, fields for Derived1), we MUST always select the actual type:

var query = source
    .Include(root => root.Items);
    .Select(
        root =>
            root.GetType() == typeof(Derived1)
                ? new Derived1()
                {
                    ... // select subset of fields
                }
                :
            root // matches anything else in the type hierarchy
    );

It is not possible to apply this on a base type (for example, fields for ConcreteBase):

var query = source
    .Include(root => root.Items);
    .Select(
        root =>
            root.GetType() == typeof(ConcreteBase)
                ? new ConcreteBase() // create instance of base type
                {
                    ... // select subset of fields -- PROBLEM: properties on Derived1 and Derived2 are unavailable
                }
                :
            root // matches anything else in the type hierarchy
    );

Thus, the only way to make that work is to expand into separate selectors for ConcreteBase and all of its derived types in the type hierarchy:

var query = source
    .Include(root => root.Items);
    .Select(
        root =>
            root.GetType() == typeof(ConcreteBase)
                ? new ConcreteBase()
                {
                    ... // select subset of fields
                }
                :
            root.GetType() == typeof(Derived1)
                ? new Derived1()
                {
                    ... // select subset of fields (identical to above)
                }
                :
            root.GetType() == typeof(Derived2)
                ? new Derived2()
                {
                     ... // select subset of fields (identical to above)
                }
                :
            root // matches nothing in this case
    );

QUALITY CHECKLIST

• Changes implemented in code
• Complies with our contributing guidelines
• N/A: Adapted tests
• Documentation updated

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.74%. Comparing base (c00f552) to head (259a7bd).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1735      +/-   ##
==========================================
- Coverage   90.76%   90.74%   -0.02%     
==========================================
  Files         468      468              
  Lines       14646    14658      +12     
  Branches     2308     2314       +6     
==========================================
+ Hits        13294    13302       +8     
- Misses        918      920       +2     
- Partials      434      436       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.