Cannot see what groups and tests are declared before tests start running

[bartekpacia]

Alternative title: Declarer knows the test suite structure, but it's private.

Overview

In Dart test files, I'd like to be able to know the test suite structure (the test and groups, and child groups, etc.) before the tests start executing.

lib/testsample.dart

String greet(String name) {
  return 'Hello, $name!';
}

test/testsample_test.dart

import 'package:test/test.dart';
import 'package:test_api/src/backend/declarer.dart';
import 'package:testsample/testsample.dart';

void main() {
  test('smoke_test', () {
    expect(greet('ABC XYZ'), equals('Hello, ABC XYZ!'));
  });

  group('single names', () {
    test('greets 1', () {
      expect(greet('Charlie'), equals('Hello, Charlie!'));
    });

    test('greets 2', () {
      expect(greet('Jack'), equals('Hello, Jack!'));
    });
  });

  group('full names', () {
    test('greets 1', () {
      expect(greet('Charlie Root'), equals('Hello, Charlie Root!'));
    });

    test('greets 2', () {
      expect(greet('Jack Ryan'), equals('Hello, Jack Ryan!'));
    });
  });

  final declarer = Declarer.current;
  if (declarer == null) {
    throw StateError('declarer is null');
  }

  declarer.build(); // impossible - returns a "Group", but it can be called only once, and that single call is made by `package:test` itself
  declarer._entries; // impossible - it's private
}

Why are almost all properties in Declarer private? It's not a public API anyway. Why not make some of them at least @protected, so that developers could extend Declarer and inject it into a Zone, replacing the existing `Declarer?

Workaround

We cannot access the test suite structure with Declarer object before the tests start executing, but once they start executing, inside of the test callback, we have access to current Zone's Invoker object, which gives us a way to learn about all the groups and tests that are declared in the Dart test file being currently executed.

test/testsample_hacky_test.dart

import 'package:test/test.dart';
import 'package:test_api/src/backend/declarer.dart';
import 'package:test_api/src/backend/group.dart';
import 'package:test_api/src/backend/group_entry.dart';
import 'package:test_api/src/backend/invoker.dart';
import 'package:test_api/src/backend/test.dart';
import 'package:testsample/testsample.dart';

void main() {
  test('test_explorer', () {
    final implicitTopLevelGroup = Invoker.current!.liveTest.groups.first;
    _printGroupEntry(implicitTopLevelGroup);
  });

  test('smoke_test', () {
    expect(greet('ABC XYZ'), equals('Hello, ABC XYZ!'));
  });

  group('single names', () {
    test('greets 1', () {
      expect(greet('Charlie'), equals('Hello, Charlie!'));
    });

    test('greets 2', () {
      expect(greet('Jack'), equals('Hello, Jack!'));
    });
  });

  group('full names', () {
    test('greets 1', () {
      expect(greet('Charlie Root'), equals('Hello, Charlie Root!'));
    });

    test('greets 2', () {
      expect(greet('Jack Ryan'), equals('Hello, Jack Ryan!'));
    });
  });

  final declarer = Declarer.current;
  if (declarer == null) {
    throw StateError('declarer is null');
  }
}

/// Prints test entry (either a [Group] or a [Test]).
///
/// If [entry] is a [Group], then its children are recursively printed as well.
void _printGroupEntry(GroupEntry entry, {int level = 0}) {
  final padding = '  ' * level;
  if (entry is Group) {
    print('$padding Group: ${entry.name}');
    for (final groupEntry in entry.entries) {
      _printGroupEntry(groupEntry, level: level + 1);
    }
  } else if (entry is Test) {
    if (entry.name == 'test_explorer') {
      // Ignore the dummy "explorer" test
      return;
    }

    print('$padding Test: ${entry.name}');
  }
}

It's quite hacky, but hey, it works.

$ dart test
00:00 +0: test/testsample_test.dart: test_explorer
 Group:
   Test: smoke_test
   Group: single names
     Test: single names greets 1
     Test: single names greets 2
   Group: full names
     Test: full names greets 1
     Test: full names greets 2
00:00 +6: All tests passed!

Another possible workaround

Create some custom Declarer and override the methods that declare groups and tests:

class CustomDeclarer extends Declarer {

  @override
  void test(String name, Function() body, /*...*/ ) {
    // register test on our own
    super.test(name, body, /*...*/);
  }

  @override
  void group(String name, void Function() body, /*...*/ ) {
    // register group on our own
    super.group(name, body, /*...*/);
  }
}

Then inject it into zoneValues of the current Zone.

I haven't had time to explore this approach further, though, so that's all I've got for now.

Credits: @mateuszwojtczak

More context

I'm working on a test framework for Flutter (link).

I need to know the Dart test suite structure in advance (i.e. before any tests start running) so that I can "register" all of the tests. Why do I need to "register" the tests? So that the native tests of the Flutter app (native tests = Java/Kotlin JUnit tests on Android, Objective-C/Swift XCTests on iOS) can request the execution of individual Dart tests (but to request execution of a test, it has to be first registered for it).

In the perfect world, I'd like to be able to write a webservice like this:

import 'dart:io';

void main() async {
  // assume groups and tests are already defined

  final server = await HttpServer.bind(InternetAddress.anyIPv4, 8080);

  await for (var request in server) {
    if (request.method == 'POST' && request.uri.path == '/run') {
      final dartTestName = request.uri.queryParameters['dartTestName'];
      if (dartTestName == null) {
        throw StateError('duh');
      }

      final testResult  = Invoker.runTestByName(dartTestName) // something like this???
      // TODO: report test results back
    }
  }
}

Why do I need to call Dart tests from native tests? Because there's a wealth of tools that understand these native mobile test frameworks (such as JUnit and XCTest), and almost none of them know that Dart and Flutter exist. So essentially, I'm building a compatibility layer between the world of native mobile tests, and the world of Flutter tests written in Dat. Flutter sorta-kinda provides this compatibility layer in its integration_test package, but it's very barebones and we're building something better.

It's a quite complicated problem and there're many parts to it - to fully understand what I mean, see Flutter issue #115751 (+ read the comments there).

Issues in dart-lang/test might be related:

• Consider to run testrunner as a service and provide an API  #23
• Define a stable-ish API for test_core and test_api #962
• Add an API to do some customization and then directly run tests in process #1310
• Refactor the Reporter API #1311

[jakemac53]

I don't see us making Declarer etc easier to use by external packages - these are intended to be private classes and we don't support their use. I understand you just want to make your product work, but we can really only support the public apis (if we support our private apis, they no longer become private, which has major implications on our versioning, which ultimately has long reaching effects). If you do use these apis please pin to exact versions of any packages you depend on this way to avoid breakage for yourself and your users.

That being said it sounds like what you want is just an official way of running individual tests (on demand). That is a super reasonable request, and the issues you linked are good ones to engage on.