Merge pull request #719 from square/sedwards/update-benchmarks-readme

Add Baseline and Perf Tests to Perf Poetry

Author: steve-the-edwards

.github/workflows/kotlin.yml

@@ -123,7 +123,7 @@ jobs:
           api-level: ${{ matrix.api-level }}
           arch: x86_64
           # Skip the benchmarks as this is running on emulators
-          script: ./gradlew connectedCheck -x :benchmarks:dungeon-benchmark:connectedReleaseAndroidTest --no-daemon --stacktrace
+          script: ./gradlew connectedCheck -x :benchmarks:dungeon-benchmark:connectedCheck -x :benchmarks:performance-poetry:complex-benchmark:connectedCheck --no-daemon --stacktrace
 
       - name: Upload results
         if: ${{ always() }}

RELEASING.md

@@ -8,7 +8,7 @@
 1. Confirm that the kotlin build is green before committing any changes
    (Note we exclude benchmarks, but you can check those too!)
    ```bash
-   ./gradlew build && ./gradlew connectedCheck -x :benchmarks:dungeon-benchmark:connectedCheck
+   ./gradlew build && ./gradlew connectedCheck -x :benchmarks:dungeon-benchmark:connectedCheck -x :benchmarks:performance-poetry:complex-benchmark:connectedCheck
    ```
 
 1. Update your tags.

benchmarks/README.md

@@ -1,22 +1,29 @@
 # benchmarks
 
-This module contains benchmarks. Used to measure and help improve Workflow performance. These tests
-should be run on physical devices.
+This module contains benchmarks. Used to measure and help improve Workflow performance. The
+deterministic tests - such as those for render passes - can be run on any device (virtual or
+otherwise), but the benchmarks should be run on physical devices, or the closest approximation
+to physical devices you can get.
 
-## dungeon-benchmark
+## Baseline Profiles
+
+The sample apps can be used to extract 'baseline profiles' to improve code loading time after first
+install. See [baseline profiles](https://developer.android.com/studio/profile/baselineprofiles).
+
+We can use
+[dungeon](dungeon-benchmark/src/main/java/com/squareup/sample/dungeon/benchmark/DungeonGatherBaselineProfile.kt)
+or [performance-poetry](performance-poetry/complex-benchmark/src/main/java/com/squareup/benchmarks/performance/poetry/complex/benchmark/ComplexPoetryGatherBaseline.kt)
+to gather profiles. Performance Poetry has more adaptability and better benchmarks at this time, so
+focus on that.
 
-This is for the /samples/dungeon/app. This includes a macrobenchmark
-[test](dungeon-benchmark/src/main/java/com/squareup/sample/dungeon/benchmark/WorkflowBaselineProfiles.kt)
-that exercises this sample app (with UiAutomator) to collect
-a [baseline profile](https://developer.android.com/studio/profile/baselineprofiles). After running
-this the `profile.txt` file can be taken off of the device and used directly in the /src/main
-directory of the application. This will include the profile into the APK for guided optimization at
-install time.
+After extracting the profile with one of those tests, the `profile.txt` file can be taken off of the
+device and used directly in the /src/main directory of the application. This will include the
+profile into the APK for guided optimization at install time.
 
 Better yet, the profile can be split up and added into each Workflow module's src directory so that
 it will be included with all APKs built using Workflow (including 3rd party). To do this a java
-program available in the android-x open source [code](https://github.com/androidx/androidx) is used
-to split the profile based on src paths. That tool is found at
+program, available in the Android-X open source [code](https://github.com/androidx/androidx), is
+used to split the profile based on src paths. That tool is found at
 [androidx-main/frameworks/support/out/androidx/development/splitBaselineProfiles].
 
 To split the profile paths for Workflow, the tool can be used as follows:
@@ -29,7 +36,22 @@ This will create an output file separated by module and then also by package as
 profile for each module can be added into its /src/main directory as `baseline-prof.txt`. Then on a
 release build this will be included with the resulting APK/binary.
 
-The other [test](dungeon-benchmark/src/main/java/com/squareup/sample/dungeon/benchmark/WorkflowBaselineBenchmark.kt)
+## dungeon-benchmark
+
+These are benchmarks for the [../samples/dungeon] app. Please instead use performance-poetry where
+possible.
+
+*PLEASE NOTE:* The dungeon app includes AI ghosts with random travel which can 'eat' the player,
+ending the game with a dialog. This obviously makes these test non-deterministic. We used these
+originally to exercise enough of Workflow for a baseline profile. We can avoid at least the dialog
+popping to end the game by making the player survive with the following change to
+[../samples/dungeon/common/src/main/java/com/squareup/sample/dungeon/Game.kt]:
+
+```kotlin
+val isPlayerEaten: Boolean get() = false // Never Eaten!
+```
+
+The [benchmark](dungeon-benchmark/src/main/java/com/squareup/sample/dungeon/benchmark/DungeonStartupBenchmark.kt)
 is used to verify the results of including the baseline profiles on the startup time. This runs the
 same scenario with and without forcing the use of the profiles. To force the use of profiles, the
 `libs.androidx.profileinstaller` dependency is included into the app under profile (dungeon in this
@@ -41,11 +63,38 @@ Module of code for performance testing related to poetry applications.
 
 ### complex-poetry
 
-This application is a modification of the samples/containers/app-poetry app which uses also the
+This application is a modified version of the samples/containers/app-poetry app which also uses the
 common components in samples/containers/common and samples/containers/poetry. It modifies this
 application to pass the Workflow
-a [SimulatedPerfConfig](performance-poetry/complex-poetry/src/main/java/com/squareup/benchmarks/performance/poetry/SimulatedPerfConfig.kt).
+a [SimulatedPerfConfig.](performance-poetry/complex-poetry/src/main/java/com/squareup/benchmarks/performance/complex/poetry/instrumentation/SimulatedPerfConfig.kt)
 
 In this case we specify that the app should be more 'complex' which adds delays into each of the
 selections that are run by Worker's which then trigger a loading state that is handled by the
-[MaybeLoadingGatekeeperWorkflow](performance-poetry/complex-poetry/src/main/java/com/squareup/benchmarks/performance/poetry/MaybeLoadingGatekeeperWorkflow.kt).
+[MaybeLoadingGatekeeperWorkflow.](performance-poetry/complex-poetry/src/main/java/com/squareup/benchmarks/performance/complex/poetry/MaybeLoadingGatekeeperWorkflow.kt)
+
+One benchmark that is included (and run in CI) as a UI test is the [RenderPassTest.](performance-poetry/complex-poetry/src/androidTest/java/com/squareup/benchmarks/performance/complex/poetry/RenderPassTest.kt)
+This measures two things:
+
+ 1. The number of "Render Passes" that the Raven scenario triggers.
+ 1. The ratio of 'fresh renderings' to 'stale renderings' in the Raven scenario.
+
+A rendering is 'fresh' if the node's state has changed. A rendering is 'stale' if its state is the
+same. Note that Workflow is designed to to have cheap, idempotent renderings and the fresh rendering
+ratio will never be 1.0 by design. However, if there are smells that have led to 'render churn' one
+will see a very poor rendering ratio and we would like a way to track that and to test that it is
+constant (or improving!).
+
+This module includes an [instrumentation package](performance-poetry/complex-poetry/src/main/java/com/squareup/benchmarks/performance/complex/poetry/instrumentation)
+that has two [WorkflowInterceptor]s that can count render passes or instrument perfetto Trace
+sections, as well as data class for tracking those.
+
+This module also includes a [robots package](performance-poetry/complex-poetry/src/main/java/com/squareup/benchmarks/performance/complex/poetry/robots)
+that provides some utility helper 'robots' for the UiAutomator [androidx.test.uiautomator.UiDevice]
+as well as scenarios specific to this Complex Poetry application.
+
+### complex-benchmark
+
+This is an Android Test module which hosts an application that can run androidx.macrobenchmarks.
+See the kdoc on [ComplexPoetryBenchmarks.](performance-poetry/complex-benchmark/src/main/java/com/squareup/benchmarks/performance/complex/poetry/benchmark/ComplexPoetryBenchmarks.kt)
+
+The results for this are stored in the same folder at [ComplexPoetryResults.txt.](performance-poetry/complex-benchmark/src/main/java/com/squareup/benchmarks/performance/complex/poetry/benchmark/ComplexPoetryResults.txt)

benchmarks/dungeon-benchmark/src/main/java/com/squareup/sample/dungeon/benchmark/BaselineProfileBenchmarkResults.txt renamed to benchmarks/dungeon-benchmark/src/main/java/com/squareup/sample/dungeon/benchmark/DungeonBaselineProfileBenchmarkResults.txt

@@ -23,24 +23,21 @@ import kotlin.time.ExperimentalTime
  */
 @RunWith(AndroidJUnit4::class)
 @OptIn(ExperimentalBaselineProfilesApi::class)
-public class WorkflowBaselineProfiles {
+class DungeonGatherBaselineProfile {
 
-  @get:Rule
-  public val baselineProfileRule: BaselineProfileRule = BaselineProfileRule()
+  @get:Rule val baselineProfileRule: BaselineProfileRule = BaselineProfileRule()
 
   private lateinit var context: Context
   private lateinit var device: UiDevice
 
-  @Before
-  public fun setUp() {
+  @Before fun setUp() {
     val instrumentation = InstrumentationRegistry.getInstrumentation()
     context = ApplicationProvider.getApplicationContext()
     device = UiDevice.getInstance(instrumentation)
   }
 
   @Test
-  @OptIn(ExperimentalTime::class)
-  public fun baselineProfiles() {
+  @OptIn(ExperimentalTime::class) fun baselineProfiles() {
     baselineProfileRule.collectBaselineProfile(
       packageName = PACKAGE_NAME,
     ) {
@@ -50,9 +47,9 @@ public class WorkflowBaselineProfiles {
     }
   }
 
-  public companion object {
+  companion object {
 
-    public fun openMazeAndNavigate(device: UiDevice) {
+    fun openMazeAndNavigate(device: UiDevice) {
       val boardsList =
         UiScrollable(UiSelector().className("androidx.recyclerview.widget.RecyclerView"))
       boardsList.waitForExists(UI_TIMEOUT_MS)
@@ -70,7 +67,7 @@ public class WorkflowBaselineProfiles {
       }
     }
 
-    public const val PACKAGE_NAME: String = "com.squareup.sample.dungeon"
-    public const val UI_TIMEOUT_MS: Long = 2000L
+    const val PACKAGE_NAME: String = "com.squareup.sample.dungeon"
+    const val UI_TIMEOUT_MS: Long = 2000L
   }
 }

benchmarks/dungeon-benchmark/src/main/java/com/squareup/sample/dungeon/benchmark/WorkflowBaselineProfiles.kt renamed to benchmarks/dungeon-benchmark/src/main/java/com/squareup/sample/dungeon/benchmark/DungeonGatherBaselineProfile.kt

@@ -3,15 +3,14 @@ package com.squareup.sample.dungeon.benchmark
 import android.content.Context
 import androidx.benchmark.macro.BaselineProfileMode
 import androidx.benchmark.macro.CompilationMode
-import androidx.benchmark.macro.FrameTimingMetric
 import androidx.benchmark.macro.StartupMode
 import androidx.benchmark.macro.StartupTimingMetric
 import androidx.benchmark.macro.junit4.MacrobenchmarkRule
 import androidx.test.core.app.ApplicationProvider
 import androidx.test.ext.junit.runners.AndroidJUnit4
 import androidx.test.platform.app.InstrumentationRegistry
 import androidx.test.uiautomator.UiDevice
-import com.squareup.sample.dungeon.benchmark.WorkflowBaselineProfiles.Companion.PACKAGE_NAME
+import com.squareup.sample.dungeon.benchmark.DungeonGatherBaselineProfile.Companion.PACKAGE_NAME
 import org.junit.Before
 import org.junit.Rule
 import org.junit.Test
@@ -25,36 +24,35 @@ import kotlin.time.ExperimentalTime
  * This test compares performance with and without profiles.
  */
 @RunWith(AndroidJUnit4::class)
-public class WorkflowBaselineBenchmark {
+class DungeonStartupBenchmark {
 
-  @get:Rule
-  public val benchmarkRule: MacrobenchmarkRule = MacrobenchmarkRule()
+  @get:Rule val benchmarkRule: MacrobenchmarkRule = MacrobenchmarkRule()
 
   private lateinit var context: Context
   private lateinit var device: UiDevice
 
-  @Before
-  public fun setUp() {
+  @Before fun setUp() {
     val instrumentation = InstrumentationRegistry.getInstrumentation()
     context = ApplicationProvider.getApplicationContext()
     device = UiDevice.getInstance(instrumentation)
   }
 
-  @Test
-  public fun benchmarkNoCompilation() {
-    benchmark(CompilationMode.None())
+  @Test fun benchmarkStartupNoCompilation() {
+    benchmarkStartup(CompilationMode.None())
   }
 
-  @Test
-  public fun benchmarkBaselineProfiles() {
-    benchmark(CompilationMode.Partial(baselineProfileMode = BaselineProfileMode.Require))
+  @Test fun benchmarkStartupBaselineProfiles() {
+    benchmarkStartup(CompilationMode.Partial(baselineProfileMode = BaselineProfileMode.Require))
   }
 
-  @OptIn(ExperimentalTime::class)
-  public fun benchmark(compilationMode: CompilationMode) {
+  @Test fun benchmarkStartupFullAOT() {
+    benchmarkStartup(CompilationMode.Full())
+  }
+
+  @OptIn(ExperimentalTime::class) fun benchmarkStartup(compilationMode: CompilationMode) {
     benchmarkRule.measureRepeated(
       packageName = PACKAGE_NAME,
-      metrics = listOf(StartupTimingMetric(), FrameTimingMetric()),
+      metrics = listOf(StartupTimingMetric()),
       iterations = 3,
       startupMode = StartupMode.COLD,
       compilationMode = compilationMode,
@@ -63,7 +61,7 @@ public class WorkflowBaselineBenchmark {
       }
     ) {
       startActivityAndWait()
-      WorkflowBaselineProfiles.openMazeAndNavigate(device)
+      DungeonGatherBaselineProfile.openMazeAndNavigate(device)
     }
   }
 }

benchmarks/dungeon-benchmark/src/main/java/com/squareup/sample/dungeon/benchmark/WorkflowBaselineBenchmark.kt renamed to benchmarks/dungeon-benchmark/src/main/java/com/squareup/sample/dungeon/benchmark/DungeonStartupBenchmark.kt

@@ -0,0 +1,64 @@
+plugins {
+  id("com.android.test")
+  id("org.jetbrains.kotlin.android")
+}
+
+// Note: We are not including our defaults from .buildscript as we do not need the base Workflow
+// dependencies that those include.
+
+android {
+  compileSdk = 32
+
+  compileOptions {
+    sourceCompatibility = JavaVersion.VERSION_1_8
+    targetCompatibility = JavaVersion.VERSION_1_8
+  }
+
+  kotlinOptions {
+    jvmTarget = "1.8"
+  }
+
+  defaultConfig {
+    minSdk = 26
+    targetSdk = 32
+
+    testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
+  }
+
+  buildTypes {
+    // This benchmark buildType is used for benchmarking, and should function like your
+    // release build (for example, with minification on). It's signed with a debug key
+    // for easy local/CI testing.
+    create("benchmark") {
+      isDebuggable = true
+      signingConfig = getByName("debug").signingConfig
+      matchingFallbacks.add("release")
+      proguardFile("baseline-proguard-rules.pro")
+    }
+  }
+
+  // Unclear: there are kotlin_builtins duplication between 1.5.20 and 1.6.10?
+  packagingOptions {
+    resources.excludes.add("**/*.kotlin_*")
+  }
+
+  targetProjectPath = ":benchmarks:performance-poetry:complex-poetry"
+  experimentalProperties["android.experimental.self-instrumenting"] = true
+}
+
+dependencies {
+  implementation(project(":benchmarks:performance-poetry:complex-poetry"))
+  implementation(project(":samples:containers:poetry"))
+  implementation(project(":workflow-core"))
+
+  implementation(libs.androidx.test.junit)
+  implementation(libs.androidx.test.espresso.core)
+  implementation(libs.androidx.test.uiautomator)
+  implementation(libs.androidx.macro.benchmark)
+}
+
+androidComponents {
+  beforeVariants(selector().all()) {
+    it.enable = it.buildType == "benchmark"
+  }
+}

benchmarks/performance-poetry/complex-benchmark/build.gradle.kts

@@ -0,0 +1,12 @@
+<?xml version="1.0" encoding="utf-8"?>
+<manifest xmlns:android="http://schemas.android.com/apk/res/android"
+    xmlns:tools="http://schemas.android.com/tools"
+    package="com.squareup.benchmarks.performance.poetry.complex.benchmark">
+
+  <queries>
+    <package android:name="com.squareup.benchmarks.performance.complex.poetry" />
+  </queries>
+  <uses-permission
+      android:name="android.permission.WRITE_EXTERNAL_STORAGE"
+      tools:ignore="ScopedStorage" />
+</manifest>

benchmarks/performance-poetry/complex-benchmark/src/main/AndroidManifest.xml

@@ -0,0 +1 @@
+-dontobfuscate