# Scenario App: Android Tests and Test Runner

End-to-end tests and test infrastructure for the Flutter engine on Android.

> [!IMPORTANT]
> There are several **known issues** with this test suite:
>
> - [#144407](https://github.com/flutter/flutter/issues/144407): "Newer Android"
>   tests are missing expected cropping and rotation.
> - [#144365](https://github.com/flutter/flutter/issues/144365): Tests that use
>   `ExternalTextureFlutterActivity` sometimes do not render.
> - [#144232](https://github.com/flutter/flutter/issues/144232): Impeller Vulkan
>   sometimes hangs on emulators.
> - [#144352](https://github.com/flutter/flutter/issues/144352): Skia Gold
>   sometimes is missing expected diffs.

---

Top topics covered in this document include (but are not limited to):

- [Running the Tests](#running-the-tests)
- [Contributing](#contributing)
- [Project History](#project-history)
- [Troubleshooting](#troubleshooting)
- [Getting Help](#getting-help)

## Introduction

This package simulates a Flutter app that uses the engine (`dart:ui`) only, in
conjunction with Android-specific embedding code that simulates the use of the
engine in a real app (such as plugins and platform views).

A custom test runner, [`run_android_tests.dart`](bin/run_android_tests.dart), is
used to run the tests on a connected device or emulator, and report the results,
including screenshots (golden files) and logs from `adb logcat`.

In the following architecture diagram:

> <a
>   href="https://github.com/flutter/flutter/wiki/The-Engine-architecture"
>   title="The Engine architecture"> <img
>   width="300"
>   alt="Anatomy of a Flutter app"
>   src="https://raw.githubusercontent.com/flutter/engine/main/docs/app_anatomy.svg"
> /> </a>

- The Dart app is represented by [`lib/main.dart`](../lib/main.dart).
- There is no framework code.
- `dart:ui` and the engine are the same as in a real app.
- Android-specific application code is in this directory
  ([`android/`](./)).
- The runner is a _custom_ test runner,
  [`run_android_tests.dart`](bin/run_android_tests.dart).

### Scope of Testing

The tests in this package are end-to-end tests that _specifically_ exercise the
engine and Android-specific embedding code. They are not [unit tests][1] for the
engine or the framework, and are not designed to test the behavior of the
framework or specific plugins, but rather to simulate the use of a framework
or plugins downstream of the engine.

In other words, we test "does the engine work on Android?" without a dependency
on either the Flutter framework, Flutter tooling, or any specific plugins.

[1]: ../../../shell/platform/android

### Golden Comparisons

Many of the Android-specific interactions with the engine are visual, such as
[external textures](https://api.flutter.dev/flutter/widgets/Texture-class.html)
or [platform views](https://docs.flutter.dev/platform-integration/android/platform-views),
and as such, the tests in this package use golden screenshot file comparisons to
verify the correctness of the engine's output.

For example, in [`ExternalTextureTests_testMediaSurface`](https://flutter-engine-gold.skia.org/search?corpus=flutter-engine&include_ignored=false&left_filter=name%3DExternalTextureTests_testMediaSurface&max_rgba=0&min_rgba=0&negative=true&not_at_head=false&positive=true&reference_image_required=false&right_filter=&sort=descending&untriaged=true), a [video](app/src/main/assets/sample.mp4) is converted to an external texture and displayed in a Flutter app. The test takes a screenshot of the app and compares it to a golden file:

<img 
  alt="Two pictures, the top one Flutter and the bottom Android"
  src="https://github.com/flutter/flutter/assets/168174/e2c34b88-d03d-4732-87e4-a86c97d006c5"
  width="300"
/>

_The top picture is the Flutter app, and the bottom picture is just Android._

See also:

- [`tools/compare_goldens`](../../../tools/compare_goldens).

## Prerequisites

If you've never worked in the `flutter/engine` repository before, you will
need to setup a development environment that is quite different from a typical
Flutter app or even working on the Flutter framework. It will take roughly
30 minutes to an hour to setup an environment, depending on your familiarity
with the tools and the speed of your internet connection.

See also:

- [Setting up the Engine](https://github.com/flutter/flutter/wiki/Setting-up-the-Engine-development-environment)
- [Rebuilding the Tests](#rebuilding-the-tests)

### Android SDK

It's highly recommended to use the engine's vendored Android SDK, which once
you have the engine set up, you can find at
`$ENGINE/src/flutter/third_party/android_tools/sdk`. Testing or running with
other versions of the SDK may work, but it's _not guaranteed_, and might have
different results.

Consider also placing this directory in the `ANDROID_HOME` environment variable:

```sh
export ANDROID_HOME=$ENGINE/src/flutter/third_party/android_tools/sdk
```

### Device or Emulator

The tests in this package require a connected device or emulator to run. The
device or emulator should be running the same version of Android as the CI
configuration (there are issues with crashes and other problems on older
emulators in particular).

> [!CAUTION]
>
> [#144561](https://github.com/flutter/flutter/issues/144561): The emulator
> vendored in the engine checkout is old and [has a known issue with Vulkan](https://github.com/flutter/flutter/issues/144232).
>
> If you're working locally, you can update your copy by running:
>
> ```sh
> $ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager --install emulator
> ```

### Additional Dependencies

While not required, it is **strongly recommended** to use an IDE such as
[Android Studio](https://developer.android.com/studio) when contributing to the
Android side of the test suite, as it will provide a better development
experience such as code completion, debugging, and more:

<img
  width="500"
  alt="Screenshot of Android Studio with Autocompletion"
  src="https://github.com/flutter/flutter/assets/168174/79b685f5-8da7-4396-abe6-9baeb29d7ce3"
/>

> [!TIP]
>
> Android Studio is expected to work _out of the box_ in this directory.
>
> If you encounter any issues, please [file a bug](#getting-help).

## Running the Tests

The [test runner](../bin/run_android_tests.dart) is a Dart script that installs
the test app and test suite on a connected device or emulator, runs the tests,
and reports the results including screenshots (golden files) and logs from
`adb logcat`.

From the `$ENGINE/src/flutter` directory, run:

```sh
dart ./testing/scenario_app/bin/run_android_tests.dart
```

By default when run locally, the test runner:

- Uses the engine-wide default graphics backend for Android.
- Uses the last built Android-specific engine artifacts (i.e. `$ENGINE/src/out/android_*/`).
- Will not diff screenshots, but does save them to a logs directory.

### Rebuilding the Tests

If you've made changes to any file in `scenario_app`, incluing the Dart code
in `lib/` or the Android code in `android/`, you will need to rebuild the
tests before running them.

To rebuild the `scenario_app` for the `android_debug_unopt_arm64` variant:

```sh
ninja -C out/android_debug_unopt_arm64 scenario_app
```

See also:

- [Compiling the Engine](https://github.com/flutter/flutter/wiki/Compiling-the-engine)

### Common Options

A list of options can be found by running:

```sh
dart ./testing/scenario_app/bin/run_android_tests.dart --help
```

Frequently used options include:

- `--out-dir`: Which engine artifacts to use (e.g.
  `--out-dir=../out/android_debug_unopt_arm64`).

- `--logs-dir`: Where to save full `adb logcat` logs and screenshots.

- `--[no]-enable-impeller`: Enables/disables use of the Impeller graphics
  backend.

- `--impeller-backend`: Use a specific Impeller backend (e.g.
  `--impeller-backend=opengles`).

- `--force-surface-producer-surface-texture`: Force the use of `SurfaceTexture`s
  for plugin code that uses `SurfaceProducer`s. This instruments the same code
  path that is used for Android API versions that are <= 28 without requiring
  an older emulator or device.

- `--smoke-test=<full.class.Name>`: Runs a specific test, instead of all tests.

### Advanced Options

When debugging the runner itself, you can use the `--verbose` flag to see more
detailed output, including additional options such as configuring which binary
of `adb` to use:

```sh
dart ./testing/scenario_app/bin/run_android_tests.dart --help --verbose
```

See also:

- [`bin/utils/options.dart`](../bin/utils/options.dart).

### CI Configuration

See [`ci/builders`](../../../ci/builders) and grep for `run_android_tests.dart`:

```sh
grep -r run_android_tests.dart ci/builders
```

> [!NOTE]
> The Impeller OpenGLES backend tests are only run on staging (`bringup: true`)
> and as such are **non-blocking**. We expect to stabilize and run these tests
> as part of a wider release of the Impeller OpenGLES backend.

#### Older Android

"Older Android" refers to "code paths to support older Android API levels".
Specifically, these configurations use
`--force-surface-producer-surface-texture` (see [above](#common-options) for
details).

| Backend           | CI Configuration            | CI History                                     | Skia Gold                 |
| ----------------- | --------------------------- | ---------------------------------------------- | ------------------------- |
| Skia              | [`ci/builders`][skia-st-ci] | [Presubmit][skia-try], [Postsubmit][skia-prod] | [Skia Gold][skia-st-gold] |
| Impeller OpenGLES | [`ci/builders`][imp-st-ci]  | [Staging][imp-staging]                         | N/A                       |

[skia-try]: https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20linux_android_emulator_skia_tests
[skia-prod]: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20linux_android_emulator_skia_tests
[imp-staging]: https://ci.chromium.org/ui/p/flutter/builders/staging/Linux%20linux_android_emulator_opengles_tests
[impeller-try]: https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20linux_android_emulator_tests
[impeller-prod]: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20linux_android_emulator_tests
[skia-st-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--no-enable-impeller%22+%22run_android_tests.dart%22+%22--force-surface-producer-surface-texture%22&type=code
[skia-st-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dtrue%26GraphicsBackend%3Dskia&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia
[imp-st-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--impeller-backend%3Dopengles%22+%22run_android_tests.dart%22+%22--force-surface-producer-surface-texture%22&type=code
[imp-st-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dtrue%26GraphicsBackend%3Dimpeller-opengles&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia

#### Newer Android

| Backend           | CI Configuration             | CI History                                             | Skia Gold                |
| ----------------- | ---------------------------- | ------------------------------------------------------ | ------------------------ |
| Skia              | [`ci/builders`][skia-ci]     | [Presubmit][skia-try], [Postsubmit][skia-prod]         | [Skia Gold][skia-gold]   |
| Impeller OpenGLES | [`ci/builders`][imp-gles-ci] | [Staging][imp-staging]                                 | N/A                      |
| Impeller Vulkan   | [`ci/builders`][imp-vk-ci]   | [Presubmit][impeller-try], [Postsubmit][impeller-prod] | [Skia Gold][imp-vk-gold] |

[skia-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--no-enable-impeller%22+%22run_android_tests.dart%22&type=code
[skia-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dfalse%26GraphicsBackend%3Dskia&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia
[imp-gles-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--impeller-backend%3Dopengles%22+%22run_android_tests.dart%22&type=code
[imp-vk-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--impeller-backend%3Dvulkan%22+%22run_android_tests.dart%22&type=code
[imp-vk-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dfalse%26GraphicsBackend%3Dimpeller-vulkan&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia

## Contributing

[![GitHub Issues or Pull Requests by label](https://img.shields.io/github/issues/flutter/flutter/e%3A%20scenario-app)](https://github.com/flutter/flutter/issues?q=is%3Aopen+is%3Aissue+label%3A%22e%3A+scenario-app%22)

Contributions to this package are welcome, as it is a critical part of the
engine's test suite.

### Anatomy of a Test

A "test" in practice is a combination of roughly 3 components:

1. An Android _[JUnit test][]_ that configures and launches an Android activity.
2. An Android _[activity][]_, which simulates the Android side of a plugin or
   platform view.
3. A Dart _[scenario][]_, which simulates the Flutter side of an application.

[junit test]: ./app/src/androidTest/java/dev/flutter/scenariosui/DrawSolidBlueScreenTest.java
[activity]: ./app/src/main/java/dev/flutter/scenarios/ExternalTextureFlutterActivity.java
[scenario]: ../lib/src/solid_blue.dart

While every test suite has exactly one JUnit-instrumented class, each test can
have many activities and scenarios, each with their own configuration, setup,
and assertions. Not all of this is desirable, but it is the current state of
the test suite.

A test might also take a screenshot. See the _Skia Gold_ links in
[CI Configuration](#ci-configuration) for examples.

### Project History

This test suite was [originally written in 2019](https://github.com/flutter/engine/pull/10007)
with a goal of:

> \[being\] suitable for embedders to do integration testing with - it has no
> dependencies on the flutter_tools or framework, and so will not fail/flake
> based on variances in those downstream.

Unfortunately, the Android side of the test suite was never fully operational,
and the tests, even if failing, were accidentally be reported as passing on CI.
In 2024, as the team got closer to shipping our new graphics backend,
[Impeller](https://docs.flutter.dev/perf/impeller) on Android, it was clear that
we needed a reliable test suite for the engine on Android, particularly for
visual tests around external textures and platform views.

So, this package was revived and updated to be a (more) reliable test suite for
the engine on Android. It's by no means complete
([contributions welcome](#contributing)), but it did successfully catch at least
one bug that would not have been detected automatically otherwise.

_Go forth and test the engine on Android!_

## Troubleshooting

If you encounter any issues, please [file a bug](#getting-help).

### My test is failing on CI

If a test is failing on CI, it's likely that the test is failing locally as
well. Try the steps in [running the Tests](#running-the-tests) to reproduce the
failure locally, and then debug the failure as you would any other test. If this
is your first time working on the engine, you may need to setup a development
environment first (see [prerequisites](#prerequisites)).

The test runner makes extensive use of logging and screenshots to help debug
failures. If you're not sure where to start, try looking at the logs and
screenshots to see if they provide any clues (you'll need them to file a bug
anyway).

`test: Android Scenario App Integration Tests (Impeller/Vulkan)`
on [LUCI](https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20Engine%20Drone/2079486/overview) produces these logs:

![Screenshot of Logs](https://github.com/flutter/flutter/assets/168174/fed8fae6-bcd2-4e6e-a074-c2c37f34a770)

The files include a screenshot of each test, and a _full_ logcat log from the
device (you might find it easier trying the `stdout` of the test first, which
uses rudimentary log filtering). In the case of multiple runs, the logs are
prefixed with the test configuration and run attempt.

### Gradle is failing due to an "not part of the dependency lock state" error

If you update dependencies in the `app/build.gradle` file, you may encounter an
error (probably in CI) that looks like:

```txt
FAILED: scenario_app/reports/lint-results.xml
vpython3 ../../flutter/testing/rules/run_gradle.py /b/s/w/ir/cache/builder/src/flutter/testing/scenario_app/android lint --no-daemon -Pflutter_jar=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/flutter.jar -Pout_dir=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/scenario_app --project-cache-dir=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/scenario_app/.gradle --gradle-user-home=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/scenario_app/.gradle

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:checkDebugAarMetadata'.
> Could not resolve all files for configuration ':app:debugRuntimeClasspath'.
   > Resolved 'org.hamcrest:hamcrest-core:1.3' which is not part of the dependency lock state
   > Resolved 'com.google.code.findbugs:jsr305:2.0.2' which is not part of the dependency lock state
```

This is because [`gradle.lockfile`](./app/gradle.lockfile) is out of date. To
update it, run:

```sh
cd testing/scenario_app/android
$ENGINE_SRC/flutter/third_party/gradle/bin/gradle app:dependencies --write-locks
```

## Getting Help

To suggest changes, or highlight problems, please [file an issue](https://github.com/flutter/flutter/issues/new?labels=e:%20scenario-app,engine,platform-android,fyi-android,team-engine).

If you're not sure where to start, or need help debugging or contributing, you
can also reach out to [`hackers-android`](https://discord.com/channels/608014603317936148/846507907876257822) on Discord, or the Flutter engine team internally. There is
no full-time maintainer of this package, so be prepared to do some of the
legwork yourself.