Running Benchmarks

There are two entry points for running benchmarks:

  • main.dart - a general Dart application for running performance benchmarks
  • local_runner.dart - an example Dart application which sets up the local environment and then calls main.dart to run performance benchmarks

local_runner.dart

This Dart application is one example for running performance benchmarks. When run, this application 1) extracts a branch from a git repository into a temporary directory, and 2) creates a symlink to the out or xcodebuild directory for proper package-root package resolution. Once setup is complete, this applications calls main.dart

The required command line arguments are

  • gitDir = a path to the git repository containing the initial target source
  • branch = the branch containing the initial target source
  • inputFile = the instrumentation or log file

Additional arguments are passed directly to main.dart. For example, if the log was recorded on one machine and is played back on another, then you might need to specify -m, to map the source paths for playback. When specifying additional arguments, any occurrences of @tmpSrcDir@ will be replaced with the absolute path of the temporary directory into which the source was extracted.

main.dart

This Dart application reads an instrumentation or local log file produced by analysis server, “replays” that interaction with the analysis server, compares the notifications and responses with what was recorded in the log, and produces a report. It assumes that the environment for playback has already been setup. The required command line arguments are

  • -i, --input The input file specifying how this client should interact with the server. If the input file name is “stdin”, then the instructions are read from stdin.
  • -m, --map , This option defines a mapping from the original source directory when the instrumentation or log file was generated to the target source directory used during performance testing. Multiple mappings can be specified. WARNING: The contents of the target directory will be modified
  • -t, --tmpSrcDir The temporary directory containing source used during performance measurement. WARNING: The contents of the target directory will be modified
  • -d, --diagnosticPort localhost port on which server will provide diagnostic web pages
  • -v, --verbose Verbose logging
  • --vv Extra verbose logging
  • -h, --help Print this help information

For each request recorded in the input file, the application sends a corresponding request to the analysis server and waits up to 60 seconds for a response to that request. If a response in not received in that time, then the application exits. Any responses that are received are compared with the recorded response.

For each analysis-complete notification recorded in the input file, the application waits for the corresponding analysis-complete notification from the running analysis server. While it is waiting for an analysis-complete notification, the application monitors the stream of notifications. If there is a period of more than 60 seconds during which no communication is received from the server, the application assumes that the server is hung and exits.