[vm/fuzzer] Migrated driver from Python to Dart.
Rationale:
Remove last part of Python in favor of Dart. Note that
this is still under heavy development, with many new
features planned. But it is a runnable system now which
we be deployed from the comfort of your own desktop or
on a cluster if needed.
Change-Id: I96fc8973bcc00c69849434087c57ff897d01b79a
Reviewed-on: https://dart-review.googlesource.com/75621
Commit-Queue: Aart Bik <ajcbik@google.com>
Reviewed-by: Jonas Termansen <sortie@google.com>
diff --git a/runtime/tools/dartfuzz/README.md b/runtime/tools/dartfuzz/README.md
index 9ee5e38..6ea6625 100644
--- a/runtime/tools/dartfuzz/README.md
+++ b/runtime/tools/dartfuzz/README.md
@@ -10,15 +10,15 @@
How to run DartFuzz
===================
+To generate a single random Dart program, run
- dart dartfuzz.dart [--help] [--seed SEED] [FILENAME]
+ dart dartfuzz.dart [--help] [--seed SEED] FILENAME
where
--help : prints help and exits
--seed : defines random seed (system-set by default)
-If no FILENAME is given, DartFuzz sends all output to stdout.
The tool provides a runnable main isolate. A typical single
test run looks as:
@@ -27,6 +27,9 @@
How to start DartFuzz testing
=============================
+To start a fuzz testing session, run
+
+ dart dartfuzz_test.dart
run_dartfuzz_test.py [--help]
[--repeat REPEAT]
@@ -38,21 +41,23 @@
--help : prints help and exits
--repeat : number of tests to run (1000 by default)
- --true_divergence : only report true divergences
+ --show-stats : show session statistics (true by default)
+ --true-divergence : only report true divergences (true by default)
+ --dart-top : sets DART_TOP explicitly through command line
--mode1 : m1
--mode2 : m2, and values one of
- jit-ia32 = Dart JIT (ia32)
- jit-x64 = Dart JIT (x64)
- jit-arm32 = Dart JIT (simarm)
- jit-arm64 = Dart JIT (simarm64)
- aot-x64 = Dart AOT (x64)
- aot-arm64 = Dart AOT (simarm64)
- js = dart2js + JS
+ jit-[debug-]ia32 = Dart JIT (ia32)
+ jit-[debug-]x64 = Dart JIT (x64)
+ jit-[debug-]arm32 = Dart JIT (simarm)
+ jit-[debug-]arm64 = Dart JIT (simarm64)
+ aot-[debug-]x64 = Dart AOT (x64)
+ aot-[debug-]arm64 = Dart AOT (simarm64)
+ js = dart2js + JS
-This fuzzer tool assumes the environment variable 'DART_TOP' points to
-the top of the Dart SDK development tree in which all proper binaries
-have been built already (e.g. testing jit-ia32 will invoke the binary
-${DART_TOP}/out/ReleaseIA32/dart to start the Dart VM).
+This fuzz testing tool assumes the environment variable 'DART_TOP'
+points to the top of the Dart SDK development tree in which all proper
+binaries have been built already (e.g. testing jit-ia32 will invoke
+the binary ${DART_TOP}/out/ReleaseIA32/dart to start the Dart VM).
Background
==========
@@ -73,9 +78,9 @@
based fuzz testing constructs random, but properly formatted input data.
Mutation-based fuzz testing applies small random changes to existing inputs
in order to detect shortcomings in a system. Profile-guided or coverage-guided
-fuzzing adds a direction to the way these random changes are applied. Multi-
-layered approaches generate random inputs that are subsequently mutated at
-various stages of execution.
+fuzz testing adds a direction to the way these random changes are applied.
+Multi-layered approaches generate random inputs that are subsequently mutated
+at various stages of execution.
The randomness of fuzz testing implies that the size and scope of testing is
no longer bounded. Every new run can potentially discover bugs and crashes
diff --git a/runtime/tools/dartfuzz/dartfuzz.dart b/runtime/tools/dartfuzz/dartfuzz.dart
index d32766f..06790d0 100644
--- a/runtime/tools/dartfuzz/dartfuzz.dart
+++ b/runtime/tools/dartfuzz/dartfuzz.dart
@@ -92,9 +92,9 @@
const fieldName = 'fld';
const methodName = 'foo';
-// Class that generates a random, but runnable Dart program for fuzz testing.
+/// Class that generates a random, but runnable Dart program for fuzz testing.
class DartFuzz {
- DartFuzz(this.seed, this.sink);
+ DartFuzz(this.seed, this.file);
void run() {
// Initialize program variables.
@@ -842,23 +842,23 @@
// Emits indented line to append to program.
void emitLn(String line, {bool newline = true}) {
- sink.write(' ' * indent);
+ file.writeStringSync(' ' * indent);
emit(line, newline: newline);
}
// Emits text to append to program.
void emit(String txt, {bool newline = false}) {
- sink.write(txt);
+ file.writeStringSync(txt);
if (newline) {
- sink.write('\n');
+ file.writeStringSync('\n');
}
}
// Random seed used to generate program.
final int seed;
- // Sink used for output.
- final IOSink sink;
+ // File used for output.
+ final RandomAccessFile file;
// Program variables.
Random rand;
@@ -896,22 +896,19 @@
return seed;
}
-// Main driver when dartfuzz.dart is run stand-alone.
+/// Main driver when dartfuzz.dart is run stand-alone.
main(List<String> arguments) {
final parser = new ArgParser()
..addOption('seed',
- abbr: 's',
- help: 'random seed (0 forces time-based seed)',
- defaultsTo: '0');
+ help: 'random seed (0 forces time-based seed)', defaultsTo: '0');
try {
final results = parser.parse(arguments);
final seed = getSeed(results['seed']);
- final sink = results.rest.isEmpty
- ? stdout
- : new File(results.rest.single).openWrite();
- new DartFuzz(seed, sink).run();
+ final file = new File(results.rest.single).openSync(mode: FileMode.write);
+ new DartFuzz(seed, file).run();
+ file.closeSync();
} catch (e) {
- print('Usage: dart dartfuzz.dart [OPTIONS] [FILENAME]');
+ print('Usage: dart dartfuzz.dart [OPTIONS] FILENAME');
print(parser.usage);
}
}
diff --git a/runtime/tools/dartfuzz/dartfuzz_test.dart b/runtime/tools/dartfuzz/dartfuzz_test.dart
new file mode 100644
index 0000000..aed0162
--- /dev/null
+++ b/runtime/tools/dartfuzz/dartfuzz_test.dart
@@ -0,0 +1,318 @@
+// Copyright (c) 2018, the Dart project authors. Please see the AUTHORS file
+// for details. All rights reserved. Use of this source code is governed by a
+// BSD-style license that can be found in the LICENSE file.
+
+import 'dart:convert';
+import 'dart:io';
+import 'dart:math';
+
+import 'package:args/args.dart';
+
+import 'dartfuzz.dart';
+
+const debug = false;
+const sigkill = 9;
+const timeout = 30; // in seconds
+
+// Exit code of running a test.
+enum ResultCode { success, timeout, error }
+
+/// Result of running a test.
+class TestResult {
+ TestResult(this.code, this.output);
+ ResultCode code;
+ String output;
+}
+
+/// Command runner.
+TestResult runCommand(List<String> cmd, Map<String, String> env) {
+ // TODO: use Dart API for some of the modes?
+ ProcessResult res = Process.runSync(
+ 'timeout', ['-s', '$sigkill', '$timeout'] + cmd,
+ environment: env);
+ if (debug) {
+ print('\nrunning $cmd yields:\n'
+ '${res.exitCode}\n${res.stdout}\n${res.stderr}\n');
+ }
+ if (res.exitCode == -sigkill) {
+ return new TestResult(ResultCode.timeout, res.stdout);
+ } else if (res.exitCode != 0) {
+ return new TestResult(ResultCode.error, res.stdout);
+ }
+ return new TestResult(ResultCode.success, res.stdout);
+}
+
+/// Abstraction for running one test in a particular mode.
+abstract class TestRunner {
+ String description();
+ TestResult run(String fileName);
+
+ // Factory.
+ static TestRunner getTestRunner(
+ Map<String, String> env, String top, String mode) {
+ if (mode.startsWith('jit')) return new TestRunnerJIT(env, top, mode);
+ if (mode.startsWith('aot')) return new TestRunnerAOT(env, top, mode);
+ if (mode.startsWith('js')) return new TestRunnerJS(env, top, mode);
+ throw ('unknown mode: $mode');
+ }
+
+ // Convert mode to tag.
+ static String getTag(String mode) {
+ if (mode.endsWith('debug-ia32')) return 'DebugIA32';
+ if (mode.endsWith('debug-x64')) return 'DebugX64';
+ if (mode.endsWith('debug-arm32')) return 'DebugSIMARM';
+ if (mode.endsWith('debug-arm64')) return 'DebugSIMARM64';
+ if (mode.endsWith('ia32')) return 'ReleaseIA32';
+ if (mode.endsWith('x64')) return 'ReleaseX64';
+ if (mode.endsWith('arm32')) return 'ReleaseSIMARM';
+ if (mode.endsWith('arm64')) return 'ReleaseSIMARM64';
+ throw ('unknown tag in mode: $mode');
+ }
+}
+
+/// Concrete test runner of Dart JIT.
+class TestRunnerJIT implements TestRunner {
+ TestRunnerJIT(Map<String, String> e, String top, String mode) {
+ tag = TestRunner.getTag(mode);
+ env = Map<String, String>.from(e);
+ env['PATH'] = "$top/out/$tag:${env['PATH']}";
+ }
+ String description() {
+ return "JIT-${tag}";
+ }
+
+ TestResult run(String fileName) {
+ return runCommand(['dart', fileName], env);
+ }
+
+ String tag;
+ Map<String, String> env;
+}
+
+/// Concrete test runner of Dart AOT.
+class TestRunnerAOT implements TestRunner {
+ TestRunnerAOT(Map<String, String> e, String top, String mode) {
+ tag = TestRunner.getTag(mode);
+ env = Map<String, String>.from(e);
+ env['PATH'] = "$top/pkg/vm/tool:${env['PATH']}";
+ env['DART_CONFIGURATION'] = tag;
+ }
+ String description() {
+ return "AOT-${tag}";
+ }
+
+ TestResult run(String fileName) {
+ TestResult result = runCommand(['precompiler2', fileName, 'snapshot'], env);
+ if (result.code != ResultCode.success) {
+ return result;
+ }
+ return runCommand(['dart_precompiled_runtime2', 'snapshot'], env);
+ }
+
+ String tag;
+ Map<String, String> env;
+}
+
+/// Concrete test runner of Dart2JS.
+class TestRunnerJS implements TestRunner {
+ TestRunnerJS(Map<String, String> e, String top, String mode) {
+ env = Map<String, String>.from(e);
+ env['PATH'] = "$top/out/ReleaseX64/dart-sdk/bin:${env['PATH']}";
+ }
+ String description() {
+ return "Dart2JS";
+ }
+
+ TestResult run(String fileName) {
+ TestResult result = runCommand(['dart2js', fileName], env);
+ if (result.code != ResultCode.success) {
+ return result;
+ }
+ return runCommand(['nodejs', 'out.js'], env);
+ }
+
+ Map<String, String> env;
+}
+
+/// Class to run a fuzz testing session.
+class DartFuzzTest {
+ DartFuzzTest(this.env, this.repeat, this.trueDivergence, this.showStats,
+ this.top, this.mode1, this.mode2);
+
+ bool runSession() {
+ setupSession();
+
+ print('\n**\n**** Dart Fuzz Testing\n**\n');
+ print('Fuzz Version : ${version}');
+ print('#Tests : ${repeat}');
+ print('Exec-Mode 1 : ${runner1.description()}');
+ print('Exec-Mode 2 : ${runner2.description()}');
+ print('Dart Dev : ${top}');
+ print('Orig Dir : ${orgDir.path}');
+ print('Temp Dir : ${tmpDir.path}\n');
+
+ showStatistics();
+ for (int i = 0; i < repeat; i++) {
+ numTests++;
+ seed = rand.nextInt(1 << 32);
+ generateTest();
+ runTest();
+ showStatistics();
+ }
+
+ cleanupSession();
+ if (numDivergences != 0) {
+ print('\n\nfailure\n');
+ return false;
+ }
+ print('\n\nsuccess\n');
+ return true;
+ }
+
+ void setupSession() {
+ rand = new Random();
+ orgDir = Directory.current;
+ tmpDir = Directory.systemTemp.createTempSync('dart_fuzz');
+ Directory.current = tmpDir;
+ fileName = 'fuzz.dart';
+ runner1 = TestRunner.getTestRunner(env, top, mode1);
+ runner2 = TestRunner.getTestRunner(env, top, mode2);
+ numTests = 0;
+ numSuccess = 0;
+ numNotRun = 0;
+ numTimeOut = 0;
+ numDivergences = 0;
+ }
+
+ void cleanupSession() {
+ Directory.current = orgDir;
+ tmpDir.delete(recursive: true);
+ }
+
+ void showStatistics() {
+ if (showStats) {
+ stdout.write('\rTests: $numTests Success: $numSuccess Not-Run: '
+ '$numNotRun: Time-Out: $numTimeOut Divergences: $numDivergences');
+ }
+ }
+
+ void generateTest() {
+ final file = new File(fileName).openSync(mode: FileMode.write);
+ new DartFuzz(seed, file).run();
+ file.closeSync();
+ }
+
+ void runTest() {
+ TestResult result1 = runner1.run(fileName);
+ TestResult result2 = runner2.run(fileName);
+ checkDivergence(result1, result2);
+ }
+
+ void checkDivergence(TestResult result1, TestResult result2) {
+ if (result1.code == result2.code) {
+ // No divergence in result code.
+ switch (result1.code) {
+ case ResultCode.success:
+ // Both were successful, inspect output.
+ if (result1.output == result2.output) {
+ numSuccess++;
+ } else {
+ reportDivergence(result1, result2, true);
+ }
+ break;
+ case ResultCode.timeout:
+ // Both had a time out.
+ numTimeOut++;
+ break;
+ case ResultCode.error:
+ // Both had an error.
+ numNotRun++;
+ break;
+ }
+ } else {
+ // Divergence in result code.
+ if (trueDivergence) {
+ // When only true divergences are requested, any divergence
+ // with at least one time out is treated as a regular time out.
+ if (result1.code == ResultCode.timeout ||
+ result2.code == ResultCode.timeout) {
+ numTimeOut++;
+ return;
+ }
+ }
+ reportDivergence(result1, result2, false);
+ }
+ }
+
+ void reportDivergence(
+ TestResult result1, TestResult result2, bool outputDivergence) {
+ numDivergences++;
+ print('\n\nDIVERGENCE on generated program $version:$seed\n');
+ if (outputDivergence) {
+ print('out1:\n${result1.output}\n\nout2:\n${result2.output}\n');
+ }
+ }
+
+ // Context.
+ final Map<String, String> env;
+ final int repeat;
+ final bool trueDivergence;
+ final bool showStats;
+ final String top;
+ final String mode1;
+ final String mode2;
+
+ // Session.
+ Random rand;
+ Directory orgDir;
+ Directory tmpDir;
+ String fileName;
+ TestRunner runner1;
+ TestRunner runner2;
+ int seed;
+
+ // Stats.
+ int numTests;
+ int numSuccess;
+ int numNotRun;
+ int numTimeOut;
+ int numDivergences;
+}
+
+/// Main driver for a fuzz testing session.
+main(List<String> arguments) {
+ // Set up argument parser.
+ final parser = new ArgParser()
+ ..addOption('repeat', help: 'number of tests to run', defaultsTo: '1000')
+ ..addFlag('true-divergence',
+ negatable: true, help: 'only report true divergences', defaultsTo: true)
+ ..addFlag('show-stats',
+ negatable: true, help: 'show session statistics', defaultsTo: true)
+ ..addOption('dart-top',
+ help: 'explicit value for \$DART_TOP', defaultsTo: '')
+ ..addOption('mode1', help: 'execution mode 1', defaultsTo: 'jit-x64')
+ ..addOption('mode2', help: 'execution mode 2', defaultsTo: 'aot-x64');
+
+ // Start fuzz testing session.
+ try {
+ final env = Platform.environment;
+ final results = parser.parse(arguments);
+ final repeat = int.parse(results['repeat']);
+ final trueDivergence = results['true-divergence'];
+ final showStats = results['show-stats'];
+ final mode1 = results['mode1'];
+ final mode2 = results['mode2'];
+ var top = results['dart-top'];
+ if (top == '') {
+ top = env['DART_TOP'];
+ }
+ final session = new DartFuzzTest(
+ env, repeat, trueDivergence, showStats, top, mode1, mode2);
+ if (!session.runSession()) {
+ exitCode = 1;
+ }
+ } catch (e) {
+ print('Usage: dart dartfuzz_test.dart [OPTIONS]\n${parser.usage}\n$e');
+ exitCode = 255;
+ }
+}
diff --git a/runtime/tools/dartfuzz/run_dartfuzz_test.py b/runtime/tools/dartfuzz/run_dartfuzz_test.py
deleted file mode 100755
index 8d33b49..0000000
--- a/runtime/tools/dartfuzz/run_dartfuzz_test.py
+++ /dev/null
@@ -1,423 +0,0 @@
-#!/usr/bin/env python3
-#
-# Copyright (c) 2018, the Dart project authors. Please see the AUTHORS file
-# for details. All rights reserved. Use of this source code == governed by a
-# BSD-style license that can be found in the LICENSE file.
-
-import abc
-import argparse
-import os
-import shutil
-import signal
-import subprocess
-import sys
-
-from enum import Enum
-from enum import unique
-
-from subprocess import DEVNULL
-from subprocess import PIPE
-from subprocess import Popen
-from subprocess import STDOUT
-from subprocess import TimeoutExpired
-
-from tempfile import mkdtemp
-
-#
-# Helper methods to run commands.
-#
-
-@unique
-class RetCode(Enum):
- """Enum representing return codes."""
- SUCCESS = 0
- TIMEOUT = 1
- ERROR = 2
-
-class FatalError(Exception):
- """Fatal error in script."""
-
-def RunCommandWithOutput(cmd, env, stdout, stderr, timeout=30):
- """Runs command piping output to files, stderr, or stdout.
-
- Args:
- cmd: list of strings, command to run.
- env: shell environment for command.
- stdout: file handle for stdout.
- stderr: file handle for stderr.
- timeout: int, timeout in seconds.
-
- Returns:
- tuple (string, string, RetCode) out, err, return code.
- """
- proc = Popen(cmd, stdout=stdout, stderr=stderr, env=env,
- universal_newlines=True, start_new_session=True)
- try:
- (out, err) = proc.communicate(timeout=timeout)
- if proc.returncode == 0:
- retcode = RetCode.SUCCESS
- else:
- retcode = RetCode.ERROR
- except TimeoutExpired:
- os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
- (out, err) = proc.communicate()
- retcode = RetCode.TIMEOUT
- return (out, err, retcode)
-
-def RunCommand(cmd, out=None, err=None, timeout=30):
- """Executes a command, and returns its return code.
-
- Args:
- cmd: list of strings, a command to execute.
- out: string, file name to open for stdout (or None).
- err: string, file name to open for stderr (or None).
- timeout: int, time out in seconds.
- Returns:
- RetCode, return code of running command.
- """
- if out is not None:
- outf = open(out, mode='w')
- else:
- outf = DEVNULL
- if err is not None:
- errf = open(err, mode='w')
- else:
- errf = DEVNULL
- (_, _, retcode) = RunCommandWithOutput(cmd, None, outf, errf, timeout)
- if outf != DEVNULL:
- outf.close()
- if errf != DEVNULL:
- errf.close()
- return retcode
-
-#
-# Execution modes.
-#
-
-class TestRunner(object):
- """Abstraction for running a test in a particular execution mode."""
- __meta_class__ = abc.ABCMeta
-
- @abc.abstractproperty
- def description(self):
- """Returns a description string of the execution mode."""
-
- @abc.abstractmethod
- def RunTest(self):
- """Run the generated test.
-
- Ensures that the current fuzz.dart in the temporary directory is executed
- under the current execution mode.
-
- Most nonzero return codes are assumed non-divergent, since systems may
- exit in different ways. This is enforced by normalizing return codes.
-
- Returns:
- tuple (string, string, RetCode) stdout-output, stderr-output, return code.
- """
-
-class TestRunnerDartJIT(TestRunner):
- """Concrete test runner of Dart JIT."""
-
- def __init__(self, mode):
- self._mode = mode
- if mode == 'jit-ia32':
- self.AddPathAndTimeout('ReleaseIA32')
- elif mode == 'jit-x64':
- self.AddPathAndTimeout('ReleaseX64')
- elif mode == 'jit-arm32':
- self.AddPathAndTimeout('ReleaseSIMARM')
- elif mode == 'jit-arm64':
- self.AddPathAndTimeout('ReleaseSIMARM64')
- else:
- raise FatalError('Unknown JIT execution mode: ' + mode)
-
- @property
- def description(self):
- return 'Dart ' + self._mode
-
- def AddPathAndTimeout(self, tag, timeout=30):
- self._env = os.environ.copy()
- self._env['PATH'] = self._env['DART_TOP'] + '/out/' + tag + ':' + self._env['PATH']
- self._timeout = timeout
-
- def RunTest(self):
- return RunCommandWithOutput(
- ['dart', 'fuzz.dart'], self._env, PIPE, STDOUT, self._timeout)
-
-class TestRunnerDartAOT(TestRunner):
- """Concrete test runner of Dart AOT."""
-
- def __init__(self, mode):
- self._mode = mode
- if mode == 'aot-x64':
- self.AddPathConfigAndTimeout('ReleaseX64')
- elif mode == 'aot-arm64':
- self.AddPathConfigAndTimeout('ReleaseSIMARM64', 30 * 60)
- else:
- raise FatalError('Unknown AOT execution mode: ' + mode)
-
- @property
- def description(self):
- return 'Dart ' + self._mode
-
- def AddPathConfigAndTimeout(self, tag, timeout=30):
- self._env = os.environ.copy()
- self._env['PATH'] = self._env['DART_TOP'] + '/pkg/vm/tool:' + self._env['PATH']
- self._env['DART_CONFIGURATION'] = tag
- self._timeout = timeout
-
- def RunTest(self):
- (out, err, retcode) = RunCommandWithOutput(
- ['precompiler2', 'fuzz.dart', 'snap'],
- self._env, PIPE, STDOUT, self._timeout)
- if retcode != RetCode.SUCCESS:
- return (out, err, retcode)
- return RunCommandWithOutput(
- ['dart_precompiled_runtime2', 'snap'],
- self._env, PIPE, STDOUT, self._timeout)
-
-class TestRunnerDart2JS(TestRunner):
- """Concrete test runner of Dart through dart2js and JS."""
-
- def __init__(self):
- self.AddPath()
-
- @property
- def description(self):
- return 'Dart as JS'
-
- def AddPath(self):
- self._env = os.environ.copy()
- self._env['PATH'] = self._env['DART_TOP'] + '/ReleaseX64/dart-sdk/bin:' + self._env['PATH']
-
- def RunTest(self):
- (out, err, retcode) = RunCommandWithOutput(
- ['dart2js', 'fuzz.dart'], self._env, PIPE, STDOUT)
- if retcode != RetCode.SUCCESS:
- return (out, err, retcode)
- return RunCommandWithOutput(
- ['nodejs', 'out.js'], self._env, PIPE, STDOUT)
-
-def GetExecutionModeRunner(mode):
- """Returns a runner for the given execution mode.
-
- Args:
- mode: string, execution mode
- Returns:
- TestRunner with given execution mode
- Raises:
- FatalError: error for unknown execution mode
- """
- if mode.startswith('jit'):
- return TestRunnerDartJIT(mode)
- if mode.startswith('aot'):
- return TestRunnerDartAOT(mode)
- if mode == 'js':
- return TestRunnerDart2JS()
- raise FatalError('Unknown execution mode: ' + mode)
-
-#
-# DartFuzzTester class.
-#
-
-class DartFuzzTester(object):
- """Tester that runs DartFuzz many times and report divergences."""
-
- def __init__(self, repeat, true_divergence, mode1, mode2):
- """Constructor for the tester.
-
- Args:
- repeat: int, number of tests to run.
- true_divergence: boolean, report true divergences only.
- mode1: string, execution mode for first runner.
- mode2: string, execution mode for second runner.
- """
- self._repeat = repeat
- self._true_divergence = true_divergence
- self._runner1 = GetExecutionModeRunner(mode1)
- self._runner2 = GetExecutionModeRunner(mode2)
-
- def __enter__(self):
- """On entry, enters new temp directory after saving current directory.
-
- Raises:
- FatalError: error when temp directory cannot be constructed.
- """
- self._save_dir = os.getcwd()
- self._tmp_dir = mkdtemp(dir='/tmp/')
- if self._tmp_dir == None:
- raise FatalError('Cannot obtain temp directory')
- os.chdir(self._tmp_dir)
- return self
-
- def __exit__(self, etype, evalue, etraceback):
- """On exit, re-enters previously saved current directory and cleans up."""
- os.chdir(self._save_dir)
- if self._num_divergences == 0:
- shutil.rmtree(self._tmp_dir)
- print('\n\nsuccess (no divergences)\n')
- else:
- print('\n\nfailure (divergences):', self._tmp_dir, '\n')
-
- def Run(self):
- """Runs DartFuzz many times and report divergences."""
- self.Setup()
- print()
- print('**\n**** Dart Fuzz Testing\n**')
- print()
- print('#Tests :', self._repeat)
- print('Exec-Mode 1 :', self._runner1.description)
- print('Exec-Mode 2 :', self._runner2.description)
- print('Dart Dev :', os.environ.get('DART_TOP'))
- print('Orig Dir :', self._save_dir)
- print('Temp Dir :', self._tmp_dir)
- print()
- self.ShowStats() # show all zeros on start
- for self._test in range(1, self._repeat + 1):
- self.RunTest()
- self.ShowStats()
-
- def Setup(self):
- """Initial setup of the testing environment."""
- # Fuzzer command.
- self._dartfuzz = self._save_dir + '/dartfuzz.dart'
- # Statistics.
- self._test = 0
- self._num_success = 0
- self._num_not_run = 0
- self._num_timed_out = 0
- self._num_divergences = 0
-
- def ShowStats(self):
- """Shows current statistics (on same line) while tester is running."""
- print('\rTests:', self._test,
- 'Success:', self._num_success,
- 'Not-run:', self._num_not_run,
- 'Timed-out:', self._num_timed_out,
- 'Divergences:', self._num_divergences,
- end='')
- sys.stdout.flush()
-
- def RunTest(self):
- """Runs a single fuzz test, comparing two execution modes."""
- self.ConstructTest()
- (out1, _, retcode1) = self._runner1.RunTest()
- (out2, _, retcode2) = self._runner2.RunTest()
- self.CheckForDivergence(out1, retcode1, out2, retcode2)
- self.CleanupTest()
-
- def ConstructTest(self):
- """Use DartFuzz to generate next fuzz.dart test.
-
- Raises:
- FatalError: error when DartFuzz fails.
- """
- # Invoke dartfuzz script on command line rather than calling py code.
- if (RunCommand(['dart', self._dartfuzz], out='fuzz.dart') != RetCode.SUCCESS):
- raise FatalError('Unexpected error while running DartFuzz')
-
- def CheckForDivergence(self, out1, retcode1, out2, retcode2):
- """Checks for divergences and updates statistics.
-
- Args:
- out1: string, output for first runner.
- retcode1: int, normalized return code of first runner.
- out2: string, output for second runner.
- retcode2: int, normalized return code of second runner.
- """
- if retcode1 == retcode2:
- # No divergence in return code.
- if retcode1 == RetCode.SUCCESS:
- # Both compilations and runs were successful, inspect generated output.
- if out1 == out2:
- # No divergence in output.
- self._num_success += 1
- else:
- # Divergence in output.
- self.ReportDivergence(out1, retcode1, out2, retcode2, True)
- elif retcode1 == RetCode.ERROR:
- # Both did not run.
- self._num_not_run += 1
- elif retcode1 == RetCode.TIMEOUT:
- # Both timed out.
- self._num_timed_out += 1
- else:
- raise FatalError('Unknown return code')
- else:
- # Divergence in return code.
- if self._true_divergence:
- # When only true divergences are requested, any divergence in return
- # code where one is a time out is treated as a regular time out.
- if RetCode.TIMEOUT in (retcode1, retcode2):
- self._num_timed_out += 1
- return
- self.ReportDivergence(out1, retcode1, out2, retcode2, False)
-
- def ReportDivergence(self, out1, retcode1, out2, retcode2, is_output_divergence):
- """Reports and saves a divergence.
-
- Args:
- out1: string, output for first runner.
- retcode1: int, normalized return code of first runner.
- out2: string, output for second runner.
- retcode2: int, normalized return code of second runner.
- is_output_divergence, boolean, denotes output divergence.
- """
- self._num_divergences += 1
- print('\n#' + str(self._num_divergences), end='')
- if is_output_divergence:
- print(' divergence in output')
- else:
- print(' divergence in return code: '
- + retcode1.name + ' vs. ' + retcode2.name)
- print('->')
- print(out1, end='')
- print('<-')
- print(out2, end='')
- print('--')
- # Save.
- ddir = self._tmp_dir + '/divergence' + str(self._num_divergences)
- os.mkdir(ddir)
- shutil.copy('fuzz.dart', ddir)
- # TODO: file bug report
-
- def CleanupTest(self):
- """Cleans up after a single test run."""
- for file_name in os.listdir(self._tmp_dir):
- file_path = os.path.join(self._tmp_dir, file_name)
- if os.path.isfile(file_path):
- os.unlink(file_path)
- elif os.path.isdir(file_path):
- pass # keep the divergences directories
-
-#
-# Main driver.
-#
-
-def main():
- # Handle arguments.
- parser = argparse.ArgumentParser()
- parser.add_argument('--repeat', default=1000, type=int,
- help='number of tests to run (default: 1000)')
- parser.add_argument('--true_divergence', default=False, action='store_true',
- help='only report true divergences')
- parser.add_argument('--mode1', default='jit-x64',
- help='execution mode 1 (default: jit-x64)')
- parser.add_argument('--mode2', default='aot-x64',
- help='execution mode 2 (default: aot-x64)')
- args = parser.parse_args()
-
- # Verify DART_TOP.
- if os.environ.get('DART_TOP') == None:
- raise FatalError('DART_TOP needs to point to the Dart SDK tree')
-
- # Run DartFuzz tester.
- with DartFuzzTester(args.repeat,
- args.true_divergence,
- args.mode1,
- args.mode2) as fuzzer:
- fuzzer.Run()
-
-if __name__ == '__main__':
- main()
