pkgs/watcher/test/directory_watcher/README.md - tools - Git at Google

 # End to end testing

 The `DirectoryWatcher` implementations combine information from OS events and
 filesystem polling, which leads to plenty of opportunities for data races
 between the two. There are also various data races related to OS event
 ordering and batching.

 The tests in `end_to_end_tests.dart` protect against both logical errors and
 races.

 All the tests work the same way: `ClientSimulator` uses a directory watcher to
 track the state of a directory, a series of filesystem changes are made in that
 directory using `FileChanger`, then `ClientSimulator` compares its inferred
 state with the actual state on disk.

 File contents vary only by length, so the file contents can be given as a single
 number in the logs.

 There are three types of error possible:

  - `ClientSimulator` thinks a file exists on disk, but it doesn't; "missing delete event"
  - `ClientSimulator` does not know about a file that exists on disk; "missing add event"
  - `ClientSimulator` knows about a file that exists on disk, but has not read
    it after it was updated, meaning it has a wrong value for its
    contents/length; "missing modify event"

 ## Example data race

 An example sequence of file operations that can cause a data race is moving a
 directory then making further modifications inside it. The OS events report the
 "new" directory but not its contents, so `DirectoryWatcher` has to list the
 contents. The list results and the OS events from subsequent operations can
 give contradictory information about the same file, with no way to know which
 is more recent and so correct. The implementations created with the help of
 these tests aim to detect such ambiguity and resolve it by polling again after
 the event arrives.

 ## Standalone tests

 The end to end tests that run on CI include a series of seeded pseudorandom file
 operation batches and a set of hardcoded tests that were derived from
 interesting random runs. These guard against common data races.

 But, they don't run for long enough on CI to give high confidence that there are
 no data races.

 So, when making changes that might affect data races it is recommended to run
 a longer "standalone" end to end test run. This should be done on whichever
 platform(s) are affected, Windows, Mac and/or Linux, by running the end to end
 test multiple times in parallel and overnight.

 ```
 # Launch in multiple terminals, enough to use 100% CPU.
 dart test/directory_watcher/end_to_end_test_runner.dart random

 # Or on Linux, install `parallel` and use that to run any number in parallel.
 parallel --ungroup --halt now,done=1 \
     -j 100 ::: \
     $(for i in (seq 1 100); do echo 'dart test/directory_watcher/end_to_end_test_runner.dart'; end)
 ```

 Run in this way the test runs until it hits a failure. If it does, it prints
 a link to a log which shows a combination of file operations `F`, watcher
 internals `W` and the events seen by the `ClientSimulator` marked `C`.
 It also prints the seed of the failure, which can be used to run the same
 pseudorandom batch of file operations to see if the exact same failure can
 be reproduced:

 ```
 dart test/directory_watcher/end_to_end_test_runner.dart seed 42
 ```

 Another way to rerun the exact same sequence of operations that failed is to
 copy the failure log into `end_to_end_tests.dart` as a test case. Only the lines
 that are file operations marked with `F` are needed. Then run:

 ```
 dart test/directory_watcher/end_to_end_test_runner.dart replay <test name>
 ```

 If a failure can be reproduced in this way then you can try removing
 irrelevant-seeming parts of the log until you have a minimal repro case. Note
 that a file operation that can't be carried out, for example a move into a
 directory that does not exist, is silently skipped over and does nothing.

 ## False negatives

 The standalone end to end tests have one known "false negative" issue, which
 is that very occasionally and under heavy load the test might not wait long
 enough before deciding that `ClientWatcher` has incorrect state. This can be
 noticed in the failure log if all the wrong tracking is about file events at the
 end of the run, and with no watcher log entries afterwards. Such failures can
 be ignored.

 TODO(davidmorgan): detect this automatically and wait longer instead of failing
 the test.
	# End to end testing

	The `DirectoryWatcher` implementations combine information from OS events and
	filesystem polling, which leads to plenty of opportunities for data races
	between the two. There are also various data races related to OS event
	ordering and batching.

	The tests in `end_to_end_tests.dart` protect against both logical errors and
	races.

	All the tests work the same way: `ClientSimulator` uses a directory watcher to
	track the state of a directory, a series of filesystem changes are made in that
	directory using `FileChanger`, then `ClientSimulator` compares its inferred
	state with the actual state on disk.

	File contents vary only by length, so the file contents can be given as a single
	number in the logs.

	There are three types of error possible:

	- `ClientSimulator` thinks a file exists on disk, but it doesn't; "missing delete event"
	- `ClientSimulator` does not know about a file that exists on disk; "missing add event"
	- `ClientSimulator` knows about a file that exists on disk, but has not read
	it after it was updated, meaning it has a wrong value for its
	contents/length; "missing modify event"

	## Example data race

	An example sequence of file operations that can cause a data race is moving a
	directory then making further modifications inside it. The OS events report the
	"new" directory but not its contents, so `DirectoryWatcher` has to list the
	contents. The list results and the OS events from subsequent operations can
	give contradictory information about the same file, with no way to know which
	is more recent and so correct. The implementations created with the help of
	these tests aim to detect such ambiguity and resolve it by polling again after
	the event arrives.

	## Standalone tests

	The end to end tests that run on CI include a series of seeded pseudorandom file
	operation batches and a set of hardcoded tests that were derived from
	interesting random runs. These guard against common data races.

	But, they don't run for long enough on CI to give high confidence that there are
	no data races.

	So, when making changes that might affect data races it is recommended to run
	a longer "standalone" end to end test run. This should be done on whichever
	platform(s) are affected, Windows, Mac and/or Linux, by running the end to end
	test multiple times in parallel and overnight.

	```
	# Launch in multiple terminals, enough to use 100% CPU.
	dart test/directory_watcher/end_to_end_test_runner.dart random

	# Or on Linux, install `parallel` and use that to run any number in parallel.
	parallel --ungroup --halt now,done=1 \
	-j 100 ::: \
	$(for i in (seq 1 100); do echo 'dart test/directory_watcher/end_to_end_test_runner.dart'; end)
	```

	Run in this way the test runs until it hits a failure. If it does, it prints
	a link to a log which shows a combination of file operations `F`, watcher
	internals `W` and the events seen by the `ClientSimulator` marked `C`.
	It also prints the seed of the failure, which can be used to run the same
	pseudorandom batch of file operations to see if the exact same failure can
	be reproduced:

	```
	dart test/directory_watcher/end_to_end_test_runner.dart seed 42
	```

	Another way to rerun the exact same sequence of operations that failed is to
	copy the failure log into `end_to_end_tests.dart` as a test case. Only the lines
	that are file operations marked with `F` are needed. Then run:

	```
	dart test/directory_watcher/end_to_end_test_runner.dart replay <test name>
	```

	If a failure can be reproduced in this way then you can try removing
	irrelevant-seeming parts of the log until you have a minimal repro case. Note
	that a file operation that can't be carried out, for example a move into a
	directory that does not exist, is silently skipped over and does nothing.

	## False negatives

	The standalone end to end tests have one known "false negative" issue, which
	is that very occasionally and under heavy load the test might not wait long
	enough before deciding that `ClientWatcher` has incorrect state. This can be
	noticed in the failure log if all the wrong tracking is about file events at the
	end of the run, and with no watcher log entries afterwards. Such failures can
	be ignored.

	TODO(davidmorgan): detect this automatically and wait longer instead of failing
	the test.