From a68792c10bde1d694c6b71bfbbc2be6719c1156a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ivan=20=E2=80=9CCLOVIS=E2=80=9D=20Canet?= Date: Sun, 16 Jun 2024 11:22:26 +0200 Subject: [PATCH 1/4] fix(suite): Add the Long suffix to the seed generation message to ensure the hint compiles --- suite/src/commonMain/kotlin/Random.kt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/suite/src/commonMain/kotlin/Random.kt b/suite/src/commonMain/kotlin/Random.kt index 273ec91..a86369e 100644 --- a/suite/src/commonMain/kotlin/Random.kt +++ b/suite/src/commonMain/kotlin/Random.kt @@ -26,7 +26,7 @@ private class ConfiguredRandom( override fun toString() = "Random generator" + when (explicitlyChosen) { true -> " with the explicitly selected seed $seed" - false -> " with seed $seed. To reproduce this execution, add 'random.setSeed($seed)' at the start of the test, before any random generation." + false -> " with seed $seed. To reproduce this execution, add 'random.setSeed(${seed}L)' at the start of the test, before any random generation." } } -- 2.51.2 From cd3f41a2264dd9ea7b58edb220a7ece8c1a46894 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ivan=20=E2=80=9CCLOVIS=E2=80=9D=20Canet?= Date: Sun, 16 Jun 2024 11:23:44 +0200 Subject: [PATCH 2/4] docs(website): Write the random control page --- docs/website/docs/features/overview.md | 4 +- docs/website/docs/features/random.md | 100 +++++++++++++++++++++++++ docs/website/mkdocs.yml | 1 + 3 files changed, 104 insertions(+), 1 deletion(-) create mode 100644 docs/website/docs/features/random.md diff --git a/docs/website/docs/features/overview.md b/docs/website/docs/features/overview.md index 9f3f22d..b9d1fe5 100644 --- a/docs/website/docs/features/overview.md +++ b/docs/website/docs/features/overview.md @@ -26,7 +26,7 @@ fun SuiteDsl.testUsers( // (1)! suite("Test fixtures") { val adminEmail by prepared { // (5)! - "account-${random.nextInt()}@mail.com" + "account-${random.nextInt()}@mail.com" // (9)! } val admin by prepared { @@ -77,3 +77,5 @@ fun SuiteDsl.testUsers( // (1)! [Learn more](finalizers.md). 8. Tests can create asynchronous tasks that run in the foreground or in the background. [Learn more](async.md). +9. Generate randomized values with full seed control using the `random` helper. + [Learn more](random.md). diff --git a/docs/website/docs/features/random.md b/docs/website/docs/features/random.md new file mode 100644 index 0000000..c54c411 --- /dev/null +++ b/docs/website/docs/features/random.md @@ -0,0 +1,100 @@ +# Randomness control + +Randomness can be useful to generate arbitrary test data (e.g. for fuzzing, property testing, generating default values…). However, tests that exhibit randomness are quickly hard to debug: re-running them may give a different result! + +Prepared brings a very simple solution to this problem: + +- When a test uses random values, the initial seed is printed to the standard output, +- Users can force a specific seed value at the start of the test. + +Together, this allows users to easily replicate failures in CI on their local machines. Let's walk through an example. + +## Example walkthrough + +Let's imagine you are sending data to another system (e.g. a REST DTO, or storing something in a database…). You want to ensure the data you send and the data you receive are always exactly the same. You also want to ensure that injection attacks are not possible. + +**Proving that something is impossible is difficult. However, using a probabilistic approach, we can reduce the risks.** + +Let's write a simple probabilistic test: +```kotlin +test("Round trip serialization check") { + val payload = Array(random.nextInt(0, 64)) { random.nextInt().toChar() } + .joinToString("") + + check(deserialize(serialize(payload)) == payload) +} +``` + +This test generates a random string of data, makes it go through the round trip of our `serialize` and `deserialize` functions, and checks that the result is identical to the original data. + +??? danger "The difference between `random` and `Random`" + Note that the test uses `random`, and not `Random`. + + `Random` is the Kotlin standard library's default random source. If you use it directly, the behavior described on this page will not activate. Prepared will display a compile-time warning on usage of `Random` in a context where `random` is available. + +??? tip "Writing this kind of tests in a real situation" + This example was somewhat simplified for the purposes of simplicity. If you want to write a test like this in a real project, we recommend making a few modifications to it. + + First, we recommend generating real data, instead of a string representation. For example, generating instances of your model classes, and using those. + + Then, we recommend increasing the number of executions. Probabilistic tests are useful when rare cases happen relatively often, which requires a large amount of executions. For example, you may want to use `repeat` to increase the number of test executions: + ```kotlin + repeat(1000) { + test("Round trip serialization check $it/1000") { + // The test itself is unchanged + } + } + ``` + +When running this test, Prepared prints the following line: +```text +» Prepared ‘randomSource’: Random generator with seed 3286522734459043202. +To reproduce this execution, add 'random.setSeed(3286522734459043202L)' at +the start of the test, before any random generation. +``` + +If the test fails in your CI environment, or another developer's machine, you can explicitly set the seed at the start of the test to reproduce the same execution: +```kotlin hl_lines="2" +test("Round trip serialization check") { + random.setSeed(3286522734459043202L) + + val payload = Array(random.nextInt(0, 64)) { random.nextInt().toChar() } + .joinToString("") + + check(deserialize(serialize(payload)) == payload) +} +``` + +!!! danger "" + If `random.setSeed()` is called when after a seed has already been generated, Prepared throws an exception. + +!!! danger "Random generators are order-dependent" + Tests can only be reproduced if all calls to the random generator are executed in the exact same order each time. + + If you have to use random values in contexts where the order is unknown, instantiate them at the start of the test, and pass the variables to your different systems. + +## Alternative syntax + +As we have seen, we can use the `random.nextInt()` syntax to generate a controllable random value. If using [prepared values](prepared-values.md), Prepared offers a shorthand: `randomInt()`, which behaves exactly in the same way as other prepared values: +```kotlin +val adminId by randomInt(0, 1000) + +val admin by prepared { + User(id = adminId(), …) +} + +test("Test name") { + check(service.create(admin()).id == adminId()) +} +``` +The `adminId` is a randomized value. Like any other prepared value, each test will receive a different instance, and referring to it multiple times within a single test returns the same value each time. + +A benefit of this approach is that prepared values print their actual value to the standard output, so it becomes easier to see which values were generated by the test. + +## Interactions with fixtures + +Random control works in exactly the same way within [prepared values](prepared-values.md) as they do within a test, because prepared values are executed in the context of a specific test. + +Specifically, setting the seed explicitly must be done before referring to any prepared values that use random values. + +However, randomness control is not available within [shared values](shared-values.md), because their result must be shared between multiple tests, and therefore a single test couldn't set the seed for all of them. We advise against using shared values that contain randomness, as they lack any way to control it. diff --git a/docs/website/mkdocs.yml b/docs/website/mkdocs.yml index c766f47..d8ba513 100644 --- a/docs/website/mkdocs.yml +++ b/docs/website/mkdocs.yml @@ -92,6 +92,7 @@ nav: - features/shared-values.md - features/finalizers.md - features/async.md + - features/random.md - Best practices: - practices/overview.md -- 2.51.2 From f1a0a46fcdd4f26a4bd3670d00b7ad628db04614 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ivan=20=E2=80=9CCLOVIS=E2=80=9D=20Canet?= Date: Sun, 16 Jun 2024 11:25:47 +0200 Subject: [PATCH 3/4] docs(website): Advertise Power Assert in the home page --- docs/website/docs/index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/website/docs/index.md b/docs/website/docs/index.md index 94cd124..fad51da 100644 --- a/docs/website/docs/index.md +++ b/docs/website/docs/index.md @@ -46,6 +46,8 @@ The goal of Prepared is to simplify how we declare tests, how we go from a thoug Assertion libraries provide utilities to compare values. Popular choices are [Kotlin.test](https://kotlinlang.org/api/latest/kotlin.test/), [Kotest Assertions](https://kotest.io/docs/assertions/assertions.html), [Strikt](https://strikt.io/), [AssertK](https://github.com/willowtreeapps/assertk)… just use the one you prefer! +Instead of any specific assertion libraries, we recommend using [Power Assert](https://kotlinlang.org/docs/power-assert.html), which is able to generate good error messages from regular Kotlin code, without needing an assertion library at all. + ## Prepared isn't an IntelliJ plugin (yet?) Prepared is a simple Kotlin library. It doesn't have a Gradle plugin, nor does it have an IntelliJ plugin. Test are reported by the runner, so your IDE can display the test report. However, IntelliJ doesn't know which lines are tests or not, so it cannot display the small green triangle to select which tests to execute. -- 2.51.2 From 6a6237741187ac7c788bdc4305b33303e0448d9f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ivan=20=E2=80=9CCLOVIS=E2=80=9D=20Canet?= Date: Sun, 16 Jun 2024 11:26:42 +0200 Subject: [PATCH 4/4] docs(website): Fix the link to the runners page --- docs/website/docs/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/website/docs/index.md b/docs/website/docs/index.md index fad51da..c00569f 100644 --- a/docs/website/docs/index.md +++ b/docs/website/docs/index.md @@ -40,7 +40,7 @@ Additionally, Prepared exposes many advanced features: ## Prepared isn't a test runner -The goal of Prepared is to simplify how we declare tests, how we go from a thought to code. Test runners are libraries that execute test batteries and report results to your build system. Prepared isn't a test runner, but [it is compatible with a few existing ones](features/runners.md). +The goal of Prepared is to simplify how we declare tests, how we go from a thought to code. Test runners are libraries that execute test batteries and report results to your build system. Prepared isn't a test runner, but [it is compatible with a few existing ones](tutorials/getting-started.md#test-runners). ## Prepared isn't an assertion library -- 2.51.2