From 8461da0c90b4f33063dda3a78eb4e573f3457f73 Mon Sep 17 00:00:00 2001 From: Kyle Martin Date: Sat, 24 Aug 2019 11:00:41 +1200 Subject: [PATCH 1/2] Markdown lint and spellcheck --- readme.md | 369 ++++++++++++++++++------------------------------------ 1 file changed, 122 insertions(+), 247 deletions(-) diff --git a/readme.md b/readme.md index f53a65b4..7fc9da76 100644 --- a/readme.md +++ b/readme.md @@ -2,31 +2,32 @@
-# 👇 Why this guide can take your testing skills to the next level +# 👇 Why this guide can take your testing skills to the next level
## 📗 45+ best practices: Super-comprehensive and exhaustive -This is a guide for JavaScript & Node.js reliability from A-Z. It summarizes and curates for you dozens of the best blog posts, books and tools the market has to offer +This is a guide for JavaScript & Node.js reliability from A-Z. It summarizes and curates for you dozens of the best blog posts, books and tools the market has to offer ## 🚢 Advanced: Goes 10,000 miles beyond the basics -Hop into a journey that travels way beyond the basics into advanced topics like testing in production, mutation testing, property-based testing and many other strategic & professional tools. Should you read every word in this guide your testing skills are likely to go way above the average +Hop into a journey that travels way beyond the basics into advanced topics like testing in production, mutation testing, property-based testing and many other strategic & professional tools. Should you read every word in this guide your testing skills are likely to go way above the average ## 🌐 Full-stack: front, backend, CI, anything + Start by understanding the ubiquitous testing practices that are the foundation for any application tier. Then, delve into your area of choice: frontend/UI, backend, CI or maybe all of them?
### Written By Yoni Goldberg + * A JavaScript & Node.js consultant * 👨‍🏫 [My testing workshop](https://www.testjavascript.com) - learn about [my workshops](https://www.testjavascript.com) in Europe & US * [Follow me on Twitter ](https://twitter.com/goldbergyoni/) * Come hear me speak at [LA](https://js.la/), [Verona](https://2019.nodejsday.it/), [Kharkiv](https://kharkivjs.org/), [free webinar](https://zoom.us/webinar/register/1015657064375/WN_Lzvnuv4oQJOYey2jXNqX6A). Future events TBD * [My JavaScript Quality newsletter](https://testjavascript.com/newsletter/) - insights and content only on strategic matters -

## `Table of Contents` @@ -55,32 +56,29 @@ Watching the watchman - measuring test quality (4 bullets) Guidelines for CI in the JS world (9 bullets) -

- # Section 0️⃣: The Golden Rule
## ⚪️ 0. The Golden Rule: Design for lean testing -:white_check_mark: **Do:** -Testing code is not like production-code - design it to be dead-simple, short, abstraction-free, flat, delightful to work with, lean. One should look at a test and get the intent instantly. +:white_check_mark: **Do:** +Testing code is not like production-code - design it to be dead-simple, short, abstraction-free, flat, delightful to work with, lean. One should look at a test and get the intent instantly. Our minds are full with the main production code, we don't have 'headspace' for additional complexity. Should we try to squeeze yet another challenging code into our poor brain it will slow the team down which works against the reason we do testing. Practically this is where many teams just abandon testing. - + The tests are an opportunity for something else - a friendly and smiley assistant, one that it's delightful to work with and delivers great value for such a small investment. Science tells we have two brain systems: system 1 which is used for effortless activities like driving a car on an empty road and system 2 which is meant for complex and conscious operations like solving a math equation. Design your test for system 1, when looking at test code it should *feel* as easy as modifying an HTML document and not like solving 2X(17 × 24). This can be achieved by selectively cherry-picking techniques, tools and test targets that are cost-effective and provide great ROI. Test only as much as needed, strive to keep it nimble, sometimes it's even worth dropping some tests and trade reliability for agility and simplicity. ![alt text](/assets/headspace.png "We have no head room for additional complexity") - + Most of the advice below are derivatives of this principle. ### Ready to start? -

# Section 1: The Test Anatomy @@ -119,7 +117,7 @@ Most of the advice below are derivatives of this principle. describe('Products Service', function() { describe('Add new product', function() { //2. scenario and 3. expectation - it('When no price is specified, then the product status is pending approval', ()=> { + it('When no price is specified, then the product status is pending approval', () => { const newProduct = new ProductService().add(...); expect(newProduct.status).to.equal('pendingApproval'); }); @@ -127,6 +125,7 @@ describe('Products Service', function() { }); ``` +
### :clap: Doing It Right Example: A test name that constitutes 3 parts @@ -146,10 +145,8 @@ describe('Products Service', function() { 3rd A - Assert: Ensure that the received value satisfies the expectation. Usually 1 line of code -
- ❌ **Otherwise:** Not only you spend long daily hours on understanding the main code, now also what should have been the simple part of the day (testing) stretches your brain
@@ -160,9 +157,9 @@ describe('Products Service', function() { ### :clap: Doing It Right Example: A test structured with the AAA pattern -![](https://img.shields.io/badge/🔧%20Example%20using%20Jest-blue.svg - "Examples with Jest") ![](https://img.shields.io/badge/🔧%20Example%20using%20Mocha-blue.svg - "Examples with Jest") +![Jest example](https://img.shields.io/badge/🔧%20Example%20using%20Jest-blue.svg + "Examples with Jest") ![Mocha example](https://img.shields.io/badge/🔧%20Example%20using%20Mocha-blue.svg + "Examples with Moch") ```javascript describe('Customer classifier', () => { @@ -195,33 +192,26 @@ test('Should be classified as premium', () => { }); ``` - - -

- - - ## ⚪ ️1.3 Describe expectations in a product language: use BDD-style assertions :white_check_mark: **Do:** Coding your tests in a declarative-style allows the reader to get the grab instantly without spending even a single brain-CPU cycle. When you write an imperative code that is packed with conditional logic the reader is thrown away to an effortful mental mood. In that sense, code the expectation in a human-like language, declarative BDD style using expect or should and not using custom code. If Chai & Jest don’t include the desired assertion and it’s highly repeatable, consider [extending Jest matcher (Jest)](https://jestjs.io/docs/en/expect#expectextendmatchers) or writing a [custom Chai plugin](https://www.chaijs.com/guide/plugins/)
- ❌ **Otherwise:** The team will write less test and decorate the annoying ones with .skip()
✏ Code Examples
-![](https://img.shields.io/badge/🔧%20Example%20using%20Mocha-blue.svg - "Examples with Mocha & Chai") ![](https://img.shields.io/badge/🔧%20Example%20using%20Jest-blue.svg +![Examples with Mocha & Chai](https://img.shields.io/badge/🔧%20Example%20using%20Mocha-blue.svg + "Examples with Mocha & Chai") ![Examples with Jest](https://img.shields.io/badge/🔧%20Example%20using%20Jest-blue.svg "Examples with Jest") - ### :thumbsdown: Anti Pattern Example: The reader must skim through not so short, and imperative code just to get the test story +### :thumbsdown: Anti Pattern Example: The reader must skim through not so short, and imperative code just to get the test story ```javascript test("When asking for an admin, ensure only ordered admins in results" , () => { @@ -248,11 +238,11 @@ test("When asking for an admin, ensure only ordered admins in results" , () => { }); ``` +
### :clap: Doing It Right Example: Skimming through the following declarative test is a breeze - ```javascript it("When asking for an admin, ensure only ordered admins in results" , () => { //assuming we've added here two admins @@ -266,16 +256,13 @@ it("When asking for an admin, ensure only ordered admins in results" , () => {
-

- -## ⚪ ️ 1.4 Stick to black-box testing: Test only public methods +## ⚪ 1.4 Stick to black-box testing: Test only public methods :white_check_mark: **Do:** Testing the internals brings huge overhead for almost nothing. If your code/API deliver the right results, should you really invest your next 3 hours in testing HOW it worked internally and then maintain these fragile tests? Whenever a public behavior is checked, the private implementation is also implicitly tested and your tests will break only if there is a certain problem (e.g. wrong output). This approach is also referred to as behavioral testing. On the other side, should you test the internals (white box approach) — your focus shifts from planning the component outcome to nitty-gritty details and your test might break because of minor code refactors although the results are fine- this dramatically increases the maintenance burden
- ❌ **Otherwise:** Your test behaves like the [child who cries wolf](https://en.wikipedia.org/wiki/The_Boy_Who_Cried_Wolf): shoot out loud false-positive cries (e.g., A test fails because a private variable name was changed). Unsurprisingly, people will soon start to ignore the CI notifications until someday a real bug will get ignored…
@@ -284,8 +271,10 @@ it("When asking for an admin, ensure only ordered admins in results" , () => {
### :thumbsdown: Anti Pattern Example: A test case is testing the internals for no good reason -![](https://img.shields.io/badge/🔧%20Example%20using%20Mocha-blue.svg + +![Examples with Mocha & Chai](https://img.shields.io/badge/🔧%20Example%20using%20Mocha-blue.svg "Examples with Mocha & Chai") + ```javascript class ProductService{ //this method is only used internally @@ -311,20 +300,17 @@ it("White-box test: When the internal methods get 0 vat, it return 0 response", - - -

-## ⚪ ️ ️1.5 Choose the right test doubles: Avoid mocks in favor of stubs and spies +## ⚪ ️1.5 Choose the right test doubles: Avoid mocks in favor of stubs and spies :white_check_mark: **Do:** Test doubles are a necessary evil because they are coupled to the application internals, yet some provide an immense value ([Read here a reminder about test doubles: mocks vs stubs vs spies](https://martinfowler.com/articles/mocksArentStubs.html)). Before using test doubles, ask a very simple question: Do I use it to test functionality that appears, or could appear, in the requirements document? If no, it’s a smell of white-box testing. For example, if you want to test what your app behaves reasonably when the payment service is down, you might stub the payment service and trigger some ‘No Response’ return to ensure that the unit under test returns the right value. This checks our application behavior/response/outcome under certain scenarios. You might also use a spy to assert that an email was sent when that service is down — this is again a behavioral check which is likely to appear in a requirements doc (“Send an email if payment couldn’t be saved”). On the flip side, if you mock the Payment service and ensure that it was called with the right JavaScript types — then your test is focused on internal things that got nothing with the application functionality and are likely to change frequently -
+
❌ **Otherwise:** Any refactoring of code mandates searching for all the mocks in the code and updating accordingly. Tests become a burden rather than a helpful friend @@ -335,8 +321,9 @@ For example, if you want to test what your app behaves reasonably when the payme
### :thumbsdown: Anti-pattern example: Mocks focus on the internals -![](https://img.shields.io/badge/🔧%20Example%20using%20Sinon-blue.svg +![Examples with Mocha & Chai](https://img.shields.io/badge/🔧%20Example%20using%20Sinon-blue.svg "Examples with Mocha & Chai") + ```javascript it("When a valid product is about to be deleted, ensure data access DAL was called once, with the right product and right config", async () => { //Assume we already added a product @@ -347,6 +334,7 @@ it("When a valid product is about to be deleted, ensure data access DAL was call dataAccessMock.verify(); }); ``` +
### :clap:Doing It Right Example: spies are focused on testing the requirements but as a side-effect are unavoidably touching to the internals @@ -362,8 +350,6 @@ it("When a valid product is about to be deleted, ensure an email is sent", async - -

## ⚪ ️1.6 Don’t “foo”, use realistic input data @@ -371,10 +357,8 @@ it("When a valid product is about to be deleted, ensure an email is sent", async :white_check_mark: **Do:** Often production bugs are revealed under some very specific and surprising input — the more realistic the test input is, the greater the chances are to catch bugs early. Use dedicated libraries like [Faker](https://www.npmjs.com/package/faker) to generate pseudo-real data that resembles the variety and form of production data. For example, such libraries can generate realistic phone numbers, usernames, credit card, company names, and even ‘lorem ipsum’ text. You may also create some tests (on top of unit tests, not instead) that randomize fakers data to stretch your unit under test or even import real data from your production environment. Want to take it to the next level? see next bullet (property-based testing).
- ❌ **Otherwise:** All your development testing will falsely seem green when you use synthetic inputs like “Foo” but then production might turn red when a hacker passes-in a nasty string like “@3e2ddsf . ##’ 1 fdsfds . fds432 AAAA” -
✏ Code Examples @@ -383,10 +367,9 @@ it("When a valid product is about to be deleted, ensure an email is sent", async ### :thumbsdown: Anti-Pattern Example: A test suite that passes due to non-realistic data -![](https://img.shields.io/badge/🔧%20Example%20using%20Jest-blue.svg +![Examples with Jest](https://img.shields.io/badge/🔧%20Example%20using%20Jest-blue.svg "Examples with Jest") - - + ```javascript const addProduct = (name, price) =>{ const productNameRegexNoSpace = /^\S*$/;//no white-space allowd @@ -407,9 +390,11 @@ test("Wrong: When adding new product with valid properties, get successful confi }); ``` +
### :clap:Doing It Right Example: Randomizing realistic input + ```javascript it("Better: When adding new valid product, get successful confirmation", async () => { const addProductResult = addProduct(faker.commerce.productName(), faker.random.number()); @@ -422,9 +407,6 @@ it("Better: When adding new valid product, get successful confirmation", async (
- - -

## ⚪ ️ 1.7 Test many input combinations using Property-based testing @@ -432,10 +414,8 @@ it("Better: When adding new valid product, get successful confirmation", async ( :white_check_mark: **Do:** Typically we choose a few input samples for each test. Even when the input format resembles real-world data (see bullet ‘Don’t foo’), we cover only a few input combinations (method(‘’, true, 1), method(“string” , false” , 0)), However, in production, an API that is called with 5 parameters can be invoked with thousands of different permutations, one of them might render our process down ([see Fuzz Testing](https://en.wikipedia.org/wiki/Fuzzing)). What if you could write a single test that sends 1000 permutations of different inputs automatically and catches for which input our code fails to return the right response? Property-based testing is a technique that does exactly that: by sending all the possible input combinations to your unit under test it increases the serendipity of finding a bug. For example, given a method — addNewProduct(id, name, isDiscount) — the supporting libraries will call this method with many combinations of (number, string, boolean) like (1, “iPhone”, false), (2, “Galaxy”, true). You can run property-based testing using your favorite test runner (Mocha, Jest, etc) using libraries like [js-verify](https://github.com/jsverify/jsverify) or [testcheck](https://github.com/leebyron/testcheck-js) (much better documentation). Update: Nicolas Dubien suggests in the comments below to [checkout fast-check](https://github.com/dubzzz/fast-check#readme) which seems to offer some additional features and also to be actively maintained
- ❌ **Otherwise:** Unconsciously, you choose the test inputs that cover only code paths that work well. Unfortunately, this decreases the efficiency of testing as a vehicle to expose bugs -
✏ Code Examples @@ -444,7 +424,7 @@ it("Better: When adding new valid product, get successful confirmation", async ( ### :clap: Doing It Right Example: Testing many input permutations with “mocha-testcheck” -![](https://img.shields.io/badge/🔧%20Example%20using%20Mocha-blue.svg +![Examples with Jest](https://img.shields.io/badge/🔧%20Example%20using%20Mocha-blue.svg "Examples with Jest") ```javascript @@ -465,9 +445,6 @@ describe('Product service', () => {
- - -

## ⚪ ️ 1.8 If needed, use only short & inline snapshots @@ -479,7 +456,7 @@ On the other hand, ‘classic snapshots’ tutorials and tools encourage to stor It’s worth noting that there are few cases where long & external snapshots are acceptable - when asserting on schema and not data (extracting out values and focusing on fields) or when the received document rarely changes
-❌ **Otherwise:** A UI test fails. The code seems right, the screen renders perfect pixels, what happened? your snapshot testing just found a difference from the origin document to current received one - a single space character was added to the markdown... +❌ **Otherwise:** A UI test fails. The code seems right, the screen renders perfect pixels, what happened? your snapshot testing just found a difference from the origin document to current received one - a single space character was added to the markdown...
@@ -489,11 +466,11 @@ It’s worth noting that there are few cases where long & external snapshots are ### :thumbsdown: Anti-Pattern Example: Coupling our test to unseen 2000 lines of code -![](https://img.shields.io/badge/🔧%20Example%20using%20Jest-blue.svg +![Examples with Jest](https://img.shields.io/badge/🔧%20Example%20using%20Jest-blue.svg "Examples with Jest") - + ```javascript -it('TestJavaScript.com is renderd correctly', () => { +it('TestJavaScript.com is renderd correctly', () => { //Arrange @@ -509,9 +486,11 @@ expect(receivedPage).toMatchSnapshot(); }); ``` +
### :clap: Doing It Right Example: Expectations are visible and focused + ```javascript it('When visiting TestJavaScript.com home page, a menu is displayed', () => { //Arrange @@ -536,7 +515,6 @@ expect(menu).toMatchInlineSnapshot(` -

## ⚪ ️1.9 Avoid global test fixtures and seeds, add data per-test @@ -544,10 +522,8 @@ expect(menu).toMatchInlineSnapshot(` :white_check_mark: **Do:** Going by the golden rule (bullet 0), each test should add and act on its own set of DB rows to prevent coupling and easily reason about the test flow. In reality, this is often violated by testers who seed the DB with data before running the tests ([also known as ‘test fixture’](https://en.wikipedia.org/wiki/Test_fixture)) for the sake of performance improvement. While performance is indeed a valid concern — it can be mitigated (see “Component testing” bullet), however, test complexity is a much painful sorrow that should govern other considerations most of the time. Practically, make each test case explicitly add the DB records it needs and act only on those records. If performance becomes a critical concern — a balanced compromise might come in the form of seeding the only suite of tests that are not mutating data (e.g. queries)
- ❌ **Otherwise:** Few tests fail, a deployment is aborted, our team is going to spend precious time now, do we have a bug? let’s investigate, oh no — it seems that two tests were mutating the same seed data -
✏ Code Examples @@ -556,9 +532,9 @@ expect(menu).toMatchInlineSnapshot(` ### :thumbsdown: Anti Pattern Example: tests are not independent and rely on some global hook to feed global DB data -![](https://img.shields.io/badge/🔧%20Example%20using%20Mocha-blue.svg +![Examples with Jest](https://img.shields.io/badge/🔧%20Example%20using%20Mocha-blue.svg "Examples with Jest") - + ```javascript before(() => { //adding sites and admins data to our DB. Where is the data? outside. At some external json or migration framework @@ -577,6 +553,7 @@ it("When querying by site name, get the right site", async () => { }); ``` +
### :clap: Doing It Right Example: We can stay within the test, each test acts on its own set of data @@ -597,7 +574,6 @@ it("When updating site name, get successful confirmation", async () => {
-
## ⚪ ️ 1.10 Don’t catch errors, expect them @@ -606,10 +582,8 @@ it("When updating site name, get successful confirmation", async () => { A more elegant alternative is the using the one-line dedicated Chai assertion: expect(method).to.throw (or in Jest: expect(method).toThrow()). It’s absolutely mandatory to also ensure the exception contains a property that tells the error type, otherwise given just a generic error the application won’t be able to do much rather than show a disappointing message to the user
- ❌ **Otherwise:** It will be challenging to infer from the test reports (e.g. CI reports) what went wrong -
✏ Code Examples @@ -618,9 +592,9 @@ A more elegant alternative is the using the one-line dedicated Chai assertion: e ### :thumbsdown: Anti-pattern Example: A long test case that tries to assert the existence of error with try-catch -![](https://img.shields.io/badge/🔧%20Example%20using%20Mocha-blue.svg +![Examples with Jest](https://img.shields.io/badge/🔧%20Example%20using%20Mocha-blue.svg "Examples with Jest") - + ```javascript it("When no product name, it throws error 400", async() => { let errorWeExceptFor = null; @@ -636,6 +610,7 @@ expect(errorWeExceptFor).not.to.be.null; }); ``` +
### :clap: Doing It Right Example: A human-readable expectation that could be understood easily, maybe even by QA or technical PM @@ -649,9 +624,6 @@ it.only("When no product name, it throws error 400", async() => {
- - -

## ⚪ ️ 1.11 Tag your tests @@ -659,10 +631,8 @@ it.only("When no product name, it throws error 400", async() => { :white_check_mark: **Do:** Different tests must run on different scenarios: quick smoke, IO-less, tests should run when a developer saves or commits a file, full end-to-end tests usually run when a new pull request is submitted, etc. This can be achieved by tagging tests with keywords like #cold #api #sanity so you can grep with your testing harness and invoke the desired subset. For example, this is how you would invoke only the sanity test group with Mocha: mocha — grep ‘sanity’
- ❌ **Otherwise:** Running all the tests, including tests that perform dozens of DB queries, any time a developer makes a small change can be extremely slow and keeps developers away from running tests -
✏ Code Examples @@ -671,10 +641,11 @@ it.only("When no product name, it throws error 400", async() => { ### :clap: Doing It Right Example: Tagging tests as ‘#cold-test’ allows the test runner to execute only fast tests (Cold===quick tests that are doing no IO and can be executed frequently even as the developer is typing) -![](https://img.shields.io/badge/🔧%20Example%20using%20Jest-blue.svg +![Examples with Jest](https://img.shields.io/badge/🔧%20Example%20using%20Jest-blue.svg "Examples with Jest") + ```javascript -//this test is fast (no DB) and we're tagging it correspondigly +//this test is fast (no DB) and we're tagging it correspondingly //now the user/CI can run it frequently describe('Order service', function() { describe('Add new order #cold-test #sanity', function() { @@ -689,23 +660,19 @@ describe('Order service', function() {
- - -

## ⚪ ️1.12 Other generic good testing hygiene + :white_check_mark: **Do:** This post is focused on testing advice that is related to, or at least can be exemplified with Node JS. This bullet, however, groups few non-Node related tips that are well-known -Learn and practice [TDD principles](https://www.sm-cloud.com/book-review-test-driven-development-by-example-a-tldr/) — they are extremely valuable for many but don’t get intimidated if they don’t fit your style, you’re not the only one. Consider writing the tests before the code in a [red-green-refactor style](https://blog.cleancoder.com/uncle-bob/2014/12/17/TheCyclesOfTDD.html), ensure each test checks exactly one thing, when you find a bug — before fixing write a test that will detect this bug in the future, let each test fail at least once before turning green, start a module by writing a quick and simplistic code that satsifies the test - then refactor gradually and take it to a production grade level, avoid any dependency on the environment (paths, OS, etc) +Learn and practice [TDD principles](https://www.sm-cloud.com/book-review-test-driven-development-by-example-a-tldr/) — they are extremely valuable for many but don’t get intimidated if they don’t fit your style, you’re not the only one. Consider writing the tests before the code in a [red-green-refactor style](https://blog.cleancoder.com/uncle-bob/2014/12/17/TheCyclesOfTDD.html), ensure each test checks exactly one thing, when you find a bug — before fixing write a test that will detect this bug in the future, let each test fail at least once before turning green, start a module by writing a quick and simplistic code that satisfies the test - then refactor gradually and take it to a production grade level, avoid any dependency on the environment (paths, OS, etc)
- ❌ **Otherwise:** You‘ll miss pearls of wisdom that were collected for decades

- # Section 2️⃣: Backend Testing ## ⚪ ️2.1 Enrich your testing portfolio: Look beyond unit tests and the pyramid @@ -720,10 +687,8 @@ A word of caution: the TDD argument in the software world takes a typical false-
- ❌ **Otherwise:** You’re going to miss some tools with amazing ROI, some like Fuzz, lint, and mutation can provide value in 10 minutes -
✏ Code Examples @@ -731,6 +696,7 @@ A word of caution: the TDD argument in the software world takes a typical false-
### :clap: Doing It Right Example: Cindy Sridharan suggests a rich testing portfolio in her amazing post ‘Testing Microservices — the sane way’ + ![alt text](assets/bp-12-rich-testing.jpeg "Cindy Sridharan suggests a rich testing portfolio in her amazing post ‘Testing Microservices — the sane way’") ☺️Example: [YouTube: “Beyond Unit Tests: 5 Shiny Node.JS Test Types (2018)” (Yoni Goldberg)](https://www.youtube.com/watch?v=-2zP494wdUY&feature=youtu.be) @@ -739,12 +705,8 @@ A word of caution: the TDD argument in the software world takes a typical false- ![alt text](assets/bp-12-Yoni-Goldberg-Testing.jpeg "A test name that constitutes 3 parts") -
- - -

## ⚪ ️2.2 Component testing might be your best affair @@ -754,10 +716,8 @@ A word of caution: the TDD argument in the software world takes a typical false- Component tests focus on the Microservice ‘unit’, they work against the API, don’t mock anything which belongs to the Microservice itself (e.g. real DB, or at least the in-memory version of that DB) but stub anything that is external like calls to other Microservices. By doing so, we test what we deploy, approach the app from outwards to inwards and gain great confidence in a reasonable amount of time.
- ❌ **Otherwise:** You may spend long days on writing unit tests to find out that you got only 20% system coverage -
✏ Code Examples @@ -766,7 +726,7 @@ Component tests focus on the Microservice ‘unit’, they work against the API, ### :clap: Doing It Right Example: Supertest allows approaching Express API in-process (fast and cover many layers) -![](https://img.shields.io/badge/🔧%20Example%20using%20Mocha-blue.svg +![Examples with Jest](https://img.shields.io/badge/🔧%20Example%20using%20Mocha-blue.svg "Examples with Jest") ![alt text](assets/bp-13-component-test-yoni-goldberg.png " [Supertest](https://www.npmjs.com/package/supertest) allows approaching Express API in-process (fast and cover many layers)") @@ -780,10 +740,8 @@ Component tests focus on the Microservice ‘unit’, they work against the API, :white_check_mark: **Do:** So your Microservice has multiple clients, and you run multiple versions of the service for compatibility reasons (keeping everyone happy). Then you change some field and ‘boom!’, some important client who relies on this field is angry. This is the Catch-22 of the integration world: It’s very challenging for the server side to consider all the multiple client expectations — On the other hand, the clients can’t perform any testing because the server controls the release dates. [Consumer-driven contracts and the framework PACT](https://docs.pact.io/) were born to formalize this process with a very disruptive approach — not the server defines the test plan of itself rather the client defines the tests of the… server! PACT can record the client expectation and put in a shared location, “broker”, so the server can pull the expectations and run on every build using PACT library to detect broken contracts — a client expectation that is not met. By doing so, all the server-client API mismatches are caught early during build/CI and might save you a great deal of frustration
- ❌ **Otherwise:** The alternatives are exhausting manual testing or deployment fear -
✏ Code Examples @@ -792,28 +750,22 @@ Component tests focus on the Microservice ‘unit’, they work against the API, ### :clap: Doing It Right Example: -![](https://img.shields.io/badge/🔧%20Example%20using%20PACT-blue.svg +![Examples with PACT](https://img.shields.io/badge/🔧%20Example%20using%20PACT-blue.svg "Examples with PACT") ![alt text](assets/bp-14-testing-best-practices-contract-flow.png ) -
- -

- ## ⚪ ️ 2.4 Test your middlewares in isolation :white_check_mark: **Do:** Many avoid Middleware testing because they represent a small portion of the system and require a live Express server. Both reasons are wrong — Middlewares are small but affect all or most of the requests and can be tested easily as pure functions that get {req,res} JS objects. To test a middleware function one should just invoke it and spy ([using Sinon for example](https://www.npmjs.com/package/sinon)) on the interaction with the {req,res} objects to ensure the function performed the right action. The library [node-mock-http](https://www.npmjs.com/package/node-mocks-http) takes it even further and factors the {req,res} objects along with spying on their behavior. For example, it can assert whether the http status that was set on the res object matches the expectation (See example below)
- ❌ **Otherwise:** A bug in Express middleware === a bug in all or most requests -
✏ Code Examples @@ -822,14 +774,14 @@ Component tests focus on the Microservice ‘unit’, they work against the API, ### :clap:Doing It Right Example: Testing middleware in isolation without issuing network calls and waking-up the entire Express machine -![](https://img.shields.io/badge/🔧%20Example%20using%20Jest-blue.svg +![Examples with Jest](https://img.shields.io/badge/🔧%20Example%20using%20Jest-blue.svg "Examples with Jest") ```javascript //the middleware we want to test const unitUnderTest = require('./middleware') const httpMocks = require('node-mocks-http'); -//Jest syntax, equivelant to describe() & it() in Mocha +//Jest syntax, equivalent to describe() & it() in Mocha test('A request without authentication header, should return http status 403', () => { const request = httpMocks.createRequest({ method: 'GET', @@ -847,22 +799,18 @@ test('A request without authentication header, should return http status 403', (
- - -

## ⚪ ️2.5 Measure and refactor using static analysis tools + :white_check_mark: **Do:** Using static analysis tools helps by giving objective ways to improve code quality and keep your code maintainable. You can add static analysis tools to your CI build to abort when it finds code smells. Its main selling points over plain linting are the ability to inspect quality in the context of multiple files (e.g. detect duplications), perform advanced analysis (e.g. code complexity) and follow the history and progress of code issues. Two examples of tools you can use are [Sonarqube](https://www.sonarqube.org/) (2,600+ [stars](https://github.com/SonarSource/sonarqube)) and [Code Climate](https://codeclimate.com/) (1,500+ [stars](https://github.com/codeclimate/codeclimate)) Credit:: [Keith Holliday](https://github.com/TheHollidayInn)
- ❌ **Otherwise:** With poor code quality, bugs and performance will always be an issue that no shiny new library or state of the art features can fix -
✏ Code Examples @@ -871,26 +819,22 @@ Credit:: - +![alt text](assets/bp-16-yoni-goldberg-quality.png " CodeClimate, a commercial tool that can identify complex methods:") +


## ⚪ ️ 2.6 Check your readiness for Node-related chaos + :white_check_mark: **Do:** Weirdly, most software testings are about logic & data only, but some of the worst things that happen (and are really hard to mitigate ) are infrastructural issues. For example, did you ever test what happens when your process memory is overloaded, or when the server/process dies, or does your monitoring system realizes when the API becomes 50% slower?. To test and mitigate these type of bad things — [Chaos engineering](https://principlesofchaos.org/) was born by Netflix. It aims to provide awareness, frameworks and tools for testing our app resiliency for chaotic issues. For example, one of its famous tools, [the chaos monkey](https://github.com/Netflix/chaosmonkey), randomly kills servers to ensure that our service can still serve users and not relying on a single server (there is also a Kubernetes version, [kube-monkey](https://github.com/asobti/kube-monkey), that kills pods). All these tools work on the hosting/platform level, but what if you wish to test and generate pure Node chaos like check how your Node process copes with uncaught errors, unhandled promise rejection, v8 memory overloaded with the max allowed of 1.7GB or whether your UX stays satisfactory when the event loop gets blocked often? to address this I’ve written, [node-chaos](https://github.com/i0natan/node-chaos-monkey) (alpha) which provides all sort of Node-related chaotic acts
- ❌ **Otherwise:** No escape here, Murphy’s law will hit your production without mercy -
✏ Code Examples @@ -898,6 +842,7 @@ Credit:: @@ -909,10 +854,8 @@ Credit:: { //adding sites and admins data to our DB. Where is the data? outside. At some external json or migration framework @@ -942,6 +885,7 @@ it("When querying by site name, get the right site", async () => { }); ``` +
### :clap: Doing It Right Example: We can stay within the test, each test acts on its own set of data @@ -968,12 +912,10 @@ it("When updating site name, get successful confirmation", async () => { :white_check_mark: **Do:** When focusing on testing component logic, UI details become a noise that should be extracted, so your tests can focus on pure data. Practically, extract the desired data from the markup in an abstract way that is not too coupled to the graphic implementation, assert only on pure data (vs HTML/CSS graphic details) and disable animations that slow down. You might get tempted to avoid rendering and test only the back part of the UI (e.g. services, actions, store) but this will result in fictional tests that don't resemble the reality and won't reveal cases where the right data doesn't even arrive in the UI -
❌ **Otherwise:** The pure calculated data of your test might be ready in 10ms, but then the whole test will last 500ms (100 tests = 1 min) due to some fancy and irrelevant animation -
✏ Code Examples @@ -982,8 +924,8 @@ it("When updating site name, get successful confirmation", async () => { ### :clap: Doing It Right Example: Separating out the UI details -![](https://img.shields.io/badge/🔧%20Example%20using%20React-blue.svg - "Examples with React") ![](https://img.shields.io/badge/🔧%20Example%20using%20React%20Testing%20Library-blue.svg +![Examples with React](https://img.shields.io/badge/🔧%20Example%20using%20React-blue.svg + "Examples with React") ![Examples with react-testing-library](https://img.shields.io/badge/🔧%20Example%20using%20React%20Testing%20Library-blue.svg "Examples with react-testing-library") ```javascript @@ -1008,6 +950,7 @@ test('When users-list is flagged to show only VIP, should display only VIP membe
### :thumbsdown: Anti Pattern Example: Assertion mix UI details and data + ```javascript test('When flagging to show only VIP, should display only VIP members', () => { // Arrange @@ -1027,12 +970,8 @@ test('When flagging to show only VIP, should display only VIP members', () => {
- - -

- ## ⚪ ️ 3.2 Query HTML elements based on attributes that are unlikely to change :white_check_mark: **Do:** Query HTML elements based on attributes that are likely to survive graphic changes unlike CSS selectors and like form labels. If the designated element doesn't have such attributes, create a dedicated test attribute like 'test-id-submit-button'. Going this route not only ensures that your functional/logic tests never break because of look & feel changes but also it becomes clear to the entire team that this element and attribute are utilized by tests and shouldn't get removed @@ -1047,11 +986,11 @@ test('When flagging to show only VIP, should display only VIP members', () => {
-### :clap: Doing It Right Example: Querying an element using a dedicated attrbiute for testing +### :clap: Doing It Right Example: Querying an element using a dedicated attribute for testing -![](https://img.shields.io/badge/🔧%20Example%20using%20React-blue.svg +![Examples with React](https://img.shields.io/badge/🔧%20Example%20using%20React-blue.svg "Examples with React") - + ```html // the markup code (part of React component)

@@ -1068,8 +1007,8 @@ test('When flagging to show only VIP, should display only VIP members', () => { const metricValue = undefined; // Act - const { getByTestId } = render(); - + const { getByTestId } = render(); + expect(getByTestId('errorsLabel')).text()).toBe("0"); }); @@ -1078,26 +1017,22 @@ test('When flagging to show only VIP, should display only VIP members', () => {
### :thumbsdown: Anti-Pattern Example: Relying on CSS attributes + ```html -{value} +{value} ``` ```javascript -// this exammple is using enzyme +// this example is using enzyme test('Whenever no data is passed, error metric shows zero', () => { // ... - expect(wrapper.find("[className='d-flex-column']").text()).toBe("0"); }); ``` -

- - -
## ⚪ ️ 3.3 Whenever possible, test with a realistic and fully rendered component @@ -1110,7 +1045,6 @@ With all that said, a word of caution is in order: this technique works for smal ❌ **Otherwise:** When poking into a component's internal by invoking its private methods, and checking the inner state - you would have to refactor all tests when refactoring the components implementation. Do you really have a capacity for this level of maintenance? -
✏ Code Examples @@ -1119,10 +1053,10 @@ With all that said, a word of caution is in order: this technique works for smal ### :clap: Doing It Right Example: Working realstically with a fully rendered component -![](https://img.shields.io/badge/🔧%20Example%20using%20React-blue.svg - "Examples with React") ![](https://img.shields.io/badge/🔧%20Example%20using%20Enzyme-blue.svg +![Examples with React](https://img.shields.io/badge/🔧%20Example%20using%20React-blue.svg + "Examples with React") ![Examples with Enzyme](https://img.shields.io/badge/🔧%20Example%20using%20Enzyme-blue.svg "Examples with Enzyme") - + ```javascript class Calendar extends React.Component { static defaultProps = {showFilters: false} @@ -1154,6 +1088,7 @@ test('Realistic approach: When clicked to show filters, filters are displayed', ``` ### :thumbsdown: Anti-Pattern Example: Mocking the reality with shallow rendering + ```javascript test('Shallow/mocked approach: When clicked to show filters, filters are displayed', () => { @@ -1175,7 +1110,6 @@ test('Shallow/mocked approach: When clicked to show filters, filters are display
- ## ⚪ ️ 3.4 Don't sleep, use frameworks built-in support for async events. Also try to speed things up :white_check_mark: **Do:** In many cases, the unit under test completion time is just unknown (e.g. animation suspends element appearance) - in that case, avoid sleeping (e.g. setTimeOut) and prefer more deterministic methods that most platforms provide. Some libraries allows awaiting on operations (e.g. [Cypress cy.request('url')](https://docs.cypress.io/guides/references/best-practices.html#Unnecessary-Waiting)), other provide API for waiting like [@testing-library/dom method wait(expect(element))](https://testing-library.com/docs/guide-disappearance). Sometimes a more elegant way is to stub the slow resource, like API for example, and then once the response moment becomes deterministic the component can be explicitly re-rendered. When depending upon some external component that sleeps, it might turn useful to [hurry-up the clock](https://jestjs.io/docs/en/timer-mocks). Sleeping is a pattern to avoid because it forces your test to be slow or risky (when waiting for a too short period). Whenever sleeping and polling is inevitable and there's no support from the testing framework, some npm libraries like [wait-for-expect](https://www.npmjs.com/package/wait-for-expect) can help with a semi-deterministic solution @@ -1183,7 +1117,6 @@ test('Shallow/mocked approach: When clicked to show filters, filters are display ❌ **Otherwise:** When sleeping for a long time, tests will be an order of magnitude slower. When trying to sleep for small numbers, test will fail when the unit under test didn't respond in a timely fashion. So it boils down to a trade-off between flakiness and bad performance -
✏ Code Examples @@ -1192,8 +1125,8 @@ test('Shallow/mocked approach: When clicked to show filters, filters are display ### :clap: Doing It Right Example: E2E API that resolves only when the async operations is done (Cypress) -![](https://img.shields.io/badge/🔧%20Example%20using%20React-blue.svg - "Examples with React") ![](https://img.shields.io/badge/🔧%20Example%20using%20React%20Testing%20Library-blue.svg +![Examples with React](https://img.shields.io/badge/🔧%20Example%20using%20React-blue.svg + "Examples with React") ![Examples with react-testing-library](https://img.shields.io/badge/🔧%20Example%20using%20React%20Testing%20Library-blue.svg "Examples with react-testing-library") ```javascript @@ -1223,6 +1156,7 @@ test('movie title appears', async () => { ``` ### :thumbsdown: Anti-Pattern Example: custom sleep code + ```javascript test('movie title appears', async () => { @@ -1235,7 +1169,6 @@ test('movie title appears', async () => { clearInterval(interval); expect(getByText('the lion king')).toBeInTheDocument(); } - }, 100); // wait for appearance and return the element @@ -1246,12 +1179,11 @@ test('movie title appears', async () => {
-
## ⚪ ️ 3.5. Watch how the content is served over the network -![](https://img.shields.io/badge/🔧%20Example%20using%20Google%20LightHouse-blue.svg +![Examples with Lighthouse](https://img.shields.io/badge/🔧%20Example%20using%20Google%20LightHouse-blue.svg "Examples with Lighthouse") ✅ **Do:** Apply some active monitor that ensures the page load under real network is optimized - this includes any UX concern like slow page load or un-minified bundle. The inspection tools market is no short: basic tools like [pingdom](https://www.pingdom.com/), AWS CloudWatch, [gcp StackDriver](https://cloud.google.com/monitoring/uptime-checks/) can be easily configured to watch whether the server is alive and response under a reasonable SLA. This only scratches the surface of what might get wrong, hence it's preferable to opt for tools that specialize in frontend (e.g. [lighthouse](https://developers.google.com/web/tools/lighthouse/), [pagespeed](https://developers.google.com/speed/pagespeed/insights/)) and perform richer analysis. The focus should be on symptoms, metrics that directly affect the UX, like page load time, [meaningful paint](https://scotch.io/courses/10-web-performance-audit-tips-for-your-next-billion-users-in-2018/fmp-first-meaningful-paint), [time until the page gets interactive (TTI)](https://calibreapp.com/blog/time-to-interactive/). On top of that, one may also watch for technical causes like ensuring the content is compressed, time to the first byte, optimize images, ensuring reasonable DOM size, SSL and many others. It's advisable to have these rich monitors both during development, as part of the CI and most important - 24x7 over the production's servers/CDN @@ -1268,13 +1200,11 @@ test('movie title appears', async () => { ![](/assets/lighthouse2.png "Lighthouse page load inspection report") -
-
-## ⚪ ️ 3.6 Stub flakky and slow resources like backend APIs +## ⚪ ️ 3.6 Stub flakey and slow resources like backend APIs :white_check_mark: **Do:** When coding your mainstream tests (not E2E tests), avoid involving any resource that is beyond your responsibility and control like backend API and use stubs instead (i.e. test double). Practically, instead of real network calls to APIs, use some test double library (like [Sinon](https://sinonjs.org/), [Test doubles](https://www.npmjs.com/package/testdouble), etc) for stubbing the API response. The main benefit is preventing flakiness - testing or staging APIs by definition are not highly stable and from time to time will fail your tests although YOUR component behaves just fine (production env was not meant for testing and it usually throttles requests). Doing this will allow simulating various API behavior that should drive your component behavior as when no data was found or the case when API throws an error. Last but not least, network calls will greatly slow down the tests @@ -1282,7 +1212,6 @@ test('movie title appears', async () => { ❌ **Otherwise:** The average test runs no longer than few ms, a typical API call last 100ms>, this makes each test ~20x slower -
✏ Code Examples @@ -1290,14 +1219,14 @@ test('movie title appears', async () => {
### :clap: Doing It Right Example: Stubbing or intercepting API calls -![](https://img.shields.io/badge/🔧%20Example%20using%20React-blue.svg - "Examples with React") ![](https://img.shields.io/badge/🔧%20Example%20using%20Jest-blue.svg +![Examples with React](https://img.shields.io/badge/🔧%20Example%20using%20React-blue.svg + "Examples with React") ![Examples with react-testing-library](https://img.shields.io/badge/🔧%20Example%20using%20Jest-blue.svg "Examples with react-testing-library") - + ```javascript // unit under test -export default function ProductsList() { +export default function ProductsList() { const [products, setProducts] = useState(false) const fetchProducts = async() => { @@ -1338,13 +1267,13 @@ test('When no products exist, show the appropriate message', () => {
-❌ **Otherwise:** UI might invest much in testing its functionality only to realizes very late that the backend returned payload (the data schema the UI has to work with) is very differnt than expected +❌ **Otherwise:** UI might invest much in testing its functionality only to realizes very late that the backend returned payload (the data schema the UI has to work with) is very different than expected
## ⚪ ️ 3.8 Speed-up E2E tests by reusing login credentials -:white_check_mark: **Do:** In E2E tests that involve a real backend and rely on a valid user token for API calls, it doesn't payoff to isolate the test to a level where a user is created and logged-in in every request. Instead, login only once before the tests execution start (i.e. before-all hook), save the token in some local storage and reuse it across requests. This seem to violate one of the core testing principle - keep the test autonomous without resources coupling. While this is a valid worry, in E2E tests performance is a key concern and creating 1-3 API requests before starting each individial tests might lead to horrible execution time. Reusing credentials doesn't mean the tests have to act on the same user records - if relying on user records (e.g. test user payments history) than make sure to generate those records as part of the test and avoid sharing their existence with other tests. Also remember that the backend can be faked - if your tests are focused on the frontend it might be better to isolate it and stub the backend API (see bullet 3.6). +:white_check_mark: **Do:** In E2E tests that involve a real backend and rely on a valid user token for API calls, it doesn't payoff to isolate the test to a level where a user is created and logged-in in every request. Instead, login only once before the tests execution start (i.e. before-all hook), save the token in some local storage and reuse it across requests. This seem to violate one of the core testing principle - keep the test autonomous without resources coupling. While this is a valid worry, in E2E tests performance is a key concern and creating 1-3 API requests before starting each individual tests might lead to horrible execution time. Reusing credentials doesn't mean the tests have to act on the same user records - if relying on user records (e.g. test user payments history) than make sure to generate those records as part of the test and avoid sharing their existence with other tests. Also remember that the backend can be faked - if your tests are focused on the frontend it might be better to isolate it and stub the backend API (see bullet 3.6).
@@ -1358,7 +1287,7 @@ test('When no products exist, show the appropriate message', () => { ### :clap: Doing It Right Example: Logging-in before-all and not before-each -![](https://img.shields.io/badge/🔨%20Example%20using%20Cypress-blue.svg +![Using Cypress to illustrate the idea](https://img.shields.io/badge/🔨%20Example%20using%20Cypress-blue.svg "Using Cypress to illustrate the idea") ```javascript @@ -1389,9 +1318,6 @@ beforeEach(setUser => () {
- - -
## ⚪ ️ 3.9 Have one E2E smoke test that just travels across the site map @@ -1402,7 +1328,6 @@ beforeEach(setUser => () { ❌ **Otherwise:** Everything might seem perfect, all tests pass, production health-check is also positive but the Payment component had some packaging issue and only the /Payment route is not rendering -
✏ Code Examples @@ -1410,8 +1335,9 @@ beforeEach(setUser => () {
### :clap: Doing It Right Example: Smoke travelling across all pages -![](https://img.shields.io/badge/🔨%20Example%20using%20Cypress-blue.svg +![Using Cypress to illustrate the idea](https://img.shields.io/badge/🔨%20Example%20using%20Cypress-blue.svg "Using Cypress to illustrate the idea") + ```javascript it('When doing smoke testing over all page, should load them all successfully', () => { // exemplified using Cypress but can be implemented easily @@ -1427,7 +1353,6 @@ it('When doing smoke testing over all page, should load them all successfully',
-
## ⚪ ️ 3.10 Expose the tests as a live collaborative document @@ -1436,7 +1361,6 @@ it('When doing smoke testing over all page, should load them all successfully', ❌ **Otherwise:** After investing top resources on testing, it's just a pity not to leverage this investment and win great value -
✏ Code Examples @@ -1445,12 +1369,13 @@ it('When doing smoke testing over all page, should load them all successfully', ### :clap: Doing It Right Example: Describing tests in human-language using cucumber-js -![](https://img.shields.io/badge/🔨%20Example%20using%20Cocumber-blue.svg "Examples using Cucumber") +![Examples using Cucumber](https://img.shields.io/badge/🔨%20Example%20using%20Cocumber-blue.svg "Examples using Cucumber") + ```javascript // this is how one can describe tests using cucumber: plain language that allows anyone to understand and collaborate Feature: Twitter new tweet - + I want to tweet something in Twitter @focus @@ -1460,28 +1385,22 @@ Feature: Twitter new tweet Given I type "Hello followers!" in the textbox Given I click on "Submit" button Then I see message "Tweet saved" - ``` ### :clap: Doing It Right Example: Visualizing our components, their various states and inputs using Storybook -![](https://img.shields.io/badge/🔨%20Example%20using%20StoryBook-blue.svg "Using StoryBook") +![Using StoryBook](https://img.shields.io/badge/🔨%20Example%20using%20StoryBook-blue.svg "Using StoryBook")
- - - ## ⚪ ️ 3.11 Detect visual issues with automated tools - -:white_check_mark: **Do:** Setup automated tools to capture UI screenshots when changes are presented and detect visual issues like content overlapping or breaking. This ensures that not only the right data is prepared but also the user can conveniently see it. This technique is not widely adopted, our testing mindset leans toward functional tests but it's the visuals what the user experience and with so many device types it's very easy to overlook some nasty UI bug. Some free tools can provide the basics - generate and save screenshots for the inspection of human eyes. While this approach might be sufficient for small apps, it's flawed as any other manual testing that demands human labor anytime something changes. On the other hand, it's quite challenging to detect UI issues automatically due to the lack of clear definition - this is where the field of 'Visual Regression' chime in and solve this puzzle by comparing old UI with the latest changes and detect differences. Some OSS/free tools can provide some of this functionality (e.g. [wraith](https://github.com/BBC-News/wraith), [PhantomCSS]([https://github.com/HuddleEng/PhantomCSS](https://github.com/HuddleEng/PhantomCSS)) but might charge signficant setup time. The commercial line of tools (e.g. [Applitools](https://applitools.com/), [Percy.io](https://percy.io/)) takes is a step further by smoothing the installation and packing advanced features like management UI, alerting, smart capturing by elemeinating 'visual noise' (e.g. ads, animations) and even root cause analysis of the DOM/css changes that led to the issue +:white_check_mark: **Do:** Setup automated tools to capture UI screenshots when changes are presented and detect visual issues like content overlapping or breaking. This ensures that not only the right data is prepared but also the user can conveniently see it. This technique is not widely adopted, our testing mindset leans toward functional tests but it's the visuals what the user experience and with so many device types it's very easy to overlook some nasty UI bug. Some free tools can provide the basics - generate and save screenshots for the inspection of human eyes. While this approach might be sufficient for small apps, it's flawed as any other manual testing that demands human labor anytime something changes. On the other hand, it's quite challenging to detect UI issues automatically due to the lack of clear definition - this is where the field of 'Visual Regression' chime in and solve this puzzle by comparing old UI with the latest changes and detect differences. Some OSS/free tools can provide some of this functionality (e.g. [wraith](https://github.com/BBC-News/wraith), [PhantomCSS]([https://github.com/HuddleEng/PhantomCSS](https://github.com/HuddleEng/PhantomCSS)) but might charge significant setup time. The commercial line of tools (e.g. [Applitools](https://applitools.com/), [Percy.io](https://percy.io/)) takes is a step further by smoothing the installation and packing advanced features like management UI, alerting, smart capturing by eliminating 'visual noise' (e.g. ads, animations) and even root cause analysis of the DOM/css changes that led to the issue
❌ **Otherwise:** How good is a content page that display great content (100% tests passed), loads instantly but half of the content area is hidden? -
✏ Code Examples @@ -1494,13 +1413,13 @@ Feature: Twitter new tweet
- ### :clap: Doing It Right Example: Configuring wraith to capture and compare UI snapshots -![](https://img.shields.io/badge/🔨%20Example%20using%20Wraith-blue.svg +![Using Cypress to illustrate the idea](https://img.shields.io/badge/🔨%20Example%20using%20Wraith-blue.svg "Using Cypress to illustrate the idea") -``` +```yml + ​# Add as many domains as necessary. Key will act as a label​ domains: @@ -1526,10 +1445,10 @@ paths: path: /subscribe ``` -### :clap: Doing It Right Example: Using Applitools to get snapshot comaprison and other advanced features +### :clap: Doing It Right Example: Using Applitools to get snapshot comparison and other advanced features -![](https://img.shields.io/badge/🔨%20Example%20using%20AppliTools-blue.svg - "Using Cypress to illustrate the idea") ![](https://img.shields.io/badge/🔨%20Example%20using%20Cypress-blue.svg +![Using Cypress to illustrate the idea](https://img.shields.io/badge/🔨%20Example%20using%20AppliTools-blue.svg + "Using Cypress to illustrate the idea") ![Using Cypress to illustrate the idea](https://img.shields.io/badge/🔨%20Example%20using%20Cypress-blue.svg "Using Cypress to illustrate the idea") ```javascript @@ -1574,16 +1493,10 @@ cy.eyesCheckWindow('mark as completed'); }); ``` - - -
- -

- # Section 4️⃣: Measuring Test Effectiveness

@@ -1596,10 +1509,8 @@ Implementation tips: You may want to configure your continuous integration (CI)
- ❌ **Otherwise:** Confidence and numbers go hand in hand, without really knowing that you tested most of the system — there will also be some fear. and fear will slow you down -
✏ Code Examples @@ -1607,21 +1518,20 @@ Implementation tips: You may want to configure your continuous integration (CI)
### :clap: Example: A typical coverage report + ![alt text](assets/bp-18-yoni-goldberg-code-coverage.png "A typical coverage report")
### :clap: Doing It Right Example: Setting up coverage per component (using Jest) -![](https://img.shields.io/badge/🔨%20Example%20using%20Jest-blue.svg +![Using Cypress to illustrate the idea](https://img.shields.io/badge/🔨%20Example%20using%20Jest-blue.svg "Using Cypress to illustrate the idea") ![alt text](assets/bp-18-code-coverage2.jpeg "Setting up coverage per component (using Jest)
- -

## ⚪ ️ 4.2 Inspect coverage reports to detect untested areas and other oddities @@ -1629,10 +1539,8 @@ Implementation tips: You may want to configure your continuous integration (CI) :white_check_mark: **Do:** Some issues sneak just under the radar and are really hard to find using traditional tools. These are not really bugs but more of surprising application behavior that might have a severe impact. For example, often some code areas are never or rarely being invoked — you thought that the ‘PricingCalculator’ class is always setting the product price but it turns out it is actually never invoked although we have 10000 products in DB and many sales… Code coverage reports help you realize whether the application behaves the way you believe it does. Other than that, it can also highlight which types of code is not tested — being informed that 80% of the code is tested doesn’t tell whether the critical parts are covered. Generating reports is easy — just run your app in production or during testing with coverage tracking and then see colorful reports that highlight how frequent each code area is invoked. If you take your time to glimpse into this data — you might find some gotchas
- ❌ **Otherwise:** If you don’t know which parts of your code are left un-tested, you don’t know where the issues might come from -
✏ Code Examples @@ -1645,7 +1553,6 @@ Implementation tips: You may want to configure your continuous integration (CI)
-

## ⚪ ️ 4.3 Measure logical coverage using mutation testing @@ -1672,7 +1579,7 @@ Knowing that all or most of the mutations were killed gives much higher confiden ### :thumbsdown: Anti Pattern Example: 100% coverage, 0% testing -![](https://img.shields.io/badge/🔨%20Example%20using%20Stryker-blue.svg +![Using Cypress to illustrate the idea](https://img.shields.io/badge/🔨%20Example%20using%20Stryker-blue.svg "Using Cypress to illustrate the idea") ```javascript function addNewOrder(newOrder) { @@ -1684,10 +1591,11 @@ function addNewOrder(newOrder) { } it("Test addNewOrder, don't use such test names", () => { - addNewOrder({asignee: "John@mailer.com",price: 120}); + addNewOrder({assignee: "John@mailer.com",price: 120}); });//Triggers 100% code coverage, but it doesn't check anything ``` +
### :clap: Doing It Right Example: Stryker reports, a tool for mutation testing, detects and counts the amount of code that is not tested (Mutations) @@ -1696,8 +1604,6 @@ it("Test addNewOrder, don't use such test names", () => {
- -

## ⚪ ️4.4 Preventing test code issues with Test linters @@ -1706,10 +1612,8 @@ it("Test addNewOrder, don't use such test names", () => {
- ❌ **Otherwise:** Seeing 90% code coverage and 100% green tests will make your face wear a big smile only until you realize that many tests aren’t asserting for anything and many test suites were just skipped. Hopefully, you didn’t deploy anything based on this false observation -
✏ Code Examples @@ -1735,7 +1639,6 @@ it("Test name", () => {*//error:no-identical-title. Assign unique titles to test

- # Section 5️⃣: CI and Other Quality Measures

@@ -1745,10 +1648,8 @@ it("Test name", () => {*//error:no-identical-title. Assign unique titles to test :white_check_mark: **Do:** Linters are a free lunch, with 5 min setup you get for free an auto-pilot guarding your code and catching significant issue as you type. Gone are the days where linting was about cosmetics (no semi-colons!). Nowadays, Linters can catch severe issues like errors that are not thrown correctly and losing information. On top of your basic set of rules (like [ESLint standard](https://www.npmjs.com/package/eslint-plugin-standard) or [Airbnb style](https://www.npmjs.com/package/eslint-config-airbnb)), consider including some specializing Linters like [eslint-plugin-chai-expect](https://www.npmjs.com/package/eslint-plugin-chai-expect) that can discover tests without assertions, [eslint-plugin-promise](https://www.npmjs.com/package/eslint-plugin-promise?activeTab=readme) can discover promises with no resolve (your code will never continue), [eslint-plugin-security](https://www.npmjs.com/package/eslint-plugin-security?activeTab=readme) which can discover eager regex expressions that might get used for DOS attacks, and [eslint-plugin-you-dont-need-lodash-underscore](https://www.npmjs.com/package/eslint-plugin-you-dont-need-lodash-underscore) is capable of alarming when the code uses utility library methods that are part of the V8 core methods like Lodash._map(…)
- ❌ **Otherwise:** Consider a rainy day where your production keeps crashing but the logs don’t display the error stack trace. What happened? Your code mistakenly threw a non-error object and the stack trace was lost, a good reason for banging your head against a brick wall. A 5min linter setup could detect this TYPO and save your day -
✏ Code Examples @@ -1756,13 +1657,11 @@ it("Test name", () => {*//error:no-identical-title. Assign unique titles to test
### :thumbsdown: Anti Pattern Example: The wrong Error object is thrown mistakenly, no stack-trace will appear for this error. Luckily, ESLint catches the next production bug + ![alt text](assets/bp-21-yoni-goldberg-eslint.jpeg "The wrong Error object is thrown mistakenly, no stack-trace will appear for this error. Luckily, ESLint catches the next production bug")
- - -

# ⚪ ️ 5.2 Shorten the feedback loop with local developer-CI @@ -1772,10 +1671,8 @@ it("Test name", () => {*//error:no-identical-title. Assign unique titles to test Practically, some CI vendors (Example: [CircleCI load CLI](https://circleci.com/docs/2.0/local-cli/)) allow running the pipeline locally. Some commercial tools like [wallaby provide highly-valuable & testing insights](https://wallabyjs.com/) as a developer prototype (no affiliation). Alternatively, you may just add npm script to package.json that runs all the quality commands (e.g. test, lint, vulnerabilities) — use tools like [concurrently](https://www.npmjs.com/package/concurrently) for parallelization and non-zero exit code if one of the tools failed. Now the developer should just invoke one command — e.g. ‘npm run quality’ — to get instant feedback. Consider also aborting a commit if the quality check failed using a githook ([husky can help](https://github.com/typicode/husky))
- ❌ **Otherwise:** When the quality results arrive the day after the code, testing doesn’t become a fluent part of development rather an after the fact formal artifact -
✏ Code Examples @@ -1783,6 +1680,7 @@ Practically, some CI vendors (Example: [CircleCI load CLI](https://circleci.com/
### :clap: Doing It Right Example: npm scripts that perform code quality inspection, all are run in parallel on demand or when a developer is trying to push new code + ```javascript "scripts": { "inspect:sanity-testing": "mocha **/**--test.js --grep \"sanity\"", @@ -1804,9 +1702,6 @@ Practically, some CI vendors (Example: [CircleCI load CLI](https://circleci.com/
- - -

# ⚪ ️5.3 Perform e2e testing over a true production-mirror @@ -1816,10 +1711,8 @@ Practically, some CI vendors (Example: [CircleCI load CLI](https://circleci.com/ The huge Kubernetes eco-system is yet to formalize a standard convenient tool for local and CI-mirroring though many new tools are launched frequently. One approach is running a ‘minimized-Kubernetes’ using tools like [Minikube](https://kubernetes.io/docs/setup/minikube/) and [MicroK8s](https://microk8s.io/) which resemble the real thing only come with less overhead. Another approach is testing over a remote ‘real-Kubernetes’, some CI providers (e.g. [Codefresh](https://codefresh.io/)) has native integration with Kubernetes environment and make it easy to run the CI pipeline over the real thing, others allow custom scripting against a remote Kubernetes.
- ❌ **Otherwise:** Using different technologies for production and testing demands maintaining two deployment models and keeps the developers and the ops team separated -
✏ Code Examples @@ -1832,19 +1725,14 @@ The huge Kubernetes eco-system is yet to formalize a standard convenient tool fo
- - - -

## ⚪ ️5.4 Parallelize test execution -:white_check_mark: **Do:** When done right, testing is your 24/7 friend providing almost instant feedback. In practice, executing 500 CPU-bounded unit test on a single thread can take too long. Luckily, modern test runners and CI platforms (like [Jest](https://github.com/facebook/jest), [AVA](https://github.com/avajs/ava) and [Mocha extensions](https://github.com/yandex/mocha-parallel-tests)) can parallelize the test into multiple processes and achieve significant improvement in feedback time. Some CI vendors do also parallelize tests across containers (!) which shortens the feedback loop even further. Whether locally over multiple processes, or over some cloud CLI using multiple machines — parallelizing demand keeping the tests autonomous as each might run on different processes +:white_check_mark: **Do:** When done right, testing is your 24/7 friend providing almost instant feedback. In practice, executing 500 CPU-bounded unit test on a single thread can take too long. Luckily, modern test runners and CI platforms (like [Jest](https://github.com/facebook/jest), [AVA](https://github.com/avajs/ava) and [Mocha extensions](https://github.com/yandex/mocha-parallel-tests)) can parallelize the test into multiple processes and achieve significant improvement in feedback time. Some CI vendors do also parallelize tests across containers (!) which shortens the feedback loop even further. Whether locally over multiple processes, or over some cloud CLI using multiple machines — parallelizing demand keeping the tests autonomous as each might run on different processes ❌ **Otherwise:** Getting test results 1 hour long after pushing new code, as you already code the next features, is a great recipe for making testing less relevant -
✏ Code Examples @@ -1852,21 +1740,19 @@ The huge Kubernetes eco-system is yet to formalize a standard convenient tool fo
### :clap: Doing It Right Example: Mocha parallel & Jest easily outrun the traditional Mocha thanks to testing parallelization ([Credit: JavaScript Test-Runners Benchmark](https://medium.com/dailyjs/javascript-test-runners-benchmark-3a78d4117b4)) + ![alt text](assets/bp-24-yonigoldberg-jest-parallel.png "Mocha parallel & Jest easily outrun the traditional Mocha thanks to testing parallelization (Credit: JavaScript Test-Runners Benchmark)")
- - -

## ⚪ ️5.5 Stay away from legal issues using license and plagiarism check + :white_check_mark: **Do:** Licensing and plagiarism issues are probably not your main concern right now, but why not tick this box as well in 10 minutes? A bunch of npm packages like [license check](https://www.npmjs.com/package/license-checker) and [plagiarism check](https://www.npmjs.com/package/plagiarism-checker) (commercial with free plan) can be easily baked into your CI pipeline and inspect for sorrows like dependencies with restrictive licenses or code that was copy-pasted from Stackoverflow and apparently violates some copyrights ❌ **Otherwise:** Unintentionally, developers might use packages with inappropriate licenses or copy paste commercial code and run into legal issues -
✏ Code Examples @@ -1874,6 +1760,7 @@ The huge Kubernetes eco-system is yet to formalize a standard convenient tool fo
### :clap: Doing It Right Example: + ```javascript //install license-checker in your CI environment or also locally npm install -g license-checker @@ -1887,21 +1774,17 @@ license-checker --summary --failOn BSD ![alt text](assets/bp-25-nodejs-licsense.png) -
- -

## ⚪ ️5.6 Constantly inspect for vulnerable dependencies + :white_check_mark: **Do:** Licensing and plagiarism issues are probably not your main concern right now, but why not tick this box as well in 10 minutes? A bunch of npm packages like license check and plagiarism check (commercial with free plan) can be easily baked into your CI pipeline and inspect for sorrows like dependencies with restrictive licenses or code that was copy-pasted from Stackoverflow and apparently violates some copyrights
- ❌ **Otherwise:** Even the most reputable dependencies such as Express have known vulnerabilities. This can get easily tamed using community tools such as [npm audit](https://docs.npmjs.com/getting-started/running-a-security-audit), or commercial tools like [snyk](https://snyk.io/) (offer also a free community version). Both can be invoked from your CI on every build -
✏ Code Examples @@ -1909,16 +1792,15 @@ license-checker --summary --failOn BSD
### :clap: Example: NPM Audit result + ![alt text](assets/bp-26-npm-audit-snyk.png "NPM Audit result")
- - -

## ⚪ ️5.7 Automate dependency updates + :white_check_mark: **Do:** Yarn and npm latest introduction of package-lock.json introduced a serious challenge (the road to hell is paved with good intentions) — by default now, packages are no longer getting updates. Even a team running many fresh deployments with ‘npm install’ & ‘npm update’ won’t get any new updates. This leads to subpar dependent packages versions at best or to vulnerable code at worst. Teams now rely on developers goodwill and memory to manually update the package.json or use tools [like ncu](https://www.npmjs.com/package/npm-check-updates) manually. A more reliable way could be to automate the process of getting the most reliable dependency versions, though there are no silver bullet solutions yet there are two possible automation roads: (1) CI can fail builds that have obsolete dependencies — using tools like [‘npm outdated’](https://docs.npmjs.com/cli/outdated) or ‘npm-check-updates (ncu)’ . Doing so will enforce developers to update dependencies. @@ -1928,10 +1810,8 @@ license-checker --summary --failOn BSD An efficient update policy may allow some ‘vesting period’ — let the code lag behind the @latest for some time and versions before considering the local copy as obsolete (e.g. local version is 1.3.1 and repository version is 1.3.8)
- ❌ **Otherwise:** Your production will run packages that have been explicitly tagged by their author as risky -
✏ Code Examples @@ -1939,33 +1819,31 @@ An efficient update policy may allow some ‘vesting period’ — let the c
### :clap: Example: [ncu](https://www.npmjs.com/package/npm-check-updates) can be used manually or within a CI pipeline to detect to which extent the code lag behind the latest versions -![alt text](assets/bp-27-yoni-goldberg-npm.png "Nncu can be used manually or within a CI pipeline to detect to which extent the code lag behind the latest versions") +![alt text](assets/bp-27-yoni-goldberg-npm.png "ncu can be used manually or within a CI pipeline to detect to which extent the code lag behind the latest versions")
-

## ⚪ ️ 5.8 Other, non-Node related, CI tips + :white_check_mark: **Do:** This post is focused on testing advice that is related to, or at least can be exemplified with Node JS. This bullet, however, groups few non-Node related tips that are well-known
  1. Use a declarative syntax. This is the only option for most vendors but older versions of Jenkins allows using code or UI
  2. Opt for a vendor that has native Docker support
  3. Fail early, run your fastest tests first. Create a ‘Smoke testing’ step/milestone that groups multiple fast inspections (e.g. linting, unit tests) and provide snappy feedback to the code committer
  4. Make it easy to skim-through all build artifacts including test reports, coverage reports, mutation reports, logs, etc
  5. Create multiple pipelines/jobs for each event, reuse steps between them. For example, configure a job for feature branch commits and a different one for master PR. Let each reuse logic using shared steps (most vendors provide some mechanism for code reuse
  6. Never embed secrets in a job declaration, grab them from a secret store or from the job’s configuration
  7. Explicitly bump version in a release build or at least ensure the developer did so
  8. Build only once and perform all the inspections over the single build artifact (e.g. Docker image)
  9. Test in an ephemeral environment that doesn’t drift state between builds. Caching node_modules might be the only exception

- ❌ **Otherwise:** You‘ll miss years of wisdom

## ⚪ ️ 5.9 Build matrix: Run the same CI steps using multiple Node versions + :white_check_mark: **Do:** Quality checking is about serendipity, the more ground you cover the luckier you get in detecting issues early. When developing reusable packages or running a multi-customer production with various configuration and Node versions, the CI must run the pipeline of tests over all the permutations of configurations. For example, assuming we use MySQL for some customers and Postgres for others — some CI vendors support a feature called ‘Matrix’ which allow running the suit of testing against all permutations of MySQL, Postgres and multiple Node version like 8, 9 and 10. This is done using configuration only without any additional effort (assuming you have testing or any other quality checks). Other CIs who doesn’t support Matrix might have extensions or tweaks to allow that
- ❌ **Otherwise:** So after doing all that hard work of writing testing are we going to let bugs sneak in only because of configuration issues? -
✏ Code Examples @@ -1980,8 +1858,6 @@ An efficient update policy may allow some ‘vesting period’ — let the c # Team - - ## Yoni Goldberg
@@ -2007,12 +1883,11 @@ An efficient update policy may allow some ‘vesting period’ — let the c

- ## [Bruno Scheufler](https://github.com/BrunoScheufler) **Role:** Tech reviewer and advisor -Took care to revise, improve, lint and polish all the texts +Took care to revise, improve, lint and polish all the texts **About:** full-stack web engineer, Node.js & GraphQL enthusiast
@@ -2022,4 +1897,4 @@ Took care to revise, improve, lint and polish all the texts **Role:** Concept, design and great advice -**About:** A savvy frontend developer, CSS expert and emojis freak \ No newline at end of file +**About:** A savvy frontend developer, CSS expert and emojis freak From a355bdc7b32c667252ae1fae114036e60ff40012 Mon Sep 17 00:00:00 2001 From: Kyle Martin Date: Sat, 24 Aug 2019 17:49:33 +1200 Subject: [PATCH 2/2] fixups --- readme.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/readme.md b/readme.md index 7fc9da76..2613f795 100644 --- a/readme.md +++ b/readme.md @@ -159,7 +159,7 @@ describe('Products Service', function() { ![Jest example](https://img.shields.io/badge/🔧%20Example%20using%20Jest-blue.svg "Examples with Jest") ![Mocha example](https://img.shields.io/badge/🔧%20Example%20using%20Mocha-blue.svg - "Examples with Moch") + "Examples with Mocha") ```javascript describe('Customer classifier', () => { @@ -218,7 +218,7 @@ test("When asking for an admin, ensure only ordered admins in results" , () => { //assuming we've added here two admins "admin1", "admin2" and "user1" const allAdmins = getUsers({adminOnly:true}); - const admin1Found, adming2Found = false; + const admin1Found, admin2Found = false; allAdmins.forEach(aSingleUser => { if(aSingleUser === "user1"){ @@ -372,7 +372,7 @@ it("When a valid product is about to be deleted, ensure an email is sent", async ```javascript const addProduct = (name, price) =>{ - const productNameRegexNoSpace = /^\S*$/;//no white-space allowd + const productNameRegexNoSpace = /^\S*$/;//no white-space allowed if(!productNameRegexNoSpace.test(name)) return false;//this path never reached due to dull input @@ -470,7 +470,7 @@ It’s worth noting that there are few cases where long & external snapshots are "Examples with Jest") ```javascript -it('TestJavaScript.com is renderd correctly', () => { +it('TestJavaScript.com is rendered correctly', () => { //Arrange @@ -1051,7 +1051,7 @@ With all that said, a word of caution is in order: this technique works for smal
-### :clap: Doing It Right Example: Working realstically with a fully rendered component +### :clap: Doing It Right Example: Working realistically with a fully rendered component ![Examples with React](https://img.shields.io/badge/🔧%20Example%20using%20React-blue.svg "Examples with React") ![Examples with Enzyme](https://img.shields.io/badge/🔧%20Example%20using%20Enzyme-blue.svg @@ -1084,7 +1084,6 @@ test('Realistic approach: When clicked to show filters, filters are displayed', // This is how the user will approach this element: by text }) - ``` ### :thumbsdown: Anti-Pattern Example: Mocking the reality with shallow rendering