Skip to content

update gherkin etc, validate against cck #1318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 134 commits into from
Aug 24, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
134 commits
Select commit Hold shift + click to select a range
7062f4e
add cck
davidjgoss Apr 30, 2020
bca4d7d
add a script to run cck tests
davidjgoss May 7, 2020
9e66fe6
direct message formatter output to file
davidjgoss May 7, 2020
f97930e
test via mocha with one run per feature
davidjgoss May 8, 2020
f239f9f
add compatibility dir to eslint
davidjgoss May 8, 2020
7510313
step defs for attachments suite
davidjgoss May 8, 2020
2945b96
add World and IWorldOptions to public API
davidjgoss May 8, 2020
38149e1
add data-tables steps
davidjgoss May 8, 2020
7f746ae
exclude uris from the cck diff
davidjgoss May 8, 2020
5591c70
implement DataTable::transpose
davidjgoss May 8, 2020
aad5595
implement steps for remaining cck bits
davidjgoss May 9, 2020
9948792
add dependency-lint setup for new cck stuff
davidjgoss May 9, 2020
29145d1
factor out ids for now
davidjgoss May 9, 2020
8679afe
add pre script for cck
davidjgoss May 9, 2020
ff69321
merge master
davidjgoss May 10, 2020
4e2f1ee
update gherkin and messages deps
davidjgoss May 10, 2020
7cd14db
add dep on cucumber query
davidjgoss May 12, 2020
e2db9bf
allow json modules to be imported
davidjgoss May 12, 2020
dda37cc
first big wave of changes to get things compiling/running
davidjgoss May 10, 2020
6b9db47
stop emitting pickleAccepted and pickleRejected messages
davidjgoss May 17, 2020
3dac234
implement lookup of worst result in event data collector
davidjgoss May 17, 2020
340ca4d
fix some of pickle runner spec
davidjgoss May 17, 2020
1ea2ac9
mostly fix up progress_bar_formatter
davidjgoss May 17, 2020
d2cc938
remove superfluous gitignore
davidjgoss May 17, 2020
c79c365
hook up cck profile so it doesnt pick up feature-test support code
davidjgoss May 23, 2020
fac9203
emit messages for step definitions
davidjgoss May 23, 2020
ebe7795
stop ignoring ids so we can see what doesn't line up
davidjgoss May 23, 2020
2cd89be
import cck stuff from monorepo for now
davidjgoss May 23, 2020
1eda297
emi meta message
davidjgoss May 24, 2020
09d0f08
refactor `emitSupportCodeMessages` to take whole library
davidjgoss May 30, 2020
540a9f9
emit before/after hooks
davidjgoss May 30, 2020
75cf679
emit step definitions before hooks
davidjgoss May 30, 2020
dd04c06
emit beforeAll and afterAll hooks too
davidjgoss May 30, 2020
25569e0
emit a zero timestamp within testRunStarted
davidjgoss May 30, 2020
fffe791
add `durations` package
davidjgoss Jun 14, 2020
6779fe2
start hooking up a stopwatch for test run timestamps
davidjgoss Jun 14, 2020
3dc0726
emit timestamps from pickle runner
davidjgoss Jun 14, 2020
5bb68cb
start to sort out attachments cck
davidjgoss Jun 14, 2020
dd0d6e4
add `log` function that attaches as cucumber log
davidjgoss Jun 14, 2020
9942d76
fix some unit test issues
davidjgoss Jun 14, 2020
ea23311
emit correct message structure for attachments
davidjgoss Jun 16, 2020
46e9c23
dont exclude body any more
davidjgoss Jun 16, 2020
970da95
merge master
davidjgoss Jul 7, 2020
cc1b57e
formatting and lint
davidjgoss Jul 7, 2020
6c22da7
fix config builder test
davidjgoss Jul 8, 2020
40d3ab9
fix helpers spec
davidjgoss Jul 8, 2020
99dc049
fix tests around attachments
davidjgoss Jul 8, 2020
68aa293
fix what pickle runner spec is testing (still fails tho)
davidjgoss Jul 9, 2020
4ce14b5
fix stopwatch in parallel runtime
davidjgoss Jul 10, 2020
07cc622
fix coordinator maybe
davidjgoss Jul 11, 2020
a775c07
fix wrong structure in test
davidjgoss Jul 11, 2020
7713cae
fix the stopwatch
davidjgoss Jul 12, 2020
3e8ee19
clean up, remove comment
davidjgoss Jul 12, 2020
9773e62
add willBeRetried to testStepFinished result, dont try to emit result…
davidjgoss Jul 12, 2020
a8acb5b
add a todo comment because this is bad
davidjgoss Jul 12, 2020
f8a686f
fix pickle runner test
davidjgoss Jul 12, 2020
a43f23f
remove todo comments that are done
davidjgoss Jul 12, 2020
5322922
merge master
davidjgoss Jul 18, 2020
b918f3b
if you say so yarn
davidjgoss Jul 18, 2020
a1f026c
use a map to make attachment steps clearer
davidjgoss Jul 18, 2020
cd6492d
split out a function to get timestamps for assertions on messages
davidjgoss Jul 18, 2020
327cea6
remove ref to pickleAccepted from test
davidjgoss Jul 18, 2020
0a9a02a
filter off meta messages from formatter feature tests
davidjgoss Jul 18, 2020
d230704
update rejected pickle message fixture
davidjgoss Jul 18, 2020
0df7b4f
update passed scenario message fixture
davidjgoss Jul 18, 2020
ab28122
update passed rule message fixture
davidjgoss Jul 18, 2020
4e71a3d
update failed message fixture
davidjgoss Jul 18, 2020
841000d
update retried message fixture
davidjgoss Jul 18, 2020
638499f
fix wrong path in fixtures
davidjgoss Jul 30, 2020
df5e703
emit messages for parameter types
davidjgoss Jul 31, 2020
fadc831
fix json formatter handling of attachments and add a unit test
davidjgoss Aug 2, 2020
995beae
merge master
davidjgoss Aug 2, 2020
50e73f6
use latest expressions packages, fix a couple of usages
davidjgoss Aug 2, 2020
6c02576
update all the monorepo deps
davidjgoss Aug 2, 2020
e215d8d
fix some test code re protbuf and messages
davidjgoss Aug 2, 2020
1c08d8f
bring attachments cck up to date
davidjgoss Aug 2, 2020
6535fd5
update cck fixture for datatable
davidjgoss Aug 2, 2020
25ec4bc
update cck fixture for hooks
davidjgoss Aug 2, 2020
ff94a72
update other cck fixtures
davidjgoss Aug 2, 2020
8e6a9f3
add new cck specs
davidjgoss Aug 2, 2020
d3e0b4e
first pass at supporting stepMatchArgumentLists
davidjgoss Aug 2, 2020
6760bb7
handle children in arg group
davidjgoss Aug 2, 2020
5408611
fix existing pickle runner tests
davidjgoss Aug 2, 2020
eeca00c
add test for stepMatchArgumentLists
davidjgoss Aug 2, 2020
f3bd88b
update fixtures
davidjgoss Aug 2, 2020
af242fa
use new library to emit meta
davidjgoss Aug 2, 2020
2f02b12
remove success bool from testRunFinished message
davidjgoss Aug 2, 2020
e20e81b
Merge branch 'master' into cck
davidjgoss Aug 8, 2020
de48fc3
fix formatter output helper for renamed step result structure, amend …
davidjgoss Aug 10, 2020
d15bb32
fix handling of gherkin parse error to line up with its changes
davidjgoss Aug 10, 2020
583f4ee
update monorepo deps again
davidjgoss Aug 10, 2020
427a23c
add downlevelIteration to make browser example work
davidjgoss Aug 10, 2020
8f1eac7
docs and change log
davidjgoss Aug 10, 2020
a31ac17
normalize uris in support code messages
davidjgoss Aug 10, 2020
6d1e92a
also output messages from feature test, for conveninent debugging
davidjgoss Aug 13, 2020
d1a34df
bump gherkin again
davidjgoss Aug 13, 2020
45391bc
bump cck dep
davidjgoss Aug 13, 2020
5abf80b
remove local versions of features and fixtures
davidjgoss Aug 13, 2020
1879b29
run against features and fixtures in cck dep
davidjgoss Aug 13, 2020
bbd3ccc
emit undefined params
davidjgoss Aug 13, 2020
db8baa0
whoops
davidjgoss Aug 13, 2020
44a552c
some cck starting to pass by ignoring ids and time
davidjgoss Aug 13, 2020
faa27df
fix testCase ordering in the test
davidjgoss Aug 13, 2020
c016cb6
undefined/ambiguous/skipped test steps emit with a zero duration
davidjgoss Aug 14, 2020
3bcba31
remove temp exclusion
davidjgoss Aug 14, 2020
885fdc5
dont compare error messages yet - need to figure out how to make this…
davidjgoss Aug 14, 2020
0246210
emit after/afterall in original declaration order
davidjgoss Aug 14, 2020
276aa86
add cck tests to main test script
davidjgoss Aug 15, 2020
fb61a3b
fix usage formatter to handle zero durations on skipped steps
davidjgoss Aug 15, 2020
29e65d2
better way to fix usage formatter
davidjgoss Aug 15, 2020
316be7c
change up order of feature
davidjgoss Aug 15, 2020
2ddaaf1
keep after(all) hooks in original order in support code library, reve…
davidjgoss Aug 16, 2020
3fb68a4
install html-formatter dep
davidjgoss Aug 16, 2020
56b2369
get minimal feature running and outputting to a file
davidjgoss Aug 17, 2020
7ee80e1
move feature patb to profile
davidjgoss Aug 17, 2020
3c0c235
run-script to pipe to html-formatter
davidjgoss Aug 17, 2020
84139c4
workaround local install issue using npx
davidjgoss Aug 17, 2020
17c28ee
better naming, add to main test script
davidjgoss Aug 17, 2020
3fa94d8
include majority of cck in formatters test
davidjgoss Aug 17, 2020
67d3a31
include cck in coverage, test formatters after coverage run
davidjgoss Aug 17, 2020
b823bc8
Merge remote-tracking branch 'origin/master' into cck
davidjgoss Aug 17, 2020
ad710a9
bump gherkin again
davidjgoss Aug 17, 2020
b6a3884
log undefined param types in summary formatter, fix test
davidjgoss Aug 17, 2020
a13f281
formatters test last in both versions
davidjgoss Aug 17, 2020
0f0af99
progress bar formatter too
davidjgoss Aug 17, 2020
56fdd79
fix time reporting
charlierudolph Aug 19, 2020
33bbe81
add missing features to formatters run
davidjgoss Aug 19, 2020
819023e
remove globby dep, just use glob
davidjgoss Aug 19, 2020
a228ec4
split seconds and nanos correctly in stopwatch
davidjgoss Aug 19, 2020
8246870
report test run timestamp + step duration sum
charlierudolph Aug 21, 2020
9e660f8
fix parallel time
charlierudolph Aug 21, 2020
3306709
fix lint
charlierudolph Aug 21, 2020
650c284
fix unit tests
charlierudolph Aug 23, 2020
c1541c0
fix lint
charlierudolph Aug 23, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@
@rerun.txt
coverage/
dist/browser-example.js
html-formatter.html
lib/
messages.ndjson
node_modules
tmp/
usage.txt
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ Please see [CONTRIBUTING.md](https://github.com/cucumber/cucumber/blob/master/CO
#### New Features

* Add support for Gherkin's [Rule/Example syntax](https://cucumber.io/docs/gherkin/reference/#rule)
* Add `transpose` method to [data table interface](docs/support_files/data_table_interface.md)
* Add `log` function to world, providing a shorthand to log plain text as [attachment(s)](docs/support_files/attachments.md)

#### Breaking changes

Expand Down
97 changes: 97 additions & 0 deletions compatibility/cck_spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
import { describe, it } from 'mocha'
import { config, expect, use } from 'chai'
import chaiExclude from 'chai-exclude'
import glob from 'glob'
import fs from 'fs'
import ndjsonParse from 'ndjson-parse'
import path from 'path'
import { PassThrough } from 'stream'
import { Cli } from '../lib'
import toString from 'stream-to-string'
import { doesHaveValue, doesNotHaveValue } from '../src/value_checker'
import { normalizeMessageOutput } from '../features/support/formatter_output_helpers'

const PROJECT_PATH = path.join(__dirname, '..')
const CCK_FEATURES_PATH = 'node_modules/@cucumber/compatibility-kit/features'
const CCK_IMPLEMENTATIONS_PATH = 'compatibility/features'

config.truncateThreshold = 100
use(chaiExclude)

describe('Cucumber Compatibility Kit', () => {
glob.sync(`${CCK_FEATURES_PATH}/**/*.ndjson`).forEach((fixturePath) => {
const suiteName = /^.+\/(.+)\.ndjson$/.exec(fixturePath)[1]
it(`passes the cck suite for '${suiteName}'`, async () => {
const args = [
'node',
path.join(PROJECT_PATH, 'bin', 'cucumber-js'),
].concat([
`${CCK_FEATURES_PATH}/${suiteName}/${suiteName}.feature`,
'--require',
`${CCK_IMPLEMENTATIONS_PATH}/${suiteName}/${suiteName}.ts`,
'--profile',
'cck',
])
const stdout = new PassThrough()
try {
await new Cli({
argv: args,
cwd: PROJECT_PATH,
stdout,
}).run()
} catch (ignored) {
console.error(ignored)
}
stdout.end()

const rawOutput = await toString(stdout)
const actualMessages = normalize(ndjsonParse(rawOutput))
const expectedMessages = ndjsonParse(
fs.readFileSync(fixturePath, { encoding: 'utf-8' })
)
expect(actualMessages)
.excludingEvery([
'meta',
// sources
'uri',
'line',
// ids
'astNodeId',
'astNodeIds',
'hookId',
'id',
'pickleId',
'pickleStepId',
'stepDefinitionIds',
'testCaseId',
'testCaseStartedId',
'testStepId',
// time
'nanos',
'seconds',
// errors
'message',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we expecting all of these to be excluded long term? I would hope with predictable ids that big block could be fixed. Do you think its worth having an issue to try to drive this list down.

Copy link
Contributor Author

@davidjgoss davidjgoss Aug 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree - when I was working against a slightly earlier CDK version I think it was only the testCase ordering that made the ids not line up.

But cck has changed from using predictable ids to uuids so not sure what the intent is there or where that leaves us.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The predictable ids and timestamps are a dead end. The cck no longer uses them.

It's a dead end because various Cucumber implementations emit messages in slightly different order than what fake-cucumber (the engine behind the cck) emits. It would be too much work to make them all emit in the same order.

So what we opted for instead is a validation mechanism where certain values (ids, timestamps) are not compared. We pair this with smoke tests that send the generated messages through 2 of the most advanced message consumers we have - cucumber-html and json-formatter. If these don't crash and a cursory eye-balling looks OK, we're confident enough that the emitted messages are ok.

])
.to.deep.eq(expectedMessages)
})
})
})

function normalize(messages: any[]): any[] {
messages = normalizeMessageOutput(
messages,
path.join(PROJECT_PATH, 'compatibility')
)
const testCases: any[] = messages.filter((message) =>
doesHaveValue(message.testCase)
)
const everythingElse: any[] = messages.filter((message) =>
doesNotHaveValue(message.testCase)
)
const testRunStarted = everythingElse.findIndex((message) =>
doesHaveValue(message.testRunStarted)
)
// move all `testCase` messages to just after `testRunStarted`
everythingElse.splice(testRunStarted + 1, 0, ...testCases)
return everythingElse
}
75 changes: 75 additions & 0 deletions compatibility/features/attachments/attachments.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
import { Before, When, World } from '../../..'
import { ReadableStreamBuffer } from 'stream-buffers'
import fs from 'fs'
import path from 'path'

Before((): void => undefined)

When('the string {string} is attached as {string}', async function (
this: World,
text: string,
mediaType: string
) {
await this.attach(text, mediaType)
})

When('the string {string} is logged', async function (
this: World,
text: string
) {
await this.log(text)
})

When('text with ANSI escapes is logged', async function (this: World) {
await this.log(
'This displays a \x1b[31mr\x1b[0m\x1b[91ma\x1b[0m\x1b[33mi\x1b[0m\x1b[32mn\x1b[0m\x1b[34mb\x1b[0m\x1b[95mo\x1b[0m\x1b[35mw\x1b[0m'
)
})

When('the following string is attached as {string}:', async function (
this: World,
mediaType: string,
text: string
) {
await this.attach(text, mediaType)
})

When('an array with {int} bytes is attached as {string}', async function (
this: World,
size: number,
mediaType: string
) {
const data = [...Array(size).keys()]
const buffer = Buffer.from(data)
await this.attach(buffer, mediaType)
})

When('a stream with {int} bytes are attached as {string}', async function (
this: World,
size: number,
mediaType: string
) {
const data = [...Array(size).keys()]
const buffer = Buffer.from(data)
const stream = new ReadableStreamBuffer({ chunkSize: 1, frequency: 1 })
stream.put(buffer)
stream.stop()
await this.attach(stream, mediaType)
})

When('a JPEG image is attached', async function (this: World) {
await this.attach(
fs.createReadStream(
path.join(
process.cwd(),
'node_modules',
'@cucumber',
'compatibility-kit',
'features',
'attachments',
'cucumber-growing-on-vine.jpg'
)
),
'image/jpg'
)
})
13 changes: 13 additions & 0 deletions compatibility/features/data-tables/data-tables.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import { When, Then, DataTable } from '../../..'
import { expect } from 'chai'

When('the following table is transposed:', function (
this: any,
table: DataTable
) {
this.transposed = table.transpose()
})

Then('it should be:', function (this: any, expected: DataTable) {
expect(this.transposed.raw()).to.deep.eq(expected.raw())
})
17 changes: 17 additions & 0 deletions compatibility/features/examples-tables/examples-tables.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import assert from 'assert'
import { Given, When, Then } from '../../..'

Given('there are {int} cucumbers', function (this: any, initialCount: number) {
this.count = initialCount
})

When('I eat {int} cucumbers', function (this: any, eatCount: number) {
this.count -= eatCount
})

Then('I should have {int} cucumbers', function (
this: any,
expectedCount: number
) {
assert.strictEqual(this.count, expectedCount)
})
21 changes: 21 additions & 0 deletions compatibility/features/hooks/hooks.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
import { When, Before, After } from '../../..'

Before(function () {
// no-op
})

When('a step passes', function () {
// no-op
})

When('a step throws an exception', function () {
throw new Error('Exception in step')
})

After(function () {
throw new Error('Exception in hook')
})

After('@some-tag or @some-other-tag', function () {
throw new Error('Exception in conditional hook')
})
6 changes: 6 additions & 0 deletions compatibility/features/minimal/minimal.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
import assert from 'assert'
import { Given } from '../../..'

Given('I have {int} cukes in my belly', function (cukeCount: number) {
assert(cukeCount)
})
22 changes: 22 additions & 0 deletions compatibility/features/parameter-types/parameter-types.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
import { Given, defineParameterType } from '../../..'
import { expect } from 'chai'

class Flight {
constructor(public readonly from: string, public readonly to: string) {}
}

defineParameterType({
name: 'flight',
regexp: /([A-Z]{3})-([A-Z]{3})/,
transformer(from: string, to: string) {
return new Flight(from, to)
},
})

Given('{flight} has been delayed {int} minutes', function (
flight: Flight,
delay: number
) {
expect(flight.from).to.eq('LHR')
expect(flight.to).to.eq('CDG')
})
42 changes: 42 additions & 0 deletions compatibility/features/rules/rules.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
import assert from 'assert'
import { Given, When, Then } from '../../..'

Given('there are {int} {float} coins inside', function (
count: number,
denomination: number
) {
// TODO: implement this
assert(count)
assert(denomination)
})

Given('there are no chocolates inside', function () {
// TODO: implement this
})

Given('there are {int} chocolates inside', function (chocolateCount: number) {
// TODO: implement this
assert(chocolateCount)
})

When(
'the customer tries to buy a {float} chocolate with a {float} coin',
function (price: number, paid: number) {
// TODO: implement this
assert(price)
assert(paid)
}
)

Then('the sale should not happen', function () {
// TODO: implement this
})

Then("the customer's change should be {int} {float} coin(s)", function (
count: number,
denomination: number
) {
// TODO: implement this
assert(count)
assert(denomination)
})
5 changes: 5 additions & 0 deletions compatibility/features/stack-traces/stack-traces.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import { When } from '../../..'

When('a step throws an exception', function () {
throw new Error('BOOM')
})
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
import { Given } from '../../..'

// eslint-disable-next-line @typescript-eslint/no-unused-vars
Given('{airport} is closed because of a strike', function (airport: any) {
throw new Error('Should not be called because airport type not defined')
})
36 changes: 34 additions & 2 deletions cucumber.js
Original file line number Diff line number Diff line change
@@ -1,13 +1,45 @@
var common = [
const feature = [
'--require-module ts-node/register',
'--require features/**/*.ts',
`--format ${
process.env.CI || !process.stdout.isTTY ? 'progress' : 'progress-bar'
}`,
'--format rerun:@rerun.txt',
'--format usage:usage.txt',
'--format message:messages.ndjson',
].join(' ')

const cck = [
'--require-module',
'ts-node/register',
'--format',
'message',
].join(' ')

const FORMATTERS_INCLUDE = [
'attachments',
'data-tables',
'examples-tables',
'minimal',
'parameter-types',
'rules',
'stack-traces',
]

const formatters = [
`node_modules/@cucumber/compatibility-kit/features/{${FORMATTERS_INCLUDE.join(
','
)}}/*.feature`,
'--require-module',
'ts-node/register',
'--require',
`compatibility/features/{${FORMATTERS_INCLUDE.join(',')}}/*.ts`,
'--format',
'message',
].join(' ')

module.exports = {
default: common,
default: feature,
cck,
formatters,
}
3 changes: 2 additions & 1 deletion dependency-lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ ignoreErrors:
shouldBeDependency: []
shouldBeDevDependency: []
unused:
- '@cucumber/compatibility-kit' # files dynamically loaded in cck test, not require'd
- '@typescript-eslint/eslint-plugin' # peer dependency of standard-with-typescript
- '@typescript-eslint/parser' # peer dependency of @typescript-eslint/eslint-plugin
- '@types/*' # type definitions
Expand All @@ -40,7 +41,7 @@ ignoreErrors:
requiredModules:
files:
dev:
- '{features,scripts,test}/**/*'
- '{compatibility,features,scripts,test}/**/*'
- '**/*_spec.ts'
- 'example/index.ts'
- '**/test_helpers.ts'
Expand Down
Loading