Bug: v4.0.x fails snapshot comparison when using a custom serializer

[utluiz]

Hi,

I tried to upgrade a project of mine to 4.0.2 and found that all tests with snapshots started failing after the upgrade. The .snap.debug file created only contains the snapshot of the last test that was executed within a test class.

I made a few tests in a sample project with minimum configuration and found that it doesn't happen if I use the default serializer, just when I use a custom one. The thing is, it used to work with 3.x and, even if I had a bad implementation, I don't think the serializer should be able to break the entire snapshot file.

Reproducing the issue

1. Create custom serializer:

public class JacksonYamlSnapshotSerializer implements SnapshotSerializer {
    private final ObjectMapper yamlMapper = new YAMLMapper()
            .enable(YAMLGenerator.Feature.LITERAL_BLOCK_STYLE)
            .disable(YAMLGenerator.Feature.WRITE_DOC_START_MARKER)
            .enable(SerializationFeature.WRITE_DATES_WITH_ZONE_ID)
            .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS)
            .disable(SerializationFeature.FAIL_ON_EMPTY_BEANS)
            .enable(SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS)
            .setSerializationInclusion(JsonInclude.Include.NON_NULL);

    @Override
    public String getOutputFormat() {
        return SerializerType.JSON.name();
    }

    @Override
    public Snapshot apply(final Object object, final SnapshotSerializerContext gen) {
        try {
            List<?> objects = Arrays.asList(object);
            return gen.toSnapshot("\n" + yamlMapper.writeValueAsString(objects));
        } catch (Exception e) {
            throw new SnapshotExtensionException("Yaml serialization failed", e);
        }
    }
}

2. Set it as the default serializer in snapshot.properties:

serializer=org.example.JacksonYamlSnapshotSerializer
comparator=au.com.origin.snapshots.comparators.PlainTextEqualsComparator
reporters=au.com.origin.snapshots.reporters.PlainTextSnapshotReporter
snapshot-dir=__snapshots__
output-dir=src/test/java
ci-env-var=CI

3. Write a test class with two or more test methods:

@ExtendWith(SnapshotExtension.class)
public class ExampleTest {
    Expect expect;

    @Test
    public void test1() {
        expect.toMatchSnapshot(new Foo("foo1", 1));
    }

    @Test
    public void test2() {
        expect.toMatchSnapshot(new Foo("foo2", 2));
    }

    record Foo(String foo, int bar) {}
}

4. Perform a first run (which should succeed) in order to generate the snapshot file like this:

org.example.ExampleTest.test1=
- foo: "foo1"
  bar: 1



org.example.ExampleTest.test2=
- foo: "foo2"
  bar: 2

5. Run the test again and you'll get an error and a snap.debug file as follows:

org.example.ExampleTest.test2=
- foo: "foo2"
  bar: 2

Project configuration

My project uses Java 17 (Corretto, latest version).

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>java-snapshot-testing-bug</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>5.9.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>5.9.1</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>io.github.origin-energy</groupId>
            <artifactId>java-snapshot-testing-junit5</artifactId>
            <version>4.0.2</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>io.github.origin-energy</groupId>
            <artifactId>java-snapshot-testing-plugin-jackson</artifactId>
            <version>4.0.2</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>2.0.6</version>
        </dependency>


        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.dataformat</groupId>
            <artifactId>jackson-dataformat-yaml</artifactId>
        </dependency>

    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>com.fasterxml.jackson</groupId>
                <artifactId>jackson-bom</artifactId>
                <version>2.14.1</version>
                <scope>import</scope>
                <type>pom</type>
            </dependency>
        </dependencies>
    </dependencyManagement>

</project>

[jackmatt2]

From a high level it looks like you've done everything right - I'll try to reproduce locally.

[jackmatt2]

Hi @utluiz

The issue is that the YAML Serializer is outputting a single blank line at the end of the output. The REGEX used by this library is failing for the SPLIT_STRING (3 new lines) and incorrectly picks up one of these new lines. This doesn't happen for String snapshots because they are bounded by the characters [ & ].

This issue should be fixed once encoding of the snapshot takes place (future update #130) in which blank lines will be escaped.

For now you can simply trim() the YAML output to fix this issue.

   @Override
    public Snapshot apply(final Object object, final SnapshotSerializerContext gen) {
        try {
            List<?> objects = Arrays.asList(object);
            String raw = yamlMapper.writeValueAsString(objects);
            return gen.toSnapshot(raw.trim());
        } catch (Exception e) {
            throw new SnapshotExtensionException("Yaml serialization failed", e);
        }
    }

If you wan't to keep the New lines at the end of the snapshot. An alternative would be to bound the output between braces [ & ] like the String Serializer does.