feat: ndjson can't support compress feature

[wubx]

Search before asking

• I had searched in the issues and found no similar issues.

Version

MySQL [(none)]> select version();
+----------------------------------------------------------------------------------------+
| version()                                                                              |
+----------------------------------------------------------------------------------------+
| 8.0.26-v0.7.165-nightly-a6b27a6-simd(rust-1.64.0-nightly-2022-08-15T02:05:18.6553749Z) |
+----------------------------------------------------------------------------------------+
1 row in set (0.001 sec)

What's Wrong?

streaming_load can't work with ndjson use gzip.

How to Reproduce?

get github archive file

wget https://data.gharchive.org/2021-01-01-{0..23}.json.gz

Databend create table:

create table g_variant(
    id varchar,
    type varchar,
    actor variant,
    repo variant,
    payload variant,
    public boolean,
    created_at datetime(0),
    org variant
);

if you use gzip compress file and use streaming_load is work ok.

curl -H "insert_sql:insert into g_variant format NdJson" -H "skip_header:0" -H 'field_delimiter: ,' -H 'record_delimiter: \n' -F "[email protected]" -u root: -XPUT "http://localhost:8000/v1/streaming_load"

if you use streaming_load compress feature is can't work:

 curl -H "insert_sql:insert into g_variant format NdJson" -H "skip_header:0" -H 'field_delimiter: ,' -H 'record_delimiter: \n' -H "compression:gzip" -F "[email protected]" -u root: -XPUT "http://localhost:8000/v1/streaming_load"

Code: 1046, displayText = Parse NDJson error at line 0, cause: stream did not contain valid UTF-8 (while in processor thread 0).

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[wubx]

wu@fz001:~/wubx/github$ curl -H "insert_sql:insert into g_variant format NdJson" -H "skip_header:0" -H 'field_delimiter: ,' -H 'record_delimiter: \n' -H "compression:gzip" -F "[email protected]" -u root: -XPUT "http://localhost:8000/v1/streaming_load"
{"id":"6385e94d-1155-4f89-9827-a80969cc5a74","state":"SUCCESS","stats":{"rows":120069,"bytes":434000615},"error":null}

wu@fz001:~/wubx/github$ echo "select version()"|mysql -h 127.0.0.1 -P3307 -uroot
version()
8.0.26-v0.8.25-nightly-6a5d857-simd(rust-1.64.0-nightly-2022-09-04T16:37:49.86847381Z)

wu@fz001:~/wubx/github$ echo "select count() from g_variant"|mysql -h 127.0.0.1 -P3307 -uroot
COUNT()
120069