Skip to content

fix TextLoader bug when there's newline between quotes #4584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/Microsoft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs
Original file line number Diff line number Diff line change
Expand Up @@ -487,7 +487,7 @@ private void ThreadProc()
// REVIEW: Avoid allocating a string for every line. This would probably require
// introducing a CharSpan type (similar to ReadOnlyMemory but based on char[] or StringBuilder)
// and implementing all the necessary conversion functionality on it. See task 3871.
text = rdr.ReadLine();
text = rdr.ReadEntry();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommend a flag for multi-line support.

Copy link
Contributor Author

@LittleLittleCloud LittleLittleCloud Dec 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it can share the same flag with EnableQuote, If we allow quote, then multi-line support is necessary to make sure TextLoader function well

if (text == null)
goto LNext;
line++;
Expand All @@ -514,7 +514,7 @@ private void ThreadProc()
if (_abort)
return;

text = rdr.ReadLine();
text = rdr.ReadEntry();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should check if moving to multi-line reading will interfere with multi-threaded reading.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How to check that, could you expand a bit?

if (text == null)
{
// We're done with this file. Queue the last partial batch.
Expand Down
34 changes: 34 additions & 0 deletions src/Microsoft.ML.Data/Utilities/StreamUtils.cs
Original file line number Diff line number Diff line change
Expand Up @@ -174,5 +174,39 @@ private static string[] Expand(string pattern)
return matchList.ToArray();
}
#endif

public static string ReadEntry(this TextReader sr)
{
string entry = string.Empty;

// get first bit
entry += sr.ReadLine();

// And get more lines until the number of quotes is even
while (GetNumberOf(entry, "\"") % 2 != 0 )
Copy link
Contributor

@justinormont justinormont Dec 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to first check that that quoting is enabled, and need to check for escaped quotes. May need a flag/field to define the escaping type, defaulting to allow either none or both.

Some tests:

  • True,"asdf,123 is perfectly ok without quoting enabled
  • True,"12\" in a foot",123 has an escaped quote w/ slash escaping
  • True,"12"" in a foot",123 has an escaped quote w/ double-double quotes (Excel style)
  • True, "asdf,123 is ok, if following Excel style as it ignores quoted fields with a space before

Copy link
Contributor

@justinormont justinormont Dec 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also need to not append to lines at the start of the file which begin with // or #:

if (entry[0] == '#' ||  entry[0] == '/' && entry[1] == '/') { ... }

Copy link
Contributor Author

@LittleLittleCloud LittleLittleCloud Dec 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So try to repeat the logic here

read line
if 'line' starts with '#' or starts with '\' => return 'line' # comments
if quote is disabled: => use the existing logic in TextLoader
else
if ( 'line' has only odd# real quote ) => read next 'line' until sum of real quote is even and return 'line' # real quote means " and it's not one of [ "", \" "]
else return 'line'

And BTW it seems the existing enable quote logic doesn't check if it's real quote or not. ( for example in lines: "blablabla""blablabla" will cause an error in FetchNextField function, but in Excel it's legal) Do we have a plan for adding support for that?

{
string line = sr.ReadLine();
entry += line;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likely want to begin a string builder instead of a +=.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll want to push thru a newline into the string builder. ReadLine() doesn't include the \n or \r or \r\n [ref].

Otherwise:

a,b,"c my missing
space",e,f

Would be read as:

a,b,"c my missingspace",e,f

Instead of:

a,b,"c my missing\nspace",e,f

It likely is not important, but it may be better to match the \n or \r or \r\n of the last line read. This would only be important if the user read the file w/ TextLoader but fed the model with an IEnumerable.

}

// Then return what we've gotten
if (entry == string.Empty)
{
return null;
}
else
{
return entry;
}
}

public static int GetNumberOf(string s, string strSearchString)
{
if(strSearchString.Length == 0 || s.Length == 0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if(strSearchString.Length == 0 || s.Length == 0)
if (strSearchString.Length == 0 || s.Length == 0)

{
return 0;
}
return (s.Length - s.Replace(strSearchString, string.Empty).Length) / strSearchString.Length;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

String.Replace() is not efficient as it reallocates.

}
}
}