-
Notifications
You must be signed in to change notification settings - Fork 1.9k
fix TextLoader bug when there's newline between quotes #4584
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -487,7 +487,7 @@ private void ThreadProc() | |
// REVIEW: Avoid allocating a string for every line. This would probably require | ||
// introducing a CharSpan type (similar to ReadOnlyMemory but based on char[] or StringBuilder) | ||
// and implementing all the necessary conversion functionality on it. See task 3871. | ||
text = rdr.ReadLine(); | ||
text = rdr.ReadEntry(); | ||
if (text == null) | ||
goto LNext; | ||
line++; | ||
|
@@ -514,7 +514,7 @@ private void ThreadProc() | |
if (_abort) | ||
return; | ||
|
||
text = rdr.ReadLine(); | ||
text = rdr.ReadEntry(); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should check if moving to multi-line reading will interfere with multi-threaded reading. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How to check that, could you expand a bit? |
||
if (text == null) | ||
{ | ||
// We're done with this file. Queue the last partial batch. | ||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -174,5 +174,39 @@ private static string[] Expand(string pattern) | |||||
return matchList.ToArray(); | ||||||
} | ||||||
#endif | ||||||
|
||||||
public static string ReadEntry(this TextReader sr) | ||||||
{ | ||||||
string entry = string.Empty; | ||||||
|
||||||
// get first bit | ||||||
entry += sr.ReadLine(); | ||||||
|
||||||
// And get more lines until the number of quotes is even | ||||||
while (GetNumberOf(entry, "\"") % 2 != 0 ) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Need to first check that that quoting is enabled, and need to check for escaped quotes. May need a flag/field to define the escaping type, defaulting to allow either none or both. Some tests:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also need to not append to lines at the start of the file which begin with if (entry[0] == '#' || entry[0] == '/' && entry[1] == '/') { ... } There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So try to repeat the logic here read And BTW it seems the existing enable quote logic doesn't check if it's real quote or not. ( for example in lines: "blablabla""blablabla" will cause an error in |
||||||
{ | ||||||
string line = sr.ReadLine(); | ||||||
entry += line; | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Likely want to begin a string builder instead of a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You'll want to push thru a newline into the string builder. Otherwise:
Would be read as:
Instead of:
It likely is not important, but it may be better to match the |
||||||
} | ||||||
|
||||||
// Then return what we've gotten | ||||||
if (entry == string.Empty) | ||||||
{ | ||||||
return null; | ||||||
} | ||||||
else | ||||||
{ | ||||||
return entry; | ||||||
} | ||||||
} | ||||||
|
||||||
public static int GetNumberOf(string s, string strSearchString) | ||||||
{ | ||||||
if(strSearchString.Length == 0 || s.Length == 0) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
{ | ||||||
return 0; | ||||||
} | ||||||
return (s.Length - s.Replace(strSearchString, string.Empty).Length) / strSearchString.Length; | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||||||
} | ||||||
} | ||||||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Recommend a flag for multi-line support.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it can share the same flag with EnableQuote, If we allow quote, then multi-line support is necessary to make sure TextLoader function well