Possible parsing bug relating to CR and CLF about appinventor-sources HOT 10 CLOSED

I'm not sure there's an error here. I'm asking Taifun to explain his comment.

from appinventor-sources.

I don't believe that is an issue.
The problem is how the text file is created.
Windows adds CRLF to end of line.
Linux does only LF.
replacing LF with "," as shown in thread, WILL NOT get rid of CR.

Will follow-up in thread

from appinventor-sources.

Right. I think the confusion is compounded by the way the repl baloon
prints \n as linefeed by doesn't treat \r specially.
I'm not sure whether we should do anything about it, though.

== Hal Abelson
[email protected]
Prof. of Comp. Sci. and Eng.
MIT Dept. of Elec. Eng. and Comp. Sci.
MIT Media Lab, Second Class Faculty

On Sun, Dec 28, 2014 at 9:39 PM, kkashi01 [email protected] wrote:

I don't believe that is an issue.
The problem is how the text file is created.
Windows adds CRLF to end of line.
Linux does only LF.
replacing LF with "," as shown in thread, WILL NOT get rid of CR.

Will follow-up in thread

—
Reply to this email directly or view it on GitHub
#169 (comment)
.

from appinventor-sources.

We won't fix this. It's a confusion coming partially from the standard windows CRLF versus LF issue and partially from the fact that \n in App Inventor strings is left as an escaped character, while \r is not. That processing is done in appinventor/blocklyeditor/src/generators/yail.js, in Blockly.Yail.quotifyForREPL. Perhaps we could treat other formatting characters specially some day, but it's not clear what to do.

from appinventor-sources.

as already said in the forum https://groups.google.com/d/msg/mitappinventortest/tzoNdh4HyYY/HU0Gelg8aZ0J

From a user point of view it doesn't matter if a file was created with Windows or Linux/Mac, the user only wants to get rid of the invisible newline character (be it CRLF or LF only)
so this currently works only for Linux/Mac files (LF), but not for Windows files (CRLF)...
If I use the replace all block to replace \n, I would expect to get the newline character(s) removed (be it CRLF or LF only)

Taifun

from appinventor-sources.

I agree. I want Lyn to comment before we change the parser.

== Hal Abelson
[email protected]
Prof. of Comp. Sci. and Eng.
MIT Dept. of Elec. Eng. and Comp. Sci.
MIT Media Lab, Second Class Faculty

On Tue, Dec 30, 2014 at 9:14 AM, Taifun [email protected] wrote:

as already said in the forum
https://groups.google.com/d/msg/mitappinventortest/tzoNdh4HyYY/HU0Gelg8aZ0J

From a user point of view it doesn't matter if a file was created with
Windows or Linux/Mac, the user only wants to get rid of the invisible
newline character (be it CRLF or LF only)
so this currently works only for Linux/Mac files (LF), but not for Windows
files (CRLF)...
If I use the replace all block to replace \n, I would expect to get the
newline character(s) removed (be it CRLF or LF only)

Taifun

—
Reply to this email directly or view it on GitHub
#169 (comment)
.

from appinventor-sources.

I just posted this on the forum:

I agree with Taifun that App Inventor's \n should match Window's CRLF
(which is \r\n, not \n\r, BTW) whenever possible.

The problem is that we can't always know when to do The Right Thing.

When reading files/strings created in Windows, we can treat every \r\n as
if it were \n, and this will make a lot of things work (e.g. using the
replace block with \n) that don't currently work.

But when writing \n in a file/string, we have no idea whether that
file/string will end up being used in Windows. So the best we can do is
simply write the \n, but this won't show up correctly when viewed in
Windows.

• lyn -

On Wed, Dec 31, 2014 at 3:16 PM, Hal Abelson [email protected] wrote:

I agree. I want Lyn to comment before we change the parser.

== Hal Abelson
[email protected]
Prof. of Comp. Sci. and Eng.
MIT Dept. of Elec. Eng. and Comp. Sci.
MIT Media Lab, Second Class Faculty

On Tue, Dec 30, 2014 at 9:14 AM, Taifun [email protected] wrote:

as already said in the forum
https://groups.google.com/d/msg/mitappinventortest/tzoNdh4HyYY/HU0Gelg8aZ0J

From a user point of view it doesn't matter if a file was created with
Windows or Linux/Mac, the user only wants to get rid of the invisible
newline character (be it CRLF or LF only)
so this currently works only for Linux/Mac files (LF), but not for
Windows files (CRLF)...
If I use the replace all block to replace \n, I would expect to get the
newline character(s) removed (be it CRLF or LF only)

Taifun

—
Reply to this email directly or view it on GitHub
#169 (comment)
.

from appinventor-sources.

I think the question is which is the lesser of the evils for App Inventor.

What bad things happen if we change

Blockly.Yail.quotifyForREPL so that \n \r is a string gets transformed to
an unescaped \n?

== Hal Abelson
[email protected]
Prof. of Comp. Sci. and Eng.
MIT Dept. of Elec. Eng. and Comp. Sci.
MIT Media Lab, Second Class Faculty

On Wed, Dec 31, 2014 at 5:33 PM, Franklyn A Turbak [email protected]
wrote:

I just posted this on the forum:

I agree with Taifun that App Inventor's \n should match Window's CRLF
(which is \r\n, not \n\r, BTW) whenever possible.

The problem is that we can't always know when to do The Right Thing.

When reading files/strings created in Windows, we can treat every \r\n as
if it were \n, and this will make a lot of things work (e.g. using the
replace block with \n) that don't currently work.

But when writing \n in a file/string, we have no idea whether that
file/string will end up being used in Windows. So the best we can do is
simply write the \n, but this won't show up correctly when viewed in
Windows.

• lyn -

On Wed, Dec 31, 2014 at 3:16 PM, Hal Abelson [email protected] wrote:

I agree. I want Lyn to comment before we change the parser.

== Hal Abelson
[email protected]
Prof. of Comp. Sci. and Eng.
MIT Dept. of Elec. Eng. and Comp. Sci.
MIT Media Lab, Second Class Faculty

On Tue, Dec 30, 2014 at 9:14 AM, Taifun [email protected] wrote:

as already said in the forum
https://groups.google.com/d/msg/mitappinventortest/tzoNdh4HyYY/HU0Gelg8aZ0J

From a user point of view it doesn't matter if a file was created with
Windows or Linux/Mac, the user only wants to get rid of the invisible
newline character (be it CRLF or LF only)
so this currently works only for Linux/Mac files (LF), but not for
Windows files (CRLF)...
If I use the replace all block to replace \n, I would expect to get the
newline character(s) removed (be it CRLF or LF only)

Taifun

—
Reply to this email directly or view it on GitHub
#169 (comment)
.

from appinventor-sources.

Just to emphasize: the transformation is "\r\n" => "\n", not "\n\r" =>
"\n". That is, in CRLF, the CR (\r) comes before the LF (\n).

I think that overall this suggested change is a win, because it will help
in App Inventor files that process files created in windows.

In terms of bad things than can happen: I don't see anything that's any
worse than what currently happens. Suppose someone wants to write a program
that adds some character (say $) to every line in a file. They can do this
by reading the file as a string, replacing "\n" by "$\n", and then writing
the string to a file. This will currently work in Mac and Linux, but not
Windows, because "\r\n" will get transformed to "\r$\n". With the suggested
changed, it still won't work in Windows, because "\r\n" will get
transformed to "$\n". But this is better than "\r$\n".

• lyn -

On Wed, Dec 31, 2014 at 5:59 PM, Hal Abelson [email protected] wrote:

I think the question is which is the lesser of the evils for App
Inventor.

What bad things happen if we change

Blockly.Yail.quotifyForREPL so that \n \r is a string gets transformed to
an unescaped \n?

== Hal Abelson
[email protected]
Prof. of Comp. Sci. and Eng.
MIT Dept. of Elec. Eng. and Comp. Sci.
MIT Media Lab, Second Class Faculty

On Wed, Dec 31, 2014 at 5:33 PM, Franklyn A Turbak [email protected]
wrote:

I just posted this on the forum:

I agree with Taifun that App Inventor's \n should match Window's CRLF
(which is \r\n, not \n\r, BTW) whenever possible.

The problem is that we can't always know when to do The Right Thing.

When reading files/strings created in Windows, we can treat every \r\n as
if it were \n, and this will make a lot of things work (e.g. using the
replace block with \n) that don't currently work.

But when writing \n in a file/string, we have no idea whether that
file/string will end up being used in Windows. So the best we can do is
simply write the \n, but this won't show up correctly when viewed in
Windows.

• lyn -

On Wed, Dec 31, 2014 at 3:16 PM, Hal Abelson [email protected] wrote:

I agree. I want Lyn to comment before we change the parser.

== Hal Abelson
[email protected]
Prof. of Comp. Sci. and Eng.
MIT Dept. of Elec. Eng. and Comp. Sci.
MIT Media Lab, Second Class Faculty

On Tue, Dec 30, 2014 at 9:14 AM, Taifun [email protected]
wrote:

as already said in the forum
https://groups.google.com/d/msg/mitappinventortest/tzoNdh4HyYY/HU0Gelg8aZ0J

From a user point of view it doesn't matter if a file was created with
Windows or Linux/Mac, the user only wants to get rid of the invisible
newline character (be it CRLF or LF only)
so this currently works only for Linux/Mac files (LF), but not for
Windows files (CRLF)...
If I use the replace all block to replace \n, I would expect to get the
newline character(s) removed (be it CRLF or LF only)

Taifun

—
Reply to this email directly or view it on GitHub
#169 (comment)
.

from appinventor-sources.

Thanks. I think we'll take the plunge and do that.

And thanks for correcting my /n/r rather than /r/n bug before I made it :)

I guess this needs a note somewhere but I'm not sure where.

== Hal Abelson
[email protected]
Prof. of Comp. Sci. and Eng.
MIT Dept. of Elec. Eng. and Comp. Sci.
MIT Media Lab, Second Class Faculty

On Wed, Dec 31, 2014 at 6:16 PM, Franklyn A Turbak [email protected]
wrote:

Just to emphasize: the transformation is "\r\n" => "\n", not "\n\r" =>
"\n". That is, in CRLF, the CR (\r) comes before the LF (\n).

I think that overall this suggested change is a win, because it will help
in App Inventor files that process files created in windows.

In terms of bad things than can happen: I don't see anything that's any
worse than what currently happens. Suppose someone wants to write a program
that adds some character (say $) to every line in a file. They can do this
by reading the file as a string, replacing "\n" by "$\n", and then writing
the string to a file. This will currently work in Mac and Linux, but not
Windows, because "\r\n" will get transformed to "\r$\n". With the suggested
changed, it still won't work in Windows, because "\r\n" will get
transformed to "$\n". But this is better than "\r$\n".

• lyn -

On Wed, Dec 31, 2014 at 5:59 PM, Hal Abelson [email protected] wrote:

I think the question is which is the lesser of the evils for App
Inventor.

What bad things happen if we change

Blockly.Yail.quotifyForREPL so that \n \r is a string gets transformed
to an unescaped \n?

== Hal Abelson
[email protected]
Prof. of Comp. Sci. and Eng.
MIT Dept. of Elec. Eng. and Comp. Sci.
MIT Media Lab, Second Class Faculty

On Wed, Dec 31, 2014 at 5:33 PM, Franklyn A Turbak <[email protected]

wrote:

I just posted this on the forum:

I agree with Taifun that App Inventor's \n should match Window's CRLF
(which is \r\n, not \n\r, BTW) whenever possible.

The problem is that we can't always know when to do The Right Thing.

When reading files/strings created in Windows, we can treat every \r\n
as if it were \n, and this will make a lot of things work (e.g. using the
replace block with \n) that don't currently work.

But when writing \n in a file/string, we have no idea whether that
file/string will end up being used in Windows. So the best we can do is
simply write the \n, but this won't show up correctly when viewed in
Windows.

• lyn -

On Wed, Dec 31, 2014 at 3:16 PM, Hal Abelson [email protected] wrote:

I agree. I want Lyn to comment before we change the parser.

== Hal Abelson
[email protected]
Prof. of Comp. Sci. and Eng.
MIT Dept. of Elec. Eng. and Comp. Sci.
MIT Media Lab, Second Class Faculty

On Tue, Dec 30, 2014 at 9:14 AM, Taifun [email protected]
wrote:

as already said in the forum
https://groups.google.com/d/msg/mitappinventortest/tzoNdh4HyYY/HU0Gelg8aZ0J

From a user point of view it doesn't matter if a file was created with
Windows or Linux/Mac, the user only wants to get rid of the invisible
newline character (be it CRLF or LF only)
so this currently works only for Linux/Mac files (LF), but not for
Windows files (CRLF)...
If I use the replace all block to replace \n, I would expect to get
the newline character(s) removed (be it CRLF or LF only)

Taifun

—
Reply to this email directly or view it on GitHub
#169 (comment)
.

from appinventor-sources.