Tee-Object removes formatting from the byte stream about powershell HOT 6 OPEN

@rhubarb-geek-nz, no argument there - but I'm not sure what you think you're contradicting / clarifying. I was merely referring to the fact that when you do something like ls --color=auto 2>&1 in a POSIX-compatible shell, you still get coloring, because stdout isn't redirected.

from powershell.

So the byte stream is passed directly if the downstream command is also native or it's being redirected to a file. It is also no longer passed if you redirect error into out, as it's assumed to be text based at that point.

That said, byte stream or text isn't what makes the difference in formatting, most console applications will omit formatting when redirected regardless of how that redirection is handled. So even if a PowerShell command downstream would be able to receive the bytes, you still wouldn't see colored text.

The first example should act similarly to the latter, but in the opposite way that you're expecting. The fact that it isn't means that we aren't actually redirecting stderr to out here, which is a different bug.

from powershell.

@SeeminglyScience, I don't know what's going on behind the scenes, but the 2>&1 redirection does appear to be effective when not captured / redirected in-session, as seen by an outside caller (in-session, with to-host output, it is a moot point).

Yet, at the same time, the native utility apparently does not see its stdout as redirected, and therefore uses coloring - arguably, that is the preferable behavior, because that's how it works in POSIX-compatible shells.

Consider the following example:

# Run from Bash on Linux or macOS
pwsh -nop -c "sh -c 'ls nosuch; grep --color=auto o <<<foo' 2>&1" 2>/dev/null

Result:

Note how:

• both outputs printed, despite discarding stderr output on the Bash side, implying that everything was received via stdout, as intended
• grep's output is still colored, implying that it didn't see its stdout as redirected.
• the original stderr line prints red, which is an arguably unfortunate side effect of having been merged into the success output stream wrapped in an [ErrorRecord] instance.

Also note now the output appeared out of sequence - which is a fundamental and seemingly unsolvable problem with how PowerShell merges stderr and stdout into its success output stream:

• See #5424

from powershell.

Yet, at the same time, the native utility apparently does not see its stdout as redirected, and therefore uses coloring - arguably, that is the preferable behavior, because that's how it works in POSIX-compatible shells.

It that true? In a standard UNIX shell, If a program is writing to the console then the file descriptor for stdout would be a terminal or a psuedo-terminal. If it was rediirected it would be a pipe or file.

Note that UNIX command tty refers to stdin

from powershell.

Thinking about this some more:

The status quo:

• It looks like only stderr is captured and eventually sent to stdout in the native 2>&1 case, whereas stdout is passed straight through, as happens when no redirections are involved (for stderr too).

  • Because stdout is passed straight through, a native utility that uses conditional coloring based on whether or not its stdout is connected to a terminal therefore still uses coloring.

  • Even the captured-by-PowerShell stderr lines retain explicit coloring, as the following example shows:

    # Run on Linux or macOS
    sh -c "printf 'It ain'\''t easy being \033[32mgreen\033[m. Or is it?\n' >&2" 2>&1
    

  • Output:

    • Note how PowerShell's default styling of error-stream output - red - was then overridden by the explicit ANSI/VT sequence in the string, and resetting the colors remained in effect for the rest of the string.

• While an outside caller of the PowerShell CLI therefore in effect does see both the original stdout and stderr output as combined stdout output, as demonstrated in my previous comment, there are two problems:

  • The loss of the original output sequencing across the streams.

    • Given the stderr-only handling by PowerShell in this case, output-sequencing problems are now more likely to surface than before, due to the stderr-only lag in processing.

    • The previous behavior, described in #5424, where sequencing problems only intermittently showed is still at play in the native 2>&1 | native case, where of necessity stdout is then captured by PowerShell too. For instance, in the following example the sequencing problem is much harder to reproduce (takes an unpredictable number of repeat invocations), due to the addition of | cat -n

      # Run on Linux or macOS
      sh -c 'ls nosuch; grep --color=auto o <<<foo' 2>&1 | cat -n
      

  • The unexpected red styling of the original stderr lines.

  • On the plus side, the non-capturing of the original stdout by PowerShell preserves the expected coloring behavior.

A potential solution:

• In the native 2>&1 and native 2>&1 | native / native 2>&1 >$file / native *> $file cases, merge the streams at the source, the way POSIX-compatible shells do:

  • In all cases, problems without output sequencing would be avoided.
  • In the native 2>&1 case, the merged-at-the-source stdout stream could then passed through by PowerShell (as without any redirections), which avoids the problem with the stderr lines getting colored red.

• Unfortunately, for the sake of backward compatibility we're stuck with the (less severe form of the) sequencing problem in scenarios such as $combinedOutput = native 2>&1, i.e. in cases where PowerShell commands operate on the results, as merging the streams at the source would forgo the existing ability to later distinguish which lines came from what stream via their data types ([string] vs. [ErrorRecrod]).

from powershell.

@cveld, to elaborate on @SeeminglyScience's comment re the behavior of Tee-Object: because the coloring behavior is typically governed by wether the input utility sees its stdout as being connected to a terminal or not, it doesn't matter which command you pipe to; the mere fact of piping itself deactivates coloring.

Thus, the native tee (/usr/bin/tee) utility behaves the same, for instance, whether run from PowerShell or a POSIX-compatible shell; e.g.:

# Run from either PowerShell or Bash

# NO coloring, due to use of a pipe:
# (`--color=auto` requests the behavior most utilities exhibit by default: use color only when printing to the terminal)
ls --color=auto | tee /dev/null

# If the input utility allows it, request *unconditional* coloring, which `tee` then passes through.
# The same goes for the PowerShell equivalent:
#       ls --color=always | Tee-Object /dev/null
ls --color=always | tee /dev/null

Tee-Object is incidental to the real problem at hand, which stems from the behavior of 2>&1 itself.

from powershell.