Home » Can I configure my shell to print STDERR and STDOUT in different colors?

Can I configure my shell to print STDERR and STDOUT in different colors?

Solutons:


Check out stderred. It uses LD_PRELOAD to hook to libc‘s write() calls, colorizing all stderr output going to a terminal. (In red by default.)

This is a harder version of Show only stderr on screen but write both stdout and stderr to file.

The applications running in the terminal use a single channel to communicate with it; the applications have two output ports, stdout and stderr, but they’re both connected to the same channel.

You can connect one of them to a different channel, add color to that channel, and merge the two channels, but this will cause two problems:

  • The merged output may not be exactly in the same order as if there had been no redirection. This is because the added processing on one of the channel takes (a little) time, so the colored channel may be delayed. If any buffering is done, the disorder will be worse.
  • Terminals use color changing escape sequences to determine the display color, e.g. ␛[31m means “switch to red foreground”. This means that if some output destined to stdout arrives just as some output for stderr is being displayed, the output will be miscolored. (Even worse, if there’s a channel switch in the middle of an escape sequence, you’ll see garbage.)

In principle, it would be possible to write a program that listens on two ptys¹, synchronously (i.e. won’t accept input on one channel while it’s processing output on the other channel), and immediately outputs to the terminal with appropriate color changing instructions. You’d lose the ability to run programs that interact with the terminal. I don’t know of any implementation of this method.

Another possible approach would be to cause the program to output the proper color changing sequences, by hooking around all the libc functions that call the write system call in a library loaded with LD_PRELOAD. See sickill’s answer for an existing implementation, or Stéphane Chazelas’s answer for a mixed approach that leverages strace.

In practice, if that’s applicable, I suggest redirecting stderr to stdout and piping into a pattern-based colorizer such as colortail or multitail, or special-purpose colorizers such as colorgcc or colormake.

¹ pseudo-terminals. Pipes wouldn’t work because of buffering: the source could write to the buffer, which would break the synchronicity with the colorizer.

Colouring user input is difficult because in half the cases, it is output by the terminal driver (with local echo) so in that case, no application running in that terminal may know when the user is going to type text and change the output colour accordingly. Only the pseudo-terminal driver (in the kernel) knows (the terminal emulator (like xterm) sends it some characters upon some keypress and the terminal driver may send back some characters for echo, but xterm can’t know whether those are from the local echo or from what the application output to the slave side of the pseudo terminal).

And then, there’s the other mode where the terminal driver is told not to echo, but the application this time outputs something. The application (like those using readline like gdb, bash…) may send that on its stdout or stderr which is going to be difficult to differentiate from something that it outputs for other things than echoing back the user input.

Then to differentiate an application’s stdout from its stderr, there are several approaches.

Many of them involve redirecting the commands stdout and stderr to pipes and those pipes read by an application to colour it. There are two problems with that:

  • Once stdout is no longer a terminal (like a pipe instead), many application tend to adapt their behavior to start buffering their output which means that output is going to be displayed in big chunks.
  • Even if it’s the same process that processes the two pipes, there’s no guarantee that the order the text written by the application on stdout and stderr will be preserved, as the reading process can’t know (if there’s something to be read from both) whether to start reading from the “stdout” pipe or the “stderr” pipe.

Another approach is to modify the application so that it does colour its stdout and stdin. It is often not possible or realistic to do.

Then a trick (for dynamically linked applications) can be to hijack (using $LD_PRELOAD as in sickill’s answer) the outputting functions called by the application to output something and include code in them that sets the foreground colour based on whether they’re meant to output something on stderr or stdout. However, that means hijacking every possible function from the C library and any other library that does a write(2) syscall directly called by the application that may potentially end up writing something on stdout or stderr (printf, puts, perror…), and even then, that may modify its behavior.

Another approach could be to use PTRACE tricks as strace or gdb do to hook ourself every time the write(2) system call is called and set the output colour based on whether the write(2) is on file descriptor 1 or 2.

However, that’s quite a big thing to do.

A trick which I’ve just been playing with is to hijack strace itself (which does the dirty work of hooking itself before every system call) using LD_PRELOAD, to tell it to change the output colour based on whether it has detected a write(2) on fd 1 or 2.

From looking at strace source code, we can see that all it outputs is done via the vfprintf function. All we need to do is to hijack that function.

The LD_PRELOAD wrapper would look like:

#define _GNU_SOURCE
#include <dlfcn.h>
#include <string.h>
#include <stdio.h>
#include <stdarg.h>
#include <unistd.h>

int vfprintf(FILE *outf, const char *fmt, va_list ap)
{
  static int (*orig_vfprintf) (FILE*, const char *, va_list) = 0;
  static int c = 0;
  va_list ap_orig;
  va_copy(ap_orig, ap);
  if (!orig_vfprintf) {
    orig_vfprintf = (int (*) (FILE*, const char *, va_list))
      dlsym (RTLD_NEXT, "vfprintf");
  }

  if (strcmp(fmt, "%ld, ") == 0) {
    int fd = va_arg(ap, long);
    switch (fd) {
    case 2:
      write(2, "e[31m", 5);
      c = 1;
      break;
    case 1:
      write(2, "e[32m", 5);
      c = 1;
      break;
    }
  } else if (strcmp(fmt, ") ") == 0) {
    if (c) write(2, "e[m", 3);
    c = 0;
  }
  return orig_vfprintf(outf, fmt, ap_orig);
}

Then, we compile it with:

cc -Wall -fpic -shared -o wrap.so wrap.c -ldl

And use it as:

LD_PRELOAD=/path/to/wrap.so strace -qqf -a0 -s0 -o /dev/null 
  -e write -e status=successful -P "$(tty)" 
  env -u LD_PRELOAD some-cmd

You’ll notice how if you replace some-cmd with bash, the bash prompt and what you type appears in red (stderr) while with zsh it appears in black (because zsh dups stderr onto a new fd to display its prompt and echo).

It does appear to work surprisingly well even for applications that you’d expect not (like those that do use colours).

The colouring mode is output on strace‘s stderr which is assumed to be the terminal. With -P "$(tty)", we’re avoid doing it for writes that don’t go to the terminal like when stdout/stderr have been redirected.

That solution has its limitations:

  • Those inherent to strace: performance issues, you can’t run other PTRACE commands like strace or gdb in it, or setuid/setgid issues
  • It’s colouring based on the writes on stdout/stderr of each individual process. So for instance, in sh -c 'echo error >&2', error would be green because echo outputs it on its stdout (which sh redirected to sh’s stderr, but all strace sees is a write(1, "errorn", 6)).

October 2021 edit. That wrapper no longer works here on a Debian unstable with strace 5.10, glibc 2.32, gcc 10.30.0, as the function to wrap now needs to be __vfprintf_chk and the formats to look for have changed. The wrapper needs to be changed to:

#define _GNU_SOURCE
#include <dlfcn.h>
#include <string.h>
#include <stdio.h>
#include <stdarg.h>
#include <unistd.h>

int __vfprintf_chk(FILE *outf, int x, const char *fmt, va_list ap)
{
  static int (*orig_vfprintf) (FILE*, int, const char *, va_list) = 0;
  static int c = 0;
  va_list ap_orig;
  va_copy(ap_orig, ap);
  if (!orig_vfprintf) {
    orig_vfprintf = (int (*) (FILE*, int, const char *, va_list))
      dlsym (RTLD_NEXT, "__vfprintf_chk");
  }

  if (strcmp(fmt, "%d") == 0) {
    int fd = va_arg(ap, long);
    switch (fd) {
    case 2:
      write(2, "e[31m", 5);
      c = 1;
      break;
    case 1:
      write(2, "e[32m", 5);
      c = 1;
      break;
    }
  } else if (strcmp(fmt, "= %lu") == 0) {
    if (c) write(2, "e[m", 3);
    c = 0;
  }
  return orig_vfprintf(outf, x, fmt, ap_orig);
}

It does show this kind of approach is a bit brittle as it uses an undocumented, unstable API (not really an API).

Related Solutions

Joining bash arguments into single string with spaces

[*] I believe that this does what you want. It will put all the arguments in one string, separated by spaces, with single quotes around all: str="'$*'" $* produces all the scripts arguments separated by the first character of $IFS which, by default, is a space....

AddTransient, AddScoped and AddSingleton Services Differences

TL;DR Transient objects are always different; a new instance is provided to every controller and every service. Scoped objects are the same within a request, but different across different requests. Singleton objects are the same for every object and every...

How to download package not install it with apt-get command?

Use --download-only: sudo apt-get install --download-only pppoe This will download pppoe and any dependencies you need, and place them in /var/cache/apt/archives. That way a subsequent apt-get install pppoe will be able to complete without any extra downloads....

What defines the maximum size for a command single argument?

Answers Definitely not a bug. The parameter which defines the maximum size for one argument is MAX_ARG_STRLEN. There is no documentation for this parameter other than the comments in binfmts.h: /* * These are the maximum length and maximum number of strings...

Bulk rename, change prefix

I'd say the simplest it to just use the rename command which is common on many Linux distributions. There are two common versions of this command so check its man page to find which one you have: ## rename from Perl (common in Debian systems -- Ubuntu, Mint,...

Output from ls has newlines but displays on a single line. Why?

When you pipe the output, ls acts differently. This fact is hidden away in the info documentation: If standard output is a terminal, the output is in columns (sorted vertically) and control characters are output as question marks; otherwise, the output is...

mv: Move file only if destination does not exist

mv -vn file1 file2. This command will do what you want. You can skip -v if you want. -v makes it verbose - mv will tell you that it moved file if it moves it(useful, since there is possibility that file will not be moved) -n moves only if file2 does not exist....

Is it possible to store and query JSON in SQLite?

SQLite 3.9 introduced a new extension (JSON1) that allows you to easily work with JSON data . Also, it introduced support for indexes on expressions, which (in my understanding) should allow you to define indexes on your JSON data as well. PostgreSQL has some...

Combining tail && journalctl

You could use: journalctl -u service-name -f -f, --follow Show only the most recent journal entries, and continuously print new entries as they are appended to the journal. Here I've added "service-name" to distinguish this answer from others; you substitute...

how can shellshock be exploited over SSH?

One example where this can be exploited is on servers with an authorized_keys forced command. When adding an entry to ~/.ssh/authorized_keys, you can prefix the line with command="foo" to force foo to be run any time that ssh public key is used. With this...

Why doesn’t the tilde (~) expand inside double quotes?

The reason, because inside double quotes, tilde ~ has no special meaning, it's treated as literal. POSIX defines Double-Quotes as: Enclosing characters in double-quotes ( "" ) shall preserve the literal value of all characters within the double-quotes, with the...

What is GNU Info for?

GNU Info was designed to offer documentation that was comprehensive, hyperlinked, and possible to output to multiple formats. Man pages were available, and they were great at providing printed output. However, they were designed such that each man page had a...

Set systemd service to execute after fstab mount

a CIFS network location is mounted via /etc/fstab to /mnt/ on boot-up. No, it is not. Get this right, and the rest falls into place naturally. The mount is handled by a (generated) systemd mount unit that will be named something like mnt-wibble.mount. You can...

Merge two video clips into one, placing them next to each other

To be honest, using the accepted answer resulted in a lot of dropped frames for me. However, using the hstack filter_complex produced perfectly fluid output: ffmpeg -i left.mp4 -i right.mp4 -filter_complex hstack output.mp4 ffmpeg -i input1.mp4 -i input2.mp4...

How portable are /dev/stdin, /dev/stdout and /dev/stderr?

It's been available on Linux back into its prehistory. It is not POSIX, although many actual shells (including AT&T ksh and bash) will simulate it if it's not present in the OS; note that this simulation only works at the shell level (i.e. redirection or...

How can I increase the number of inodes in an ext4 filesystem?

It seems that you have a lot more files than normal expectation. I don't know whether there is a solution to change the inode table size dynamically. I'm afraid that you need to back-up your data, and create new filesystem, and restore your data. To create new...

Why doesn’t cp have a progress bar like wget?

The tradition in unix tools is to display messages only if something goes wrong. I think this is both for design and practical reasons. The design is intended to make it obvious when something goes wrong: you get an error message, and it's not drowned in...