Home » Understanding IFS

Understanding IFS

Solutons:


  1. Yes, they are the same.
  2. Yes.
  3. In bash, and similar shells, you could do something like IFS=$' tn'. Otherwise, you could insert the literal control codes by using [space] CTRL+V
    CTRL+V [enter]
    . If you are planning to do this, however, it’s better to use another variable to temporarily store the old IFS value, and then restore it afterwards (or temporarily override it for one command by using the var=foo command syntax).
    • The first code snippet will put the entire line read, verbatim, into $line, as there are no field separators to perform word splitting for. Bear in mind however that since many shells use cstrings to store strings, the first instance of a NUL may still cause the appearance of it being prematurely terminated.
    • The second code snippet may not put an exact copy of the input into $line. For example, if there are multiple consecutive field separators, they will be made into a single instance of the first element. This is often recognised as loss of surrounding whitespace.
    • The third code snippet will do the same as the second, except it will only split on a space (not the usual space, tab, or newline).

Q1: Yes. “Field splitting” and “word splitting” are two terms for the same concept.

Q2: Yes. If IFS is unset (i.e. after unset IFS), it is equivalent IFS being set to $' tn' (a space, a tab and a newline). If IFS is set to an empty value (that’s what “null” means here) (i.e. after IFS= or IFS='' or IFS=""), no field splitting is performed at all (and $*, which normally uses the first character of $IFS, uses a space character).

Q3: If you want to have the default IFS behavior, you can use unset IFS. If you want to set IFS explicitly to this default value, you can put the literal characters space, tab, newline in single quotes. In ksh93, bash or zsh, you can use IFS=$' tn'. Portably, if you want to avoid having a literal tab character in your source file, you can use

IFS=" $(echo t | tr t \t)
"

Q4: With IFS set to an empty value, read -r line sets line to the whole line except its terminating newline. With IFS=" ", spaces at the beginning and at the end of the line are trimmed. With the default value of IFS, tabs and spaces are trimmed.

Q1. Field splitting.

Is field splitting the same as word splitting ?

Yes, both point to the same idea.

Q2: When is IFS null?.

Is setting IFS='' the same as null, the same as an empty string too?

Yes, all three mean the same: No field/word splitting shall be performed.
Also, this affects printing fields (as with echo "$*") all fields will be concatenated together with no space.

Q3: (part a) Unset IFS.

In the POSIX specification, I read the following:

If IFS is not set, the shell shall behave as if the value of IFS is <space><tab><newline>.

Which is exactly equivalent to:

With an unset IFS, the shell shall behave as if IFS is default.

That means that the ‘Field splitting’ will be exactly the same with a default IFS value, or unset.
That does NOT mean that IFS will work the same way in all conditions.
Being more specific, executing OldIFS=$IFS will set the var OldIFS to null, not the default. And trying to set IFS back, as this, IFS=OldIFS will set IFS to null, not keep it unset as it were before. Watch out !!.

Q3: (part b) Restore IFS.

How could I restore the value of IFS to default.
Say I want to restore the default value of IFS. How do I do that? (more specifically, how do I refer to <tab> and <newline>?)

For zsh, ksh, and bash (AFAIK), IFS could be set to the default value as:

IFS=$' tn'        # works with zsh, ksh, bash.

Done, you need to read nothing else.

But if you need to re-set IFS for sh, it may become complex.

Let’s take a look from easiest to complete with no drawbacks (except complexity).

1.- Unset IFS.

We could just unset IFS (Read Q3 part a, above.).

2.- Swap chars.

As a workaround, swapping the value of tab and newline makes it simpler to set the value of IFS, and then it works in a equivalent way.

Set IFS to <space><newline><tab>:

sh -c 'IFS=$(echo " nt"); printf "%s" "$IFS"|xxd'      # Works.

3.- A simple? solution:

If there are child scripts that need IFS correctly set, you could always manually write:

IFS='   
'

Where the sequence manually typed was: IFS='spacetabnewline', sequence which has actually been correctly typed above (If you need to confirm, edit this answer). But a copy/paste from your browser will break because the browser will squeeze/hide the whitespace. It makes it difficult to share the code as written above.

4.- Complete solution.

To write code that can be safely copied usually involves unambiguous printable escapes.

We need some code that “produces” the expected value. But, even if conceptually correct, this code will NOT set a trailing n:

sh -c 'IFS=$(echo " tn"); printf "%s" "$IFS"|xxd'      # wrong.

That happens because, under most shells, all trailing newlines of $(...) or `...` command substitutions are removed on expansion.

We need to use a trick for sh:

sh -c 'IFS="$(printf " tnx")"; IFS="${IFS%x}"; printf "$IFS"|xxd'  # Correct.

An alternative way may be to set IFS as an environment value from bash (for example) and then call sh (the versions of it that accept IFS to be set via the environment), as this:

env IFS=$' tn' sh -c 'printf "%s" "$IFS"|xxd'

In short, sh makes resetting IFS to default quite an odd adventure.

Q4: In actual code:

Finally, how would this code:

while IFS= read -r line
do
    echo $line
done < /path_to_text_file

behave if we we change the first line to

while read -r line # Use the default IFS value

or to:

while IFS=' ' read -r line

First: I do not know if the echo $line (with the var NOT quoted) is there on porpouse, or not.
It introduces a second level of ‘field splitting’ that read does not have.
So I’ll answer both. 🙂

With this code (so you could confirm). You’ll need the useful xxd:

#!/bin/ksh
# Correctly set IFS as described above.
defIFS="$(printf " tnx")"; defIFS="${defIFS%x}";
IFS="$defIFS"
printf "IFS value: "
printf "%s" "$IFS"| xxd -p

a="   bar   baz   quz   "; l="${#a}"
printf "var value          : %${l}s-" "$a" ; printf "%sn" "$a" | xxd -p

printf "%sn" "$a" | while IFS='x' read -r line; do
    printf "IFS --x--          : %${l}s-" "$line" ;
    printf "%s" "$line" |xxd -p; done;

printf 'Values      quoted :n' ""  # With values quoted:
printf "%sn" "$a" | while IFS='' read -r line; do
    printf "IFS null    quoted : %${l}s-" "$line" ;
    printf "%s" "$line" |xxd -p; done;

printf "%sn" "$a" | while IFS="$defIFS" read -r line; do
    printf "IFS default quoted : %${l}s-" "$line" ;
    printf "%s" "$line" |xxd -p; done;

unset IFS; printf "%sn" "$a" | while read -r line; do
    printf "IFS unset   quoted : %${l}s-" "$line" ;
    printf "%s" "$line" |xxd -p; done;
    IFS="$defIFS"   # set IFS back to default.

printf "%sn" "$a" | while IFS=' ' read -r line; do
    printf "IFS space   quoted : %${l}s-" "$line" ;
    printf "%s" "$line" |xxd -p; done;

printf '%sn' "Values unquoted :"   # Now with values unquoted:
printf "%sn" "$a" | while IFS='x' read -r line; do
    printf "IFS --x-- unquoted : "
    printf "%s, " $line; printf "%s," $line |xxd -p; done

printf "%sn" "$a" | while IFS='' read -r line; do
    printf "IFS null  unquoted : ";
    printf "%s, " $line; printf "%s," $line |xxd -p; done

printf "%sn" "$a" | while IFS="$defIFS" read -r line; do
    printf "IFS defau unquoted : ";
    printf "%s, " $line; printf "%s," $line |xxd -p; done

unset IFS; printf "%sn" "$a" | while read -r line; do
    printf "IFS unset unquoted : ";
    printf "%s, " $line; printf "%s," $line |xxd -p; done
    IFS="$defIFS"   # set IFS back to default.

printf "%sn" "$a" | while IFS=' ' read -r line; do
    printf "IFS space unquoted : ";
    printf "%s, " $line; printf "%s," $line |xxd -p; done

I get:

$ ./stackexchange-Understanding-IFS.sh
IFS value: 20090a
var value          :    bar   baz   quz   -20202062617220202062617a20202071757a2020200a
IFS --x--          :    bar   baz   quz   -20202062617220202062617a20202071757a202020
Values      quoted :
IFS null    quoted :    bar   baz   quz   -20202062617220202062617a20202071757a202020
IFS default quoted :       bar   baz   quz-62617220202062617a20202071757a
IFS unset   quoted :       bar   baz   quz-62617220202062617a20202071757a
IFS space   quoted :       bar   baz   quz-62617220202062617a20202071757a
Values unquoted :
IFS --x-- unquoted : bar, baz, quz, 6261722c62617a2c71757a2c
IFS null  unquoted : bar, baz, quz, 6261722c62617a2c71757a2c
IFS defau unquoted : bar, baz, quz, 6261722c62617a2c71757a2c
IFS unset unquoted : bar, baz, quz, 6261722c62617a2c71757a2c
IFS space unquoted : bar, baz, quz, 6261722c62617a2c71757a2c

The first value is just the correct value of IFS='spacetabnewline'

Next line is all the hex values that the var $a has, and a newline ‘0a’ at the end as it is going to be given to each read command.

The next line, for which IFS is null, does not perform any ‘field spliting’, but the newline is removed (as expected).

The next three lines, as IFS contains an space, remove the initial spaces and set the var line to the balance remaining.

The last four lines shows what an unquoted variable will do. The values will be split on the (several) spaces and will be printed as: bar,baz,qux,

Related Solutions

Joining bash arguments into single string with spaces

[*] I believe that this does what you want. It will put all the arguments in one string, separated by spaces, with single quotes around all: str="'$*'" $* produces all the scripts arguments separated by the first character of $IFS which, by default, is a space....

AddTransient, AddScoped and AddSingleton Services Differences

TL;DR Transient objects are always different; a new instance is provided to every controller and every service. Scoped objects are the same within a request, but different across different requests. Singleton objects are the same for every object and every...

How to download package not install it with apt-get command?

Use --download-only: sudo apt-get install --download-only pppoe This will download pppoe and any dependencies you need, and place them in /var/cache/apt/archives. That way a subsequent apt-get install pppoe will be able to complete without any extra downloads....

What defines the maximum size for a command single argument?

Answers Definitely not a bug. The parameter which defines the maximum size for one argument is MAX_ARG_STRLEN. There is no documentation for this parameter other than the comments in binfmts.h: /* * These are the maximum length and maximum number of strings...

Bulk rename, change prefix

I'd say the simplest it to just use the rename command which is common on many Linux distributions. There are two common versions of this command so check its man page to find which one you have: ## rename from Perl (common in Debian systems -- Ubuntu, Mint,...

Output from ls has newlines but displays on a single line. Why?

When you pipe the output, ls acts differently. This fact is hidden away in the info documentation: If standard output is a terminal, the output is in columns (sorted vertically) and control characters are output as question marks; otherwise, the output is...

mv: Move file only if destination does not exist

mv -vn file1 file2. This command will do what you want. You can skip -v if you want. -v makes it verbose - mv will tell you that it moved file if it moves it(useful, since there is possibility that file will not be moved) -n moves only if file2 does not exist....

Is it possible to store and query JSON in SQLite?

SQLite 3.9 introduced a new extension (JSON1) that allows you to easily work with JSON data . Also, it introduced support for indexes on expressions, which (in my understanding) should allow you to define indexes on your JSON data as well. PostgreSQL has some...

Combining tail && journalctl

You could use: journalctl -u service-name -f -f, --follow Show only the most recent journal entries, and continuously print new entries as they are appended to the journal. Here I've added "service-name" to distinguish this answer from others; you substitute...

how can shellshock be exploited over SSH?

One example where this can be exploited is on servers with an authorized_keys forced command. When adding an entry to ~/.ssh/authorized_keys, you can prefix the line with command="foo" to force foo to be run any time that ssh public key is used. With this...

Why doesn’t the tilde (~) expand inside double quotes?

The reason, because inside double quotes, tilde ~ has no special meaning, it's treated as literal. POSIX defines Double-Quotes as: Enclosing characters in double-quotes ( "" ) shall preserve the literal value of all characters within the double-quotes, with the...

What is GNU Info for?

GNU Info was designed to offer documentation that was comprehensive, hyperlinked, and possible to output to multiple formats. Man pages were available, and they were great at providing printed output. However, they were designed such that each man page had a...

Set systemd service to execute after fstab mount

a CIFS network location is mounted via /etc/fstab to /mnt/ on boot-up. No, it is not. Get this right, and the rest falls into place naturally. The mount is handled by a (generated) systemd mount unit that will be named something like mnt-wibble.mount. You can...

Merge two video clips into one, placing them next to each other

To be honest, using the accepted answer resulted in a lot of dropped frames for me. However, using the hstack filter_complex produced perfectly fluid output: ffmpeg -i left.mp4 -i right.mp4 -filter_complex hstack output.mp4 ffmpeg -i input1.mp4 -i input2.mp4...

How portable are /dev/stdin, /dev/stdout and /dev/stderr?

It's been available on Linux back into its prehistory. It is not POSIX, although many actual shells (including AT&T ksh and bash) will simulate it if it's not present in the OS; note that this simulation only works at the shell level (i.e. redirection or...

How can I increase the number of inodes in an ext4 filesystem?

It seems that you have a lot more files than normal expectation. I don't know whether there is a solution to change the inode table size dynamically. I'm afraid that you need to back-up your data, and create new filesystem, and restore your data. To create new...

Why doesn’t cp have a progress bar like wget?

The tradition in unix tools is to display messages only if something goes wrong. I think this is both for design and practical reasons. The design is intended to make it obvious when something goes wrong: you get an error message, and it's not drowned in...