How to Read Files Line by Line in Bash

Bash is one of the most popular and poweful shell interpreters in Linux and other Unix-based systems. It comes packed with a wide variety of tools and features for simple automation to complex scripts.

One of the most common tasks in scripting that you might come across is reading files. Specifically, reading files line by line. For example, you might need to parse logs, or read data from a hidden file.

Lucky for us, Bash offers some very cool features and tools that we can use to read files line by line.

In this tutorial, we will explore the various methods and techniques that we can use to read files line by line using Bash scripting.

Method 1 - Using While Loop

One of the most straightforward ways to read a file line by line in Bash is by using a while loop. This method is simple and efficient for small to medium-sized files.

Suppose we have a file called server.log, we can read it line by line using a while loop as shown:

#!/bin/bash
file_path="server.log"

# Check if the file exists
if [ ! -f "$file_path" ]; then
    echo "File not found: $file_path"
    exit 1
fi

# Read the file line by line using a while loop
while IFS= read -r line; do
    echo "Line: $line"
done < "$file_path"

In the example script above, we start by invoking a while loop that reads each line from the specified file path.

We also set the Internal Field Seperator to an empty string to ensure that we preserve any leading or trailing whitepsace characters.

NOTE: We also use the -r option in the read command to prevent backslashes from escaping characters.

Method 2 - Using Read without While Loop

We can also remove the while loop and just take advantage of the read command to recursively read a file line by line.

Take for example the script shown below:

file_path="server.log"
# Check if the file exists
if [ ! -f "$file_path" ]; then
    echo "File not found: $file_path"
    exit 1
fi
# Read and process each line in a loop
while read -r line; do
    echo "Line: $line"
done < "$file_path"

You will notice that this technique is pretty similar to the while loop only without the loop.

Method 3 - Using mapfile (Bash 4.0+)

Starting from Bash version 4.0, we can use the mapfile command to read lines from a file into an array. This method is especially useful for large files as it is more memory-efficient compared to using while loops.

#!/bin/bash

file_path="server.log"

# Check if the file exists
if [ ! -f "$file_path" ]; then
    echo "File not found: $file_path"
    exit 1
fi

# Read the file into an array using mapfile
mapfile -t lines < "$file_path"

# Process each line in the array
for line in "${lines[@]}"; do
    echo "Line: $line"
done

In this case, we use the mapfile command to read the file into the lines array. We can then access each line as an element of the array.

Method 4 - Using sed Command

We can also use the sed (stream editor) utility to read a file line by line and perform text processing as shown in the example below:

#!/bin/bash
file_path="server.log"

# Check if the file exists
if [ ! -f "$file_path" ]; then
    echo "File not found: $file_path"
    exit 1
fi

# Read the file line by line using sed
sed 's/^/Line: /' "$file_path"

This should basically read the file line by line and append the string Line: to each entry.

Method 5 - Using awk Command

As you can guess, where there is sed, awk can work as well.

Since awk is very versatile when it comes to text processing, we can use it to read files line by line and perform various operations on them.

The following shows how to do just that while printing the contents of the file.

file_path="example.txt"

# Check if the file exists
if [ ! -f "$file_path" ]; then
    echo "File not found: $file_path"
    exit 1
fi

# Read the file line by line using awk
awk '{print "Line:", $0}' "$file_path"

In this script, awk processes each line and prints it with a prefix “Line:”.

Conclusion

Reading files line by line in Bash is a fundamental task when working with text files. In this guide, we’ve explored various methods and techniques, ranging from simple while loops and read commands to more advanced tools like mapfile.