Linux utilities that every developer should know

Introduction

The job of each developer is the constant occurrence of problems and the search for their solutions. The speed of solving problem depends on the knowledge, skills and abilities to work in the development environment. However, we should not forget about the ability to search for those solutions. When you click on the first link in the Google search results, we can get a valid solution in 90% of cases. Here we must admit with confidence that almost all the problems have already been solved before. However, you must agree that it is cool if you could use it immediately and not search.
In our daily work, we use a lot of different tools: from searching for a regular file to the creation of a file system. This post describes the “must-have” tools that seem trivial to someone, and someone will say: “And how did I live without it before?”.
I divided these tools into groups by generalized problems.
Before we start, it is worth mentioning an incredible thing that makes life easier for every developer: a zsh command shell in combination with oh-my-zsh. The shell zsh itself is very convenient, as it provides many features like typo correction, programmable auto-completion, the ability to quickly navigate both directories and history, and much more. Oh-my-zsh is a way to make zsh even better. Oh-my-zsh adds a huge set of aliases for frequently used commands to zsh, but we will not dwell on it anymore since its consideration requires a separate article.
So, let’s move on to our selection of linux-utilites.

Work with files

Basic utilities for working with the file system every Linux user should know. Here are some useful commands.
Move (rename) file:

mv /path/to/source /path/to/destination

Copy file to directory:

cp /path/to/source /path/to/destination_directory

Copy the directory with all the files inside it (the content of the directory source_directory will be copied to destination_directory):

cp -r /path/to/source_directory /path/to/destination_directory

Copy several files:

cp /path/to/file1 /path/to/file2 /path/to/destination_directory

Delete a specific file:

rm /path/to/file

Delete files with extension .txt:

rm -rf *.txt

Delete directory with all files inside (flag -f to delete without confirmation):

rm -rf /path/to/directory

Change file owner:

chown user:group /path/to/file

Change the owner of the directory and all files inside it:

chown -R user:group /path/to/directory

Give all users the permission to read and write to the directory and all files inside it:

chmod -R a+rw /path/to/directory

Make the file permission the same as the other file (you can use it in chown):

chmod --reference=/path/to/source /path/to/destination

Command chmod, as you can see from the example above, allows you to use a fairly simple syntax in which to use and memorize:
• users – u (owner), g (group), o (others), a (all);
• permissions – r (read), w (write), x (execute).

Work with disk drives

More advanced set includes utilities mount and dd.
Using the mount, you can “attach” the file system on a block device to the root file system.
Utility dd (as it is called the “disk destroyer”) provides copying by blocks. You need to use it with extreme caution, as an ill-considered launch can permanently destroy the data on the device.

Consider a few examples.

Mount disk /dev/sdb1 with mount point /mnt/usb (directory /mnt/usb should already exist):

mount /dev/sdb1 /mnt/usb

Mount device with file system ext4 only for reading:

mount /dev/sdb1 /mnt/usb -t ext4 -o ro -o noload

Unmount mount point:

umount /mnt/usb

Force unmount file system:

umount -f /mnt/usb

Copy blocks of one device to another:

dd if=/path/to/input of=/path/to/output

Write the image to the device:

dd bs=4M if=/path/to/linux.iso of=/dev/sdx

Reading logs

There are plenty of ways to “look” at a file from different angles. Using head and tail we can read the beginning and end of the file, respectively. Let us proceed directly to the examples of the use of these commands.
Display the first 20 lines from the file:

head -n 20 access.log

Display the last 30 lines of the file:

tail -n 30 error.log

Launch tail in the tracking mode of the new line:

tail -f access.log

These commands allow you to view the contents of the file partially. In the case of the line output file content is better to use the utility. Less, which has many more shortcuts to navigate and search. Consider examples.
Open file for paginated output::

less access.log

Open file with line number displaying:

less -N access.log

Shortcuts:
• SHIFT+G – go to the end of the file;
• g – go to the beginning of the file;
• / template – find the following match;
• ? template – find the previous match;
• n – go to the next match;
• SHIFT+N – go to the previous match.

File system search

The need to search for information in the file system occurs quite often (search for configuration files, search for the file that generated the error message in the logs etc), and utilities such as find and grep. Let’s turn to examples of using the utility find.
Find files with name netdata.conf:

find -name 'netdata.conf'

Find all files with extension .conf:

find / -name '*.conf'

Find all files with name apache2:

find / -type f -name 'apache2'

Find all directories with a name nginx:

1
find / -type d -name 'nginx'

Find all files larger than 100MB in your home directory:

find ~ -size +100M

Find in the home directory all files with a size less than 100MB:

find ~ -size -100M

Find all empty files in the home directory:

find ~ -empty

Delete all empty files in the home directory ({} replaced by the file name):

find ~ -empty -exec rm -rf {} \;

Searching for a file by meta tags is fine, but what if you need to dig deeper? For these purposes, you can use the grep utility – search and filter by the pattern.
Find word Forbidden in file error.log:

grep 'Forbidden' error.log


Find word forbidden in file error.log (case-insensitive search):

grep -i 'forbidden' error.log

Display the number of matches found:

grep -c 'Forbidden' error.log

Display an additional 2 lines after the match:

grep -i -A2 'forbidden' error.log

Display an additional 2 lines before the match:

grep -i -B2 'forbidden' error.log

Display an additional 2 lines before and after the match:

grep -i -C2 'forbidden' error.log

Find a phrase Access denied in all files in the folder ~/.pm2/logs:

grep -i -r 'access denied' ~/.pm2/logs

Dealing with processes

Quite often it is necessary to manage processes or simply to obtain information about all or a specific process. Consider examples of the use of such commands.
Display a list of all processes:

ps aux

Display only node processes:

ps aux | grep node

Display processes as a tree, show only pid and command:

ps -e -o pid,args --forest

Send signal SIGTERM (sent by default) to the process with pid 8888:

kill -SIGTERM 8888

Send SIGKILL (force terminate the process) to the process with pid 8888:

kill -9 8888

Stop all processes named node:

killall node

Display processes whose parent is the process with pid 3607:

pgrep -P 3607

Display the processes that opened the file /etc/hosts:

lsof /etc/hosts

Find the process which took port 80:

lsof -i :80

Running processes in background

To start the process in the background, it is enough to add an ampersand (&) at the end of the command, but this option has a drawback: if the shell session ends – all its background processes will also stop. So, it is better to use screen or even better – the service, but we will not consider the services. Let us turn to examples.
Run the process in the background:

ping google.com &

Run several processes in the background:

ping goole.com & nmap 192.168.1.* &

Display a list of background processes:

jobs -l

Get access to the process (put it into priority mode):

%1

Bring the process back to background:
CTRL+Z

%1 &

A little hint for background processes: to stop the process, put it into priority mode and click CTRL+C or use the command kill.
Using screen saves us from the problem that arises with &. In the screen session, you can start any process and then detach session. Consider a few commands.
Start a new screen session:

screen

Transfer session to detached mode:
CTRL+A+D
Start the process in a new session in detached mode:

screen -d -m ping google.com

View the list of sessions:

screen -ls

Attach screen session:

screen -R [session id]

System resources utilization

The system slows down, and there is not enough space … how to identify the source of all these problems? There are several useful tools to identify the source of such problems. That gives a starting point for further investigation.
Launch interactive process monitor:

top

htop

Run I/O monitor:

iotop

Display space usage:

du /path/to/directory

Analyze space usage:

ncdu /path/to/directory

View system information

To understand the system parameters, you can use the mass of available utilities that provide the ability to obtain information about the characteristics of the system.
Display OS name and version:

lsb_release -a

See the full list of all devices:

lshw

See processor information:

lscpu

Display RAM Information:

free -h

See information about all mount points:

df -h

Display information about all available block devices:

lsblk

Deal with the network

In this section, we consider the commands for working with the network – usage of the commands curl, tcpdump and nmap. Utility curl is widely used if you need something to “pull up” from the network, tcpdump is used for packet analysis, and nmap – is a network scanner that allows you to identify systems and services running on them. Let’s go directly to the examples.
Run HEAD request (get headers only):

curl -I http://google.com

Run POST request with data sending:

curl -d 'first_name=John&last_name=Doe' http://google.com

Send JSON to server:

curl -d '{"name":"John"}' -H 'content-type: application/json' http://google.com

Download file (similar to using wget):

curl -O https://www.google.com/images/branding/googlelogo/2x/googlelogo_light_color_272x92dp.png

Interesting use case.
Command curl is a reiteration of browser’s requests. In Google Chrome’s Developers tools -> in the tab “Network” there is an opportunity to copy curl command for the selected query (find the desired query > Right mouse button click > Copy > Copy as cURL). After that, the copied command can be inserted into the terminal and execute the request again. The provided command contains all the headers, parameters and the body of the request.
Let us turn back to examples of the use of the following utilities.
Track packets that were sent from the local machine:

tcpdump src 127.0.0.1

Track packets that came to the local machine through a specific network interface:

tcpdump dst 127.0.0.1 -i eth0

Display a list of network interfaces:

tcpdump --list-interfaces

Track packages that are gone from the local machine to google.com:

tcpdump src 127.0.0.1 and dst google.com -n

Display captured packets in ASCII:

tcpdump dst google.com and port 80 -A

Track packets to a specific IP and port:

tcpdump dst google.com and port 443 -n

Scan a specific server:

nmap -sP 217.160.0.201

Scan the local network:

nmap -sP 192.168.1.*

Try to determine the server OS:

nmap -O 192.168.1.8

Scan server ports (use –sV to determine the version of the service):

nmap -Pn 192.168.1.8

Run a quick scan (scanning of the standard services):

nmap -F 192.168.1.8

Scan specific ports (use –open to display only open ports):

nmap -Pn -p 80,443 192.168.1.8

Scan port combination (U – UDP port, T – TCP port, 21-25 – port range):

nmap -p U:53,111,137,T:21-25,80,139,8080 192.168.1.1

Scan 10 top ports (ssh, ftp, http …):

nmap -Pn --top-ports 10 192.168.1.8

Other utilities

In this section, we will look at useful utilities that are outside categories, but they are used quite often. Immediately proceed to the examples.
Start the process with an update every 500 ms:

watch -n0.5 ls -laS

Reset terminal:

reset

Display the calendar for the current year:

cal -j

Display calendar for June 2021:

cal -j 6 2021

Display information about the current user:

id

Display current user name:

whoami

Convert string to base64:

echo Hello | base64

Decode base64 string:

echo "SGVsbG8K" | base64 -d

Format json:

echo '{"status":1}' | jq

curl https://opinionated-quotes-api.gigalixirapp.com/v1/quotes | jq

Conclusion

In this article we reviewed a selection of Linux utilities which we use in our daily work.
And we have some bonus for those have read till the line – you can download this utilities list in printable version.
Of course, this list is not limited to this set, there are plenty of others, but the article is already a big one.
Finally, I want to say – read the manual, there are descriptions of all possible parameters and examples of use. And remember – search and you shall find!