The job of each developer is the constant occurrence of problems and the search for their solutions. The speed of solving problem depends on the knowledge, skills and abilities to work in the development environment. However, we should not forget about the ability to search for those solutions. When you click on the first link in the Google search results, we can get a valid solution in 90% of cases. Here we must admit with confidence that almost all the problems have already been solved before. However, you must agree that it is cool if you could use it immediately and not search. In our daily work, we use a lot of different tools: from searching for a regular file to the creation of a file system. This post describes the “must-have” tools that seem trivial to someone, and someone will say: “And how did I live without it before?”. I divided these tools into groups by generalized problems. Before we start, it is worth mentioning an incredible thing that makes life easier for every developer: a zsh command shell in combination with oh-my-zsh. The shell zsh itself is very convenient, as it provides many features like typo correction, programmable auto-completion, the ability to quickly navigate both directories and history, and much more. Oh-my-zsh is a way to make zsh even better. Oh-my-zsh adds a huge set of aliases for frequently used commands to zsh, but we will not dwell on it anymore since its consideration requires a separate article. So, let’s move on to our selection of linux-utilites.
Basic utilities for working with the file system every Linux user should know. Here are some useful commands. Move (rename) file:
mv /path/to/source /path/to/destination
Copy file to directory:
cp /path/to/source /path/to/destination_directory
Copy the directory with all the files inside it (the content of the directory source_directory will be copied to destination_directory):
cp -r /path/to/source_directory /path/to/destination_directory
Copy several files:
cp /path/to/file1 /path/to/file2 /path/to/destination_directory
Delete a specific file:
rm /path/to/file
Delete files with extension .txt:
rm -rf *.txt
Delete directory with all files inside (flag -f to delete without confirmation):
rm -rf /path/to/directory
Change file owner:
chown user:group /path/to/file
Change the owner of the directory and all files inside it:
chown -R user:group /path/to/directory
Give all users the permission to read and write to the directory and all files inside it:
chmod -R a+rw /path/to/directory
Make the file permission the same as the other file (you can use it in chown):
chmod --reference=/path/to/source /path/to/destination
Command chmod, as you can see from the example above, allows you to use a fairly simple syntax in which to use and memorize: • users – u (owner), g (group), o (others), a (all); • permissions – r (read), w (write), x (execute).
More advanced set includes utilities mount and dd. Using the mount, you can “attach” the file system on a block device to the root file system. Utility dd (as it is called the “disk destroyer”) provides copying by blocks. You need to use it with extreme caution, as an ill-considered launch can permanently destroy the data on the device.
Consider a few examples.
Mount disk /dev/sdb1 with mount point /mnt/usb (directory /mnt/usb should already exist):
mount /dev/sdb1 /mnt/usb
Mount device with file system ext4 only for reading:
mount /dev/sdb1 /mnt/usb -t ext4 -o ro -o noload
Unmount mount point:
umount /mnt/usb
Force unmount file system:
umount -f /mnt/usb
Copy blocks of one device to another:
dd if=/path/to/input of=/path/to/output
Write the image to the device:
dd bs=4M if=/path/to/linux.iso of=/dev/sdx
There are plenty of ways to “look” at a file from different angles. Using head and tail we can read the beginning and end of the file, respectively. Let us proceed directly to the examples of the use of these commands. Display the first 20 lines from the file:
head -n 20 access.log
Display the last 30 lines of the file:
tail -n 30 error.log
Launch tail in the tracking mode of the new line:
tail -f access.log
These commands allow you to view the contents of the file partially. In the case of the line output file content is better to use the utility. Less, which has many more shortcuts to navigate and search. Consider examples. Open file for paginated output::
less access.log
Open file with line number displaying:
less -N access.log
Shortcuts: • SHIFT+G – go to the end of the file; • g – go to the beginning of the file; • / template – find the following match; • ? template – find the previous match; • n – go to the next match; • SHIFT+N – go to the previous match.
The need to search for information in the file system occurs quite often (search for configuration files, search for the file that generated the error message in the logs etc), and utilities such as find and grep. Let’s turn to examples of using the utility find. Find files with name netdata.conf:
find -name 'netdata.conf'
Find all files with extension .conf:
find / -name '*.conf'
Find all files with name apache2:
find / -type f -name 'apache2'
Find all directories with a name nginx:
1 find / -type d -name 'nginx'
Find all files larger than 100MB in your home directory:
find ~ -size +100M
Find in the home directory all files with a size less than 100MB:
find ~ -size -100M
Find all empty files in the home directory:
find ~ -empty
Delete all empty files in the home directory ({} replaced by the file name):
find ~ -empty -exec rm -rf {} \;
Searching for a file by meta tags is fine, but what if you need to dig deeper? For these purposes, you can use the grep utility – search and filter by the pattern. Find word Forbidden in file error.log:
grep 'Forbidden' error.log
Find word forbidden in file error.log (case-insensitive search):
grep -i 'forbidden' error.log
Display the number of matches found:
grep -c 'Forbidden' error.log
Display an additional 2 lines after the match:
grep -i -A2 'forbidden' error.log
Display an additional 2 lines before the match:
grep -i -B2 'forbidden' error.log
Display an additional 2 lines before and after the match:
grep -i -C2 'forbidden' error.log
Find a phrase Access denied in all files in the folder ~/.pm2/logs:
grep -i -r 'access denied' ~/.pm2/logs
Quite often it is necessary to manage processes or simply to obtain information about all or a specific process. Consider examples of the use of such commands. Display a list of all processes:
ps aux
Display only node processes:
ps aux | grep node
Display processes as a tree, show only pid and command:
ps -e -o pid,args --forest
Send signal SIGTERM (sent by default) to the process with pid 8888:
kill -SIGTERM 8888
Send SIGKILL (force terminate the process) to the process with pid 8888:
kill -9 8888
Stop all processes named node:
killall node
Display processes whose parent is the process with pid 3607:
pgrep -P 3607
Display the processes that opened the file /etc/hosts:
lsof /etc/hosts
Find the process which took port 80:
lsof -i :80
To start the process in the background, it is enough to add an ampersand (&) at the end of the command, but this option has a drawback: if the shell session ends – all its background processes will also stop. So, it is better to use screen or even better – the service, but we will not consider the services. Let us turn to examples. Run the process in the background:
ping google.com &
Run several processes in the background:
ping goole.com & nmap 192.168.1.* &
Display a list of background processes:
jobs -l
Get access to the process (put it into priority mode):
%1
Bring the process back to background: CTRL+Z
%1 &
A little hint for background processes: to stop the process, put it into priority mode and click CTRL+C or use the command kill. Using screen saves us from the problem that arises with &. In the screen session, you can start any process and then detach session. Consider a few commands. Start a new screen session:
screen
Transfer session to detached mode: CTRL+A+D Start the process in a new session in detached mode:
screen -d -m ping google.com
View the list of sessions:
screen -ls
Attach screen session:
screen -R [session id]
The system slows down, and there is not enough space … how to identify the source of all these problems? There are several useful tools to identify the source of such problems. That gives a starting point for further investigation. Launch interactive process monitor:
top
htop
Run I/O monitor:
iotop
Display space usage:
du /path/to/directory
Analyze space usage:
ncdu /path/to/directory
To understand the system parameters, you can use the mass of available utilities that provide the ability to obtain information about the characteristics of the system. Display OS name and version:
lsb_release -a
See the full list of all devices:
lshw
See processor information:
lscpu
Display RAM Information:
free -h
See information about all mount points:
df -h
Display information about all available block devices:
lsblk
In this section, we consider the commands for working with the network – usage of the commands curl, tcpdump and nmap. Utility curl is widely used if you need something to “pull up” from the network, tcpdump is used for packet analysis, and nmap – is a network scanner that allows you to identify systems and services running on them. Let’s go directly to the examples. Run HEAD request (get headers only):
curl -I http://google.com
Run POST request with data sending:
curl -d 'first_name=John&last_name=Doe' http://google.com
Send JSON to server:
curl -d '{"name":"John"}' -H 'content-type: application/json' http://google.com
Download file (similar to using wget):
curl -O https://www.google.com/images/branding/googlelogo/2x/googlelogo_light_color_272x92dp.png
Interesting use case. Command curl is a reiteration of browser’s requests. In Google Chrome’s Developers tools -> in the tab “Network” there is an opportunity to copy curl command for the selected query (find the desired query > Right mouse button click > Copy > Copy as cURL). After that, the copied command can be inserted into the terminal and execute the request again. The provided command contains all the headers, parameters and the body of the request. Let us turn back to examples of the use of the following utilities. Track packets that were sent from the local machine:
tcpdump src 127.0.0.1
Track packets that came to the local machine through a specific network interface:
tcpdump dst 127.0.0.1 -i eth0
Display a list of network interfaces:
tcpdump --list-interfaces
Track packages that are gone from the local machine to google.com:
tcpdump src 127.0.0.1 and dst google.com -n
Display captured packets in ASCII:
tcpdump dst google.com and port 80 -A
Track packets to a specific IP and port:
tcpdump dst google.com and port 443 -n
Scan a specific server:
nmap -sP 217.160.0.201
Scan the local network:
nmap -sP 192.168.1.*
Try to determine the server OS:
nmap -O 192.168.1.8
Scan server ports (use –sV to determine the version of the service):
nmap -Pn 192.168.1.8
Run a quick scan (scanning of the standard services):
nmap -F 192.168.1.8
Scan specific ports (use –open to display only open ports):
nmap -Pn -p 80,443 192.168.1.8
Scan port combination (U – UDP port, T – TCP port, 21-25 – port range):
nmap -p U:53,111,137,T:21-25,80,139,8080 192.168.1.1
Scan 10 top ports (ssh, ftp, http …):
nmap -Pn --top-ports 10 192.168.1.8
In this section, we will look at useful utilities that are outside categories, but they are used quite often. Immediately proceed to the examples. Start the process with an update every 500 ms:
watch -n0.5 ls -laS
Reset terminal:
reset
Display the calendar for the current year:
cal -j
Display calendar for June 2021:
cal -j 6 2021
Display information about the current user:
id
Display current user name:
whoami
Convert string to base64:
echo Hello | base64
Decode base64 string:
echo "SGVsbG8K" | base64 -d
Format json:
echo '{"status":1}' | jq
curl https://opinionated-quotes-api.gigalixirapp.com/v1/quotes | jq
In this article we reviewed a selection of Linux utilities which we use in our daily work. And we have some bonus for those have read till the line – you can download this utilities list in printable version. Of course, this list is not limited to this set, there are plenty of others, but the article is already a big one. Finally, I want to say – read the manual, there are descriptions of all possible parameters and examples of use. And remember – search and you shall find!
Learn more about how we engage and what our experts can do for your business
Written by:
PHP developer
Blockchain Developer with more than 3 years of experience in developing decentralized applications. Diverse knowledge of crypto-economic protocols and cryptocurrency frameworks, focused attention on user experience design, good grasp of cryptography fundamentals.
More and more IoT solutions are launched and rolled out on the market. No wonder IoT programming providers have become deeply concerned about optimizing the…
The use of microservices architecture is already widespread among global companies. For instance, Amazon, Coca-Cola, and Netflix started out as monolithic apps but have evolved…
This post is not a tutorial. There are enough of them on the Internet. It is always more interesting to look at real production app….
Each programmer must have come across the necessity to check user’s input a number of times. Having a 12-year experience in web development, I have…
Preface Today we want to share our experience of building offline-first React-Native application, using Redux-like approach on our NodeJS backend server. Indeed we just kind…
Wearable IoT Trends: Personal and Business Use in 2022 Wearable IoT devices remain one of the most popular trends of IoT, which in turn is…