The job of each developer is the constant occurrence of problems and the search for their solutions. The speed of solving problem depends on the knowledge, skills and abilities to work in the development environment. However, we should not forget about the ability to search for those solutions. When you click on the first link in the Google search results, we can get a valid solution in 90% of cases. Here we must admit with confidence that almost all the problems have already been solved before. However, you must agree that it is cool if you could use it immediately and not search.
In our daily work, we use a lot of different tools: from searching for a regular file to the creation of a file system. This post describes the “must-have” tools that seem trivial to someone, and someone will say: “And how did I live without it before?”.
I divided these tools into groups by generalized problems.
Before we start, it is worth mentioning an incredible thing that makes life easier for every developer: a zsh command shell in combination with oh-my-zsh. The shell zsh itself is very convenient, as it provides many features like typo correction, programmable auto-completion, the ability to quickly navigate both directories and history, and much more. Oh-my-zsh is a way to make zsh even better. Oh-my-zsh adds a huge set of aliases for frequently used commands to zsh, but we will not dwell on it anymore since its consideration requires a separate article.
So, let’s move on to our selection of linux-utilites.
Basic utilities for working with the file system every Linux user should know. Here are some useful commands.
Move (rename) file:
Copy file to directory:
Copy the directory with all the files inside it (the content of the directory source_directory will be copied to destination_directory):
Copy several files:
Delete a specific file:
Delete files with extension .txt:
Delete directory with all files inside (flag -f to delete without confirmation):
Change file owner:
Change the owner of the directory and all files inside it:
Give all users the permission to read and write to the directory and all files inside it:
Make the file permission the same as the other file (you can use it in chown):
Command chmod, as you can see from the example above, allows you to use a fairly simple syntax in which to use and memorize:
• users – u (owner), g (group), o (others), a (all);
• permissions – r (read), w (write), x (execute).
More advanced set includes utilities mount and dd.
Using the mount, you can “attach” the file system on a block device to the root file system.
Utility dd (as it is called the “disk destroyer”) provides copying by blocks. You need to use it with extreme caution, as an ill-considered launch can permanently destroy the data on the device.
Consider a few examples.
Mount disk /dev/sdb1 with mount point /mnt/usb (directory /mnt/usb should already exist):
Mount device with file system ext4 only for reading:
Unmount mount point:
Force unmount file system:
Copy blocks of one device to another:
Write the image to the device:
There are plenty of ways to “look” at a file from different angles. Using head and tail we can read the beginning and end of the file, respectively. Let us proceed directly to the examples of the use of these commands.
Display the first 20 lines from the file:
Display the last 30 lines of the file:
Launch tail in the tracking mode of the new line:
These commands allow you to view the contents of the file partially. In the case of the line output file content is better to use the utility. Less, which has many more shortcuts to navigate and search. Consider examples.
Open file for paginated output::
Open file with line number displaying:
Shortcuts:
• SHIFT+G – go to the end of the file;
• g – go to the beginning of the file;
• / template – find the following match;
• ? template – find the previous match;
• n – go to the next match;
• SHIFT+N – go to the previous match.
The need to search for information in the file system occurs quite often (search for configuration files, search for the file that generated the error message in the logs etc), and utilities such as find and grep. Let’s turn to examples of using the utility find.
Find files with name netdata.conf:
find -name 'netdata.conf'Find all files with extension .conf:
find / -name '*.conf'Find all files with name apache2:
find / -type f -name 'apache2'
Find all directories with a name nginx:
1
find / -type d -name 'nginx'
Find all files larger than 100MB in your home directory:
Find in the home directory all files with a size less than 100MB:
Find all empty files in the home directory:
Delete all empty files in the home directory ({} replaced by the file name):
Searching for a file by meta tags is fine, but what if you need to dig deeper? For these purposes, you can use the grep utility – search and filter by the pattern.
Find word Forbidden in file error.log:
grep 'Forbidden' error.log
Find word forbidden in file error.log (case-insensitive search):
grep -i 'forbidden' error.log
Display the number of matches found:
grep -c 'Forbidden' error.log
Display an additional 2 lines after the match:
grep -i -A2 'forbidden' error.log
Display an additional 2 lines before the match:
grep -i -B2 'forbidden' error.log
Display an additional 2 lines before and after the match:
grep -i -C2 'forbidden' error.log
Find a phrase Access denied in all files in the folder ~/.pm2/logs:
grep -i -r 'access denied' ~/.pm2/logs
Quite often it is necessary to manage processes or simply to obtain information about all or a specific process. Consider examples of the use of such commands.
Display a list of all processes:
ps aux
Display only node processes:
ps aux | grep node
Display processes as a tree, show only pid and command:
ps -e -o pid,args --forest
Send signal SIGTERM (sent by default) to the process with pid 8888:
kill -SIGTERM 8888
Send SIGKILL (force terminate the process) to the process with pid 8888:
kill -9 8888
Stop all processes named node:
killall node
Display processes whose parent is the process with pid 3607:
pgrep -P 3607
Display the processes that opened the file /etc/hosts:
lsof /etc/hosts
Find the process which took port 80:
lsof -i :80
To start the process in the background, it is enough to add an ampersand (&) at the end of the command, but this option has a drawback: if the shell session ends – all its background processes will also stop. So, it is better to use screen or even better – the service, but we will not consider the services. Let us turn to examples.
Run the process in the background:
Run several processes in the background:
Display a list of background processes:
Get access to the process (put it into priority mode):
Bring the process back to background:
CTRL+Z
%1 &
A little hint for background processes: to stop the process, put it into priority mode and click CTRL+C or use the command kill.
Using screen saves us from the problem that arises with &. In the screen session, you can start any process and then detach session. Consider a few commands.
Start a new screen session:
screen
Transfer session to detached mode:
CTRL+A+D
Start the process in a new session in detached mode:
View the list of sessions:
Attach screen session:
The system slows down, and there is not enough space … how to identify the source of all these problems? There are several useful tools to identify the source of such problems. That gives a starting point for further investigation.
Launch interactive process monitor:
Run I/O monitor:
Display space usage:
Analyze space usage:
To understand the system parameters, you can use the mass of available utilities that provide the ability to obtain information about the characteristics of the system.
Display OS name and version:
See the full list of all devices:
See processor information:
Display RAM Information:
See information about all mount points:
Display information about all available block devices:
In this section, we consider the commands for working with the network – usage of the commands curl, tcpdump and nmap. Utility curl is widely used if you need something to “pull up” from the network, tcpdump is used for packet analysis, and nmap – is a network scanner that allows you to identify systems and services running on them. Let’s go directly to the examples.
Run HEAD request (get headers only):
curl -I http://google.com
Run POST request with data sending:
curl -d 'first_name=John&last_name=Doe' http://google.com
Send JSON to server:
curl -d '{"name":"John"}' -H 'content-type: application/json' http://google.com
Download file (similar to using wget):
curl -O https://www.google.com/images/branding/googlelogo/2x/googlelogo_light_color_272x92dp.png
Interesting use case.
Command curl is a reiteration of browser’s requests. In Google Chrome’s Developers tools -> in the tab “Network” there is an opportunity to copy curl command for the selected query (find the desired query > Right mouse button click > Copy > Copy as cURL). After that, the copied command can be inserted into the terminal and execute the request again. The provided command contains all the headers, parameters and the body of the request.
Let us turn back to examples of the use of the following utilities.
Track packets that were sent from the local machine:
Track packets that came to the local machine through a specific network interface:
Display a list of network interfaces:
Track packages that are gone from the local machine to google.com:
tcpdump src 127.0.0.1 and dst google.com -n
Display captured packets in ASCII:
tcpdump dst google.com and port 80 -A
Track packets to a specific IP and port:
tcpdump dst google.com and port 443 -n
Scan a specific server:
nmap -sP 217.160.0.201
Scan the local network:
nmap -sP 192.168.1.*
Try to determine the server OS:
nmap -O 192.168.1.8
Scan server ports (use –sV to determine the version of the service):
nmap -Pn 192.168.1.8
Run a quick scan (scanning of the standard services):
nmap -F 192.168.1.8
Scan specific ports (use –open to display only open ports):
nmap -Pn -p 80,443 192.168.1.8
Scan port combination (U – UDP port, T – TCP port, 21-25 – port range):
nmap -p U:53,111,137,T:21-25,80,139,8080 192.168.1.1
Scan 10 top ports (ssh, ftp, http …):
nmap -Pn --top-ports 10 192.168.1.8
In this section, we will look at useful utilities that are outside categories, but they are used quite often. Immediately proceed to the examples.
Start the process with an update every 500 ms:
Reset terminal:
Display the calendar for the current year:
Display calendar for June 2021:
Display information about the current user:
Display current user name:
Convert string to base64:
Decode base64 string:
Format json:
In this article we reviewed a selection of Linux utilities which we use in our daily work.
And we have some bonus for those have read till the line – you can download this utilities list in printable version.
Of course, this list is not limited to this set, there are plenty of others, but the article is already a big one.
Finally, I want to say – read the manual, there are descriptions of all possible parameters and examples of use. And remember – search and you shall find!
Written by:
PHP developer
Blockchain Developer with more than 3 years of experience in developing decentralized applications. Diverse knowledge of crypto-economic protocols and cryptocurrency frameworks, focused attention on user experience design, good grasp of cryptography fundamentals.
2024 WEBBYLAB. All rights reserved.
Get a consultation with at Webbylab
