A collection of proven Unix commands to help you identify application issues
Identifying an application issue can be very time-consuming. That’s why it’s essential to know various troubleshooting commands which can make it easier for us to spot the problem.
Unix-like systems come with many powerful in-built commands. In this article, I’ve gathered the top ones which I’ve been using a lot. You can use the examples here to create your own cheat sheet for solving issues.
Let’s get started!
The lsof Command
lsof stands for “list open files.” It’s undoubtedly one of the most powerful troubleshooting commands because you can get a lot of information from it.
Retrieve open files belonging to active process
Without any arguments, lsof lists all open files belonging to all active processes. This can come in handy when you want to identify which process is using a file you are trying to delete.
To try it out, type this command in your Terminal:
In another scenario, there might be a suspicious process you want to examine. You can check the files which are used by that process to receive more information.
Note that you might get “Permission denied” for some processes. It’s because your user is not allowed to view processes owned by other users. In such case, run the command as root and you’ll be able to see all processes by all users.
List open files by user
If you want to find out which files are in use by a certain user, type this command:
$ lsof -u someuser
You’d be surprised how many open files there are. To narrow down the results, you might want to include the grep command.
To demo this option, I’ve started a Java application in the background.
$ lsof -u kirshi | grep java
End all user processes
If you want to end all processes of a user, you can easily do that by executing:
kill -9 `lsof -t -u someuser`
Retrieve processes running on a specific port
Sometimes you want to list all running processes on a specific port. To achieve this, execute:
lsof -i :8090
Note that the -i argument lists all listening and established connections.
If you’re not sure about the port number, you can query the ports within a specified range:
$ lsof -i :8090-9090
Another way to find a process is to use a PID id:
lsof -p 23619
The result will contain all processes belonging to 23619 PID id.
The netstat Command
netstat is a great tool for monitoring network connections.
Find ports in use
You can see all ports in use, including some other useful information, by typing:
- The -t option checks for TCP connections.
- The -u option checks for UDP connections.
- The -l option tells netstat to list only LISTENING connections. If you want to see all connections, use the -a option instead.
- The -p option shows the PID id of the process.
- The -n option shows numerical addresses, instead of trying to resolve host, port, or user names.
If you want to check a specific port, just add the grep command:
netstat -tulpn | grep 8090
Find user behind a process
In some cases, it can be useful to know the username behind the running process. Add the -e option to retrieve the username. Don’t forget to remove the -n option; otherwise, the username won’t be resolved.
$ sudo netstat -tulpe | grep 8090
Again, if you are querying as root, you’ll have privileges to see details about all processes.
Check server status
If you want to check if a server like http or smtp is running, you can use grep to filter out the results:
$ sudo netstat -tuple | grep smtp
The curl Command
curl is usually used for transferring data from or to a server without user interaction. It supports protocols like HTTP, FTP, SCP, SFTP, SMTP, LDAP, and many others.
Check service health
It can come in handy when you want to determine if a site or service is up.
To check if a site is alive, run the following command:
$ curl -Is http://www.google.com
HTTP/1.1 200 OK
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Date: Sun, 07 Feb 2021 22:22:10 GMT
Expires: Sun, 07 Feb 2021 22:22:10 GMT
Set-Cookie: NID=208=WV1ENCL_yTQadGsKWAuYHUyBL0Htelh0nkkSNjnFpX2XjHEBj58-ldS9vjAl6MXH0jFwQC0oabKtcmS6jia2Wn9V-1C0CckpYyW17i8l3VTsQkL2RDLS43OBKxuE5UXiSfm9iSLbIxVV94oNcjC2SPuUfUIzwCJqgxDGi7Q3qRM; expires=Mon, 09-Aug-2021 22:22:10 GMT; path=/; domain=.google.com; HttpOnly
- The -s option is used for quiet mode, which disables the progress meter and the error messages.
- The -I option sends a HEAD HTTP request and doesn’t return the request body.
Test telnet functionality
curl is a quick way to test a telnet functionality. For instance, you may want to confirm that the connection from your local machine to a remote host using a specific TCP port is working.
Let’s test a Tomcat server running on port 8090:
curl -v telnet://127.0.0.1:8090
* Trying 127.0.0.1:8090...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8090 (#0)
If I stop the server, I’ll get this error message:
curl: (7) Failed to connect to 127.0.0.1 port 8090: Connection refused
Test connection between your app and another service
You can also verify if there is a connection between your application and another service, such as a database.
$ curl -I -s myapp:5000
myapp is the name of your application and 5000 is the database’s port number you’re trying to reach.
The ps Command
ps is another useful command to display information about running processes.
Find PID of a process
The ps command is often combined with -ef options:
$ ps -ef
- The -e option displays all processes.
- The -f provides detailed information about the processes.
You can use grep to narrow down the results:
$ ps -ef | grep tomcat
You can also use it with the aux options for even more detailed information.
- The -a option display the processes of all users.
- The -u stands for a user-oriented format.
- The -x also shows processes running in the background.
Sort processes by memory
It can be useful to sort the processes by memory usage to find out what’s eating up your resources:
$ ps aux --sort=-%mem
Extra Tip: cat Command
You’re surely familiar with the cat command. Although it’s not usually used for troubleshooting, it still can come in handy to identify a problem.
For example, when you receive an error message that a script has failed, let’s say at line 21, you want to see that line. You could use an external tool to visualize the file to check the line numbers. But you can also achieve that using cat:
$ cat -n show-time.sh
2 while true; do
3 sleep 5
4 date +"Hi, the current time is %F %T"
- The -n option shows the line numbers in a file.
If you’re dealing with a long file, you can use the less command to jump to the desired line number straight from the command line:
$ less +4 -N show-time.sh
4 date +"Hi, the current time is %F %T"
- The -N option shows the line numbers.
- + sign and a number jump to that line in the file
We’ve seen how to make use of commands like curl, netstat, ps, and lsof. These commands can be useful in many other scenarios. I recommend going through the manual pages to discover more use cases.
If you’ve enjoyed this tutorial, you might also like my articles about other Unix tips and tricks:
- Lesser-Known but Powerful Unix Commands to Polish Your Tech Skills
- Write More Efficient Unix Scripts With Heredoc
- Automate Your Scripts With Expect
Thank you for reading and see you next time!