A collection of proven Unix commands to help you identify application issues

decorative: computer keyboard lit by pink light
Photo by Aryan Dhiman on Unsplash

Identifying an application issue can be very time-consuming. That’s why it’s essential to know various troubleshooting commands which can make it easier for us to spot the problem.

Unix-like systems come with many powerful in-built commands. In this article, I’ve gathered the top ones which I’ve been using a lot. You can use the examples here to create your own cheat sheet for solving issues.

Let’s get started!

The lsof Command

lsof stands for “list open files.” It’s undoubtedly one of the most powerful troubleshooting commands because you can get a lot of information from it.

Retrieve open files belonging to active process

Without any arguments, lsof lists all open files belonging to all active processes. This can come in handy when you want to identify which process is using a file you are trying to delete.

To try it out, type this command in your Terminal:

$ lsof

Example output:

In another scenario, there might be a suspicious process you want to examine. You can check the files which are used by that process to receive more information.

Note that you might get “Permission denied” for some processes. It’s because your user is not allowed to view processes owned by other users. In such case, run the command as root and you’ll be able to see all processes by all users.

List open files by user

If you want to find out which files are in use by a certain user, type this command:

$ lsof -u someuser

You’d be surprised how many open files there are. To narrow down the results, you might want to include the grep command.

To demo this option, I’ve started a Java application in the background.

$ lsof -u kirshi | grep java

Output extract:

End all user processes

If you want to end all processes of a user, you can easily do that by executing:

kill -9 `lsof -t -u someuser`

Retrieve processes running on a specific port

Sometimes you want to list all running processes on a specific port. To achieve this, execute:

lsof -i :8090

Example output:

Note that the -i argument lists all listening and established connections.

If you’re not sure about the port number, you can query the ports within a specified range:

$ lsof -i :8090-9090

Another way to find a process is to use a PID id:

lsof -p 23619

The result will contain all processes belonging to 23619 PID id.

The netstat Command

netstat is a great tool for monitoring network connections.

Find ports in use

You can see all ports in use, including some other useful information, by typing:

netstat -tulpn
  • The -t option checks for TCP connections.
  • The -u option checks for UDP connections.
  • The -l option tells netstat to list only LISTENING connections. If you want to see all connections, use the -a option instead.
  • The -p option shows the PID id of the process.
  • The -n option shows numerical addresses, instead of trying to resolve host, port, or user names.

If you want to check a specific port, just add the grep command:

netstat -tulpn | grep 8090

Example output:

Find user behind a process

In some cases, it can be useful to know the username behind the running process. Add the -e option to retrieve the username. Don’t forget to remove the -n option; otherwise, the username won’t be resolved.

$ sudo netstat -tulpe | grep 8090

Again, if you are querying as root, you’ll have privileges to see details about all processes.

Check server status

If you want to check if a server like http or smtp is running, you can use grep to filter out the results:

$ sudo netstat -tuple | grep smtp

The curl Command

curl is usually used for transferring data from or to a server without user interaction. It supports protocols like HTTP, FTP, SCP, SFTP, SMTP, LDAP, and many others.

Check service health

It can come in handy when you want to determine if a site or service is up.

To check if a site is alive, run the following command:

$ curl -Is http://www.google.com
HTTP/1.1 200 OK
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Date: Sun, 07 Feb 2021 22:22:10 GMT
Server: gws
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Transfer-Encoding: chunked
Expires: Sun, 07 Feb 2021 22:22:10 GMT
Cache-Control: private
Set-Cookie: NID=208=WV1ENCL_yTQadGsKWAuYHUyBL0Htelh0nkkSNjnFpX2XjHEBj58-ldS9vjAl6MXH0jFwQC0oabKtcmS6jia2Wn9V-1C0CckpYyW17i8l3VTsQkL2RDLS43OBKxuE5UXiSfm9iSLbIxVV94oNcjC2SPuUfUIzwCJqgxDGi7Q3qRM; expires=Mon, 09-Aug-2021 22:22:10 GMT; path=/; domain=.google.com; HttpOnly
  • The -s option is used for quiet mode, which disables the progress meter and the error messages.
  • The -I option sends a HEAD HTTP request and doesn’t return the request body.

Test telnet functionality

curl is a quick way to test a telnet functionality. For instance, you may want to confirm that the connection from your local machine to a remote host using a specific TCP port is working.

Let’s test a Tomcat server running on port 8090:

curl -v telnet://127.0.0.1:8090
* Trying 127.0.0.1:8090...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8090 (#0)

If I stop the server, I’ll get this error message:

curl: (7) Failed to connect to 127.0.0.1 port 8090: Connection refused

Test connection between your app and another service

You can also verify if there is a connection between your application and another service, such as a database.

$ curl -I -s myapp:5000

myapp is the name of your application and 5000 is the database’s port number you’re trying to reach.

The ps Command

ps is another useful command to display information about running processes.

Find PID of a process

The ps command is often combined with -ef options:

$ ps -ef
  • The -e option displays all processes.
  • The -f provides detailed information about the processes.

Sample output:

You can use grep to narrow down the results:

$ ps -ef | grep tomcat

You can also use it with the aux options for even more detailed information.

  • The -a option display the processes of all users.
  • The -u stands for a user-oriented format.
  • The -x also shows processes running in the background.

Sort processes by memory

It can be useful to sort the processes by memory usage to find out what’s eating up your resources:

$ ps aux --sort=-%mem

Extra Tip: cat Command

You’re surely familiar with the cat command. Although it’s not usually used for troubleshooting, it still can come in handy to identify a problem.

For example, when you receive an error message that a script has failed, let’s say at line 21, you want to see that line. You could use an external tool to visualize the file to check the line numbers. But you can also achieve that using cat:

$ cat -n show-time.sh 
1  #!/bin/bash
2  while true; do
3      sleep 5 
4      date +"Hi, the current time is %F %T"
5  done
  • The -n option shows the line numbers in a file.

If you’re dealing with a long file, you can use the less command to jump to the desired line number straight from the command line:

$ less +4 -N show-time.sh
4     date +"Hi, the current time is %F %T"
5     done
  • The -N option shows the line numbers.
  • + sign and a number jump to that line in the file

Conclusion

We’ve seen how to make use of commands like curl, netstat, ps, and lsof. These commands can be useful in many other scenarios. I recommend going through the manual pages to discover more use cases.

If you’ve enjoyed this tutorial, you might also like my articles about other Unix tips and tricks:

Thank you for reading and see you next time!

Proven Unix Commands for Easier Troubleshooting was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.