Working in a Linux environment requires knowledge of Linux cli tools and troubleshooting. In this article, it would be presented CLI tools that are most important for troubleshooting. Short story Let say you are working with Linux and your work would be: 1% once-time setup and 99% troubleshooting. So as we can see from these homemade statistics you going to spend most of your time in CLI and finding “why does not work”. This is not a trivial task. It requires knowledge of how Linux works, how subsystems of Linux works, how complete “line” from typing command up to running, delivering some results works, logs. Yes, logs are the most important. So let me start with naming tools and then a short description of some of them. fs2chk du/dh strace/ltrace lsof ldd tcpdump netstat mailq/showq traceroute/tracepath ping/telnet/nc dig curl nmap top ps pkg manager(rpm, yum, apt, others) lsmod awk sed vim
fs2chk
Used for file system integrity check. Not all time you will have regular shutdown or reboot – sometimes it happens power goes off and your server gets back with a file system issue. Usually, in that case is best to use fs2chk
du/dh
Command du is used to show space usage per partition. Sometimes happen your partition are used with some dumb files/logs (crash files) and you need to see what is happening because application or service you try to run report free space issue. Command dh is used to check directory with sub- directories and discover what file occupied most of the space.
strace/ltrace
Running application it just stops at some point. You don’t have logs. Nothing. And there is no debug switch (for example ssh -vvv, where you can see all steps that are done) or any other way to see what is hell going on. So strace is for functions that is used and ltrace are library that are used at some point. We mostly need trace tool to see details of operation for some applications. Example: we run applications or services and we see an issue and we are not sure why.
lsof
You have an issue removing files/files because they are used by some unknown application. Or you want to see how much is open files so you can see if the maximum limit for open files is reached.
ldd
When the application does not run at all – the usual suspect is missing the library. This tool is handy to discover which library missing.
tcpdump
Connection to server sometimes has issues. So the best way to check what is going on and to troubleshooting is tcpdump. You can pick up the interface, type of protocol, from/to IP, or any low level for all TCP/IP layers.
netstat
Simple to see the status of open ports, connection state, and other information that we need to see if services running properly on a given port.
mailq/showq
The most vital service in each company is email. And sometimes you need to see what is happening with email (sending or recv). Best tool for this is mailq/showq (it is the same tool, showq is a new one that replace mailq on older Linux)
ethtool
On a low level could happen issue with our ethernet connection and we want to review our cable/port on Linux. So the tool is best for this job. Beside this, you can review other specific ethernet things (auto-negotiation etc)
traceroute/tracepath
Ideal to see between server and client if there is a network path issue as also a delay between them.
ping/telnet/nc
The very handy tool on the first step to see if is server up as also services.
dig
With this tool, you can perform all DNS troubleshooting. Review MX records, A, NS, etc.
curl
One of the best tools for troubleshooting different protocols: HTTP, RTMP, FTP, etc. It also has a benchmark integrated for a view of response (DNS, first byte, etc).
nmap
A very good tool for discovering services, open ports, and other useful information. Also, you can use on your servers to check if there is some unusual thing and secure them.
top
Active process list with memory, CPU, parent/child connection, and other information that helps to see where the issue starts.
ps
Process list, you can check and see what is currently running (very quick, the first step for troubleshooting) pkg manager(rpm, yum, apt, others) In troubleshooting we need to verify or to find some library or application – so this tool is best for that operation.
lsmod
People who never have issues with kernel modules would never use this tool or get this tool seriously. The tool provides information about loaded kernel modules as also usage, memory, etc.
awk/sed/grep
In a bunch of logs sometimes is a need to find proper information. All these tools are swiss knives for bash scripting and handy for parsing logs for specific information.
vim
The best editor in the world. Learn so you can answer on an interview how to quit vim.
