Skip to content

LINUX HAX.org

Latest commit

 

History

History
922 lines (786 loc) · 29.9 KB

LINUX HAX.org

File metadata and controls

922 lines (786 loc) · 29.9 KB

Linux Hax

THE GOLDEN RULES

  • Check your firewall
  • Do you use sudo and therefor have different runtime variables (or a different environment)? This couldn’t be confirmed by my colleagues :/
  • Check the logfile file permissions if no log-messages show up

Oracle Linux

In Oracle Linux most of the host-related config can be changed in /etc/sysconfig/

Change the hostname

Edit the following files:

  • /etc/sysconfig/network
  • /etc/hosts

Change keyboard layout

Change suspicious-looking entries in /etc/sysconfig/keyboard from “en” into “de”

Ubuntu

Hint for the desktop:

  • Move windows: control-windows arrow

Change hostname

Change these files:

  • /etc/hosts (fqdn, then shorthostname, seperated via tab)
  • /etc/hostname (short hostname)

chkconfig (autostart configuration)

#equivalents to these:
chkconfig --add <service> chkconfig --level 345 <service> on
chkconfig --del <service>
#are:
update-rc.d <service> defaults update-rc.d <service> start 20 3 4 5
update-rc.d -f <service> remove

Gnome Terminal

The other options can be easily changed in the terminal, but the tab shifting needs to be configured like so:

gsettings set org.gnome.Terminal.Legacy.Keybindings:/org/gnome/terminal/legacy/keybindings/ next-tab '<Primary>Tab'
gsettings set org.gnome.Terminal.Legacy.Keybindings:/org/gnome/terminal/legacy/keybindings/ prev-tab '<Primary><Shift>Tab'

Icinga Monitoring

Nrpe Tipps:

Check multiple disks in one service

This assumes an nrpe setup is already at place and only this check has to be added. It also requires the file check_multi at /usr/lib64/nagios/plugins/ with execute permissions.

# check if multiple disks are attached at all
df -h # df -l if df -h doesnt respond
# check if check_multi config file exists
cat /etc/nagios/check_multi_disk.cmd
# make sure this exists in /etc/nagios/nrpe.cfg:
command[check_disk]=/usr/lib64/nagios/plugins/check_multi -f /etc/nagios/check_multi_disk.cmd -r 32
# check the command locally:
/usr/lib64/nagios/plugins/check_nrpe -H localhost -c check_disk
/usr/lib64/nagios/plugins/check_nrpe -H 127.0.0.1 -c check_disk #alternative for ssl handshake error

Example content of /etc/nagios/check_multi_disk.cmd

command[disk1]=/usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p /
command[disk2]=/usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p /home
command[disk3]=/usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p /boot

What to check? Basically everything that starts with /dev:

[root@somehost-blade1 ~]# df -h | grep ^/dev
/dev/mapper/ol_somehost--blade1-root   50G  6.9G   44G  14% /
/dev/mapper/ol_nhhbc015--blade1-home  225G   94M  225G   1% /home
/dev/sda1                             497M  168M  330M  34% /boot

Debugging tips

Get more information on config reload errors:

/usr/lib/icinga2/safe-reload /etc/sysconfig/icinga2

Basic nrpe setup ontop of xinetd

Setup on monitoring-server

Required for the localhost template to work that comes with the icinga installation

yum install nagios-plugins-all

In order to trigger remote nagios-plugins this package is necessary

yum install nagios-plugins-nrpe

Setup on each client

Install these packages. Nagios-plugins-all is an overkill, but it is really small so its okay.

yum install -y nrpe nagios-plugins-nrpe nagios-plugins-all xinetd

/etc/xinetd.d/nrpe has to exist in a specific way:

cat << EOF > /etc/xinetd.d/nrpe
# default: off
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
        flags           = REUSE
        socket_type     = stream
        type            = UNLISTED
        port            = 5666
        wait            = no
        user            = nagios
        group           = nagios
        server          = /usr/sbin/nrpe
        server_args     = -c /etc/nagios/nrpe.cfg --inetd
        log_on_failure  += USERID
        disable         = no
        only_from       = 127.0.0.1 monitoring-server
}
EOF

Restarting xinetd is required

chkconfig xinetd on
service xinetd restart

Check if the setup works locally

/usr/lib64/nagios/plugins/check_nrpe -H localhost -c check_disk
/usr/lib64/nagios/plugins/check_nrpe -H 127.0.0.1 -c check_disk #alternative for ssl handshake error

Add iptables exception

If a connection using telnet on port 22 is possible, but port 5666 shows “no route to host” the reason is probably an iptables-rule.

Check current Iptables rules:

iptables --list

Works on Centos 7:

iptables -A INPUT -s monitoring-server -p tcp -m tcp --dport 5666 -m state --state NEW,ESTABLISHED -j ACCEPT
yum install iptables-services
service iptables reload

SQLite

There are a few steps to see the tables in an SQLite database:

Connect to the db

sqlite3 grafana.db # assumes grafana.db exists in current directory

Query the db

# List the tables in your database:
.tables

# List how the table looks:
.schema tablename

# Print the entire table:
SELECT * FROM tablename;

# List all of the available SQLite prompt commands:
.help

# Close sqlite
.exit

MySQL

What to do when you cant start mysql as root (maybe not exactly right)

service mysqld stop
/usr/bin/mysqld start --skip-grant-tables -user=root
# mysqld_safe --skip-grant-tables & # alternatively
mysql -u root
# mysql # alternatively
UPDATE mysql.user SET Password=PASSWORD('newpwd') WHERE User='root';
FLUSH PRIVILEGES;

What to do when your mysql wont start generally

service mysqld stop
mv /var/lib/mysql/mysql.sock /var/lib/mysql/mysql.sock.bak
service mysqld start

Overall important commands

mysqladmin -u root -p'abc' password '123456' --change root password
show databases;
use testdatabase; --enables you to query the selected database
show tables;
describe testtable; --show available columns / attributes
select * from testtable where id=1\G --display columns nicely; for tables with many columns

Mysql 5.7.5 or lower

SET PASSWORD FOR 'user-name-here'@'hostname' = PASSWORD('new-password');

Oracle Enterprise Linux OREL

Add Epel repository

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm

Then check whether it’s enabled

yum repolist

Free up disk-space

Source: https://uhesse.com/2011/06/01/adrci-a-survival-guide-for-the-dba/

[oracle@uhesse ~]$ adrci

ADRCI: Release 11.2.0.2.0 - Production on Wed Jun 1 10:20:08 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

ADR base = "/u01/app/oracle"
adrci> show home
ADR Homes:
diag/tnslsnr/uhesse/listener
diag/rdbms/orcl/orcl
adrci> set homepath <insert one of the ADR homes>
adrci> purge -age 2880 -type trace
adrci> show tracefile -rt

Package management

Differences in software architectures:

  • x86-64: Initial description for 64-bit architecture
  • amd64 : x86-64 for amd or intel processors (basically just a new name for the same thing)
  • i386 : 32 bit architecture

RPM

Some usefull hacks

#Check whether a folder belongs to a package
rpm -q -whatprovides /etc/profile
#List all installed packages
rpm -qa | less
#Search for a specific package
rpm -qa | grep -i name
#Uninstall a package
rpm -e name
#Update a package with a new rpm in the current directory
rpm -U local_upgrade.rpm

Apt

Hold back packages, to prevent updates

For me this was useful to prevent apt from overwriting my self-compiled git.

sudo apt-mark hold <package>
sudo apt-mark showhold

/boot Partition full

fire this up multiple times

sudo apt-get autoremove

Yum

Add a repository to yum

For CentOS and Red Hat Enterprise Linux 6.x:

# Centos extras repository (contains for example nrpe checks)
sudo wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
sudo rpm -Uvh epel-release-latest-6*.rpm
# RHEL/CentOS 6 32-Bit
wget http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -ivh epel-release-6-8.noarch.rpm
# RHEL/CentOS 6 64-Bit
wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -ivh epel-release-6-8.noarch.rpm

Alternatively you can just grab the .repo configration file

# For Oracle Linux 6.x:
cd /etc/yum.repos.d/
wget http://public-yum.oracle.com/public-yum-ol6.repo

If this shows: “Error: Cannot retrieve metalink for repository: epel. Please verify its path and try again” Maybe it is because of old certificates. An alternative would be to use repositorise with only http and not https. However, sticking with https, try:

yum --disablerepo=epel -y update  ca-certificates

Only download packages via yum

yum install yum-downloadonly
yum install --downloadonly --downloaddir=<directory> <package>

Proxy settings

Proxy for cpan

perl -MCPAN -e shell
o conf init /proxy/
http://172.16.8.250:3128
o conf commit
exit

Proxy for git

git config --global http.proxy http://172.16.8.250:3128; git config --global https.proxy  http://172.16.8.250:3128
git config --global --unset http.proxy; git config --global --unset https.proxy

Proxy for bash

#Just add the following to the end of you bashrc:
export http_proxy=http://172.16.8.250:3128 && export https_proxy=http://172.16.8.250:3128 && export ftp_proxy=http://172.16.8.250:3128
#Unset them:
export http_proxy= && export https_proxy= && export ftp_proxy=

Git

Cleanup

#cleanup all branches which were merged
git branch --merged | egrep -v "(^\*|master|main|dev)" | xargs git branch -d

# remove remote tracking branches
git remote prune origin

Author fix

Here’s how to do it with rebase and keep both the commit date and the author date:

git -c rebase.instructionFormat='%s%nexec GIT_COMMITTER_DATE="%cD" GIT_AUTHOR_DATE="%aD" git commit --amend --no-edit --reset-author' rebase -f <commit/branch before wrong author and email, or --root to rebase all>

Manual installation

Git 2.x Installation on Linux (taken from here: https://oracle-base.com/articles/linux/git-2-installation-on-linux)

The version of Git available from the Yum repositories is typically quite old. This article describes the manual installation of Git 2.x on Linux. The same method should work for all the RHEL clones (Oracle Linux, CentOS) and Fedora.

Install the prerequisite packages and remove the any current Git installation.

yum install curl-devel expat-devel gettext-devel openssl-devel zlib-devel -y
yum install gcc perl-ExtUtils-MakeMaker -y
yum remove git -y

Download the latest Git installation from kernel.org and unpack it.

cd /usr/src
wget https://www.kernel.org/pub/software/scm/git/git-2.2.2.tar.gz
tar xzf git-2.2.2.tar.gz

Install it.

cd git-2.2.2
make prefix=/usr/local/git all
make prefix=/usr/local/git install
echo "export PATH=$PATH:/usr/local/git/bin" >> /etc/bashrc
source /etc/bashrc

Test it, by checking the version.

git --version

Initial Settings

Set General information required by git

git config --global user.name "flyck"
git config --global user.email "felix.brilej@googlemail.com"
git config --global core.autocrlf false #Leave all line-ending business to the user

Show general information required by git

git config user.name
git config user.email
git config http.proxy

Fixes this error: “Local SSL issuer certificate error”

  • Alternatively i could just install that ssl certificate, probably the more advanced solution
git config --global http.sslVerify false

Install ssh key

# check if ssh-agent is running
ps -e | grep [s]sh-agent
# start the ssh-agent if it's not running
ssh-agent /bin/bash
# Unlock the ssh key
ssh-add ~/.ssh/id_rsa

Undo last commit

git commit ....  git reset --soft HEAD~1
# edit files as needed
git add ...  git commit -c ORIG_HEAD

if your commit is total crap:

git reset --hard HEAD~1

Clone Subdir

Suppose your project is in a directory called “project”, and you want only those commits which touch project/dirB.

git clone project/ subproject/
cd subproject
git filter-branch --prune-empty --subdirectory-filter dirB HEAD

The subproject will now contain the git history which touches dirB.

Show tags and their type

Taken from: https://stackoverflow.com/questions/40479712/how-can-i-tell-if-a-given-git-tag-is-annotated-or-lightweight/40480534

git show-ref -d --tags       |
cut -b 42-                   | # to remove the commit-id
sort                         |
sed 's/\^{}//'               | # remove ^{} markings
uniq -c                      | # count identical lines
sed 's/2\ refs\/tags\// a /' | # 2 identicals = annotated
sed 's/1\ refs\/tags\//lw /'

Remote branches vs local branches

https://git-scm.com/book/en/v2/Git-Branching-Remote-Branches

A command which gets me out of “detached HEAD” error message and properly connects local and remote branch:

git checkout -b my-crazy-feature origin/feature/my-crazy-feature

How to fetch all remote branches:

git branch -r | grep -v '\->' | while read remote; do git branch --track "${remote#origin/}" "$remote"; done
git fetch --all --tags
git pull --all

Count Contributions

Counts the contributions in local branches

git log --shortstat --pretty="%cE" | sed 's/\(.*\)@.*/\1/' | grep -v "^$" | awk 'BEGIN { line=""; } !/^ / { if (line=="" || !match(line, $0)) {line = $0 "," line }} /^ / { print line " # " $0; line=""}' | sort | sed -E 's/# //;s/ files? changed,//;s/([0-9]+) ([0-9]+ deletion)/\1 0 insertions\(+\), \2/;s/\(\+\)$/\(\+\), 0 deletions\(-\)/;s/insertions?\(\+\), //;s/ deletions?\(-\)//' | awk 'BEGIN {name=""; files=0; insertions=0; deletions=0;} {if ($1 != name && name != "") { print name ": " files " files changed, " insertions " insertions(+), " deletions " deletions(-), " insertions-deletions " net"; files=0; insertions=0; deletions=0; name=$1; } name=$1; files+=$2; insertions+=$3; deletions+=$4} END {print name ": " files " files changed, " insertions " insertions(+), " deletions " deletions(-), " insertions-deletions " net";}'

Fix: Local SSL issuer certificate error

Alternative: install that ssl certificate, probably the more advanced solution

git config --global http.sslVerify false

Fix: GNU TLS Error on clone

The error message: 
# happens after installing git using the normal ubuntu repositories
apt-get install git
git clone https://tfs.somesite.com
error: gnutls_handshake() failed: A TLS packet with unexpected length was received. while accessing https://tfs.somesite.com
sudo apt-get remove git
sudo apt-get install dpkg-dev
# download everything to get ready to compile
mkdir ~/git-openssl
cd ~/git-openssl
sudo apt-get source git
sudo apt-get build-dep git
sudo apt-get install libcurl4-openssl-dev
sudo dpkg-source -x git_1.7.9.5-1.dsc
cd git_1.7.9.5
vim debian/control # replace all instances of “libcurl4-gnutls-dev” with “libcurl4-openssl-dev” (I used sudo vim debian/control) using vim: :%s/libcurl4-gnutls-dev/libcurl4-openssl-dev/gc
sudo dpkg-buildpackage -rfakeroot -uc -b
# if it's faling on test, you can remove the line TEST=test from the file debian/rules (I used sudo vim debian/rules to edit the file)
sudo dpkg -i ../git_1.7.9.5-1_amd64.deb
sudo apt-mark hold git # makes it so git doesn't get overwritten by updates

Mailing

Fetchmail

Show certificate-chain openssl s_client -connect pop.gmx.net:995 -showcerts

  • the shown certificat has to copied to a .pem file from BEGIN to END
  • next the program “c_rehash” has to be executed in that directory 
    c_rehash .
  • in case the parent directory is registered as “sslcertpath” the certificate error is now resolved 
    ssl sslcertck sslcertpath /home/rtracker/.fetchmail/certs

Alternative for Ubuntu This is knowledge I gained from a whole lot of debugging, I’m not exactly sure about this one

  • Also add the CA as certifcate
  • When testing the openssl directly add the CA as a ca-cert file
sudo cp /usr/share/ca-certificates/outlook.dc.somesite.com.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
openssl s_client -CAfile ~/.fetchmail/certs/somesite.hh.pem -connect 10.120.20.218:995 -showcerts

Convert a .cer into a .pem file

openssl x509 -inform der -in certificate.cer -out certificate.pem

Perl

List all of a modules dependencies with just a one liner.

Requires that Devel::Modlist is installed.

perl -d -MDevel::Modlist=nocore script.pl

SSH

Realize password-less login via ssh-key from one to another system

  1. ssh root@alpha
    1. enter password
  2. ssh-keygen -t rsa -P ” -f ~/.ssh/id_dsa
  3. ssh-copy-id -i ~/.ssh/id_dsa.pub root@omega
  4. ssh root@omega (for testing)

Enable / Disable remote root access

#Set a root password
sudo passwd root
#Reverting that back:
sudo passwd -l root

Allow remote root login: in /etc/ssh/sshd_config: # PermitRootLogin yes service ssh reload

mpssh

Execute one and the same bash-command on multiple systems

mpssh -f ~/banks.txt -v "uptime"

Firewall

Selinux

Some quick commands:

# check information comming from selinux:
tail /var/log/audit/audit.log
# audit2allow tool is part of the following package (turns audit logs to allow-policies):
yum install setroubleshoot
# increase loglevel:
semodule -DB

vSphere

Increase VM disk-space

  1. Delete snapshots (need permission of people in charge for this)
  2. Using the vsphere interface, add a harddisk-device to the system
  3. Identify the new harddisk (a reboot might be required) 
    fdisk -l
# The disk that doesnt have a valid partition table is the one we just added
    • SCSI bus might need to be rescanned, therefor do the following: 
      echo "- - -" > /sys/class/scsi_host/hostX/scan #do this for all hosts in hopes to find the correct one
  4. Format the identified disk (optional, disk partition is not necessary for data-only disks) 
    fdisk /dev/identified_disk

    what follows is an interactive dialog

    • “n” for “new Partition”
    • “p” for “primary partition”
    • “1” for partition number, since so far we dont have any partition number on this disk
    • first cylinder: “enter”
    • last cylinder: “enter”
    • “t” to change the partitinos system ID, in this case “1” will be set automatically
    • Hex Code: 8e for Linux LVM
    • “w” to write the changes and exit
    • Alternatively you can also use cfdisk, for me cfdisk was able to create a proper partition while fdisk couldnt
  5. Check on those changes by firing um fdisk again 
    fdisk -l
  6. Add the newly created disk to the volumegroup
    • pvcreate <disk>
    • vgdisplay
    • vgextend <volumegroup> <disk>
    • pvscan
  7. Add the added space to the volumepath
    • Extend the volumepath 
      lvdisplay
lvextend <volumepath> <disk>
# alternatively:
lvextend -l +100%FREE -n <volumepath>
  8. Extend the filesystem. For extX file systems fire up the following: 
    resize2fs <volumepath>

Eth0 configuration after vmware clone

vim /etc/udev/rules.d/70-persistent-net.rules be carefull when more than one adapter is present. otherwise: delete the first block and change the last word in the second block to from eth1 eth0

Basics

Handy tool to analyze full disks: ncdu

/bin/bash

ls

Only display the filesize and filename

ls -lh | awk '{print $5 "\t" $9}'

Bash colortable

Black 0;30 Dark Gray 1;30 Blue 0;34 Light Blue 1;34 Green 0;32 Light Green 1;32 Cyan 0;36 Light Cyan 1;36 Red 0;31 Light Red 1;31 Purple 0;35 Light Purple 1;35 Brown 0;33 Yellow 1;33 Light Gray 0;37 White 1;37

Colorful motd (in red)

In ubuntu other motds may come from /etc/update-motd/

echo -en "\033[1;34m" > /etc/motd
echo "Text of your motd file....." >> /etc/motd
echo -en "\033[0m" >> /etc/motd

Change bash prompt expression

Put one of the following settings in ~/.bashrc

# yellow (for non-root users)
export PS1='\[\033[33m\]\u@\H:\w \[\033[m\]'
# red (for root)
export PS1='\[\033[1;31m\]\u@\H:\w \[\033[m\]'

More good examples

Open process in background

There are multiple solutions for this, it might be interesting looking into this indepth

firefox & disown
firefox &
firefox #now press C-z

Change a network config safely

Use the “at”-tool to reset the config after 5 minutes (at-jobs). Useful for ipconfig or firewall config-changes to ensure connectivity.

Change the timezone

cp /usr/share/zoneinfo/Europe/Berlin /etc/localtime

Check the privileges or permissions of a user

List the privileges like this:

sudo -l -U myuser

How to read the syntax: http://toroid.org/sudoers-syntax

Copy files between systems (rsync)

more useful examples: http://www.tecmint.com/rsync-local-remote-file-synchronization-commands/

rsync -avz source root@192.168.0.1:/destination/

Good ASCII Art generators

Cronjobs

Warning! Cronjobs may restart every minute and as a side effect send report-mails if an error occurs.

Sample cronjob:

59 23 * * * /home/john/bin/backup.sh > /home/john/logs/backup.log 2>&1

Process substitution

The normal way:

curl http://somesite/file1 > file1
curl http://somesite/file2 > file2
diff file1 file2

Using process substitution we can have this way faster:

diff <(curl http://somesite/file1) <(curl http://somesite/file2)

Process substitution gives you similar capabilities to piping. Except piping only allows you to pipe the output from a single command into another. In the diff scenario, we need to pipe the output from mutltiple commands into another. And that’s what process substitution allows us to do.

Filter a log-file for errors / warnings

grep -iE "(warning|error|critical)" /var/log/fetchmail.log
# -i flag to ignore the case
# -E flag to enable regular expressions

Find out Mac-Address for a different System

Prerequesite: Have another system in the same subnet (we call it System A)

#From system A
arp -a
nhhrvl038 (10.120.38.254) at 00:24:a8:68:c3:00 [ether] on eth0
#Ping them once from System A
ping system_b_IP
#Again from system A
arp -a
nhhrvl038 (10.120.38.254) at 00:24:a8:68:c3:00 [ether] on eth0
? (system_b_IP) at 00:0c:29:46:b5:55 [ether] on eth0

find, grep or sed goodies

find -type f -exec sed -i -e 's/sqlplus/sqlplus64/g' {} \;
#search for string in current directory
grep -R 'string' dir/
# fill in a specific configuration:
sudo -E sed -i -e "s/password_secret =.*/password_secret = $(pwgen -s 128 1)/" /etc/graylog/server/server.conf

Install cpan modules with dependencies

perl -MCPAN -e 'my $c = "CPAN::HandleConfig";
$c->load(doit => 1, autoconfig => 1);
$c->edit(prerequisites_policy => "follow"); $c->edit(build_requires_install_policy => "yes");
$c->commit'

Logrotate

Example from AGerler:

cat /etc/logrotate.d/fetchmail
/var/log/fetchmail.log {

weekly
create 0644 rtracker adm
rotate 4
compress

delaycompress
}
  • weekly: time interval
  • create: chmod parameter, owner und group
  • rotate: Amount of “rotates” a logfile persists before it gets deleted
  • compress: whether the log file should be zipped or not
  • delaycompress: the log only gets zipped after the rotation has been completed (file-handle reasons)

Shrink a logfile

Reduce a logfile to its last 200 lines while preserving the access rights.

logfile="/path/to/logfile"; echo "$(tail -n 200 $logfile)" > $logfile

Mount a cd

mount -t iso9660 /dev/scd0 /media/cdrom/

Port Analysis

Look at opened ports:

netstat -tulpen

Look at the processes behind them (actually active ports):

# Use a grep filter here because you will be getting way too much info
netstat -tapen | grep

Rename filenames coming from Windows users

Lowercase filenames

for i in *; do mv $i `echo $i | tr [:upper:] [:lower:]`; done

Replace blanks in filenames with _

for i in *; do mv "${i}" `echo ${i} | sed 's/ /_/g'`; done

Screen

Howto install screen on Oracle Linux 6.X

  1. Download final package sources from here: http://ftp.gnu.org/gnu/screen/
  2. Install required packages 
    yum remove screen # remove the preinstalled screen version
yum install ncurses-devel # fixes the termcap error you might get
yum install gcc # fixes a whole bunch of requirement errors

Show all processes that run under a certain user

ps -ef | grep certain_user

Unpack a .tar.gz file

tar -xzf datei.tar.gz
  • -x extract
  • -f file to be unpacked
  • -z first gzip, then tar

Debugging tools

http://brendangregg.com/ebpf.html

Commands helpful for performance analysis

Using the terminal:

df -h #check if disk-space is available
df -i #show inodes as additional information
free -mh #show RAM and swap
lscpu  # Show CPU information (number of cores, etc.)
mytop -u root --prompt # current mysql performance, inklusive Sessions:
vmstat 2 10 # Gives hints about CPU and memory during execution of performance-impaired process
iostat -xk 2 12 # Gives hints about I/O during execution of performance-impaired process
cat /proc/meminfo # Infos about processes in general
htop # Which processes take the most storage? Careful, green means actual space, yellow means provisioned

Using vSphere:

  • Check the VMs “Perfomance”-Tab, the CPU and RAM specifically
  • Check the Hypervisors “Performance”-Tab (sort by hosts and clusters)
  • Check the used Storages for Warnings

A big tool which does it all (but may be complicated to use):

sar
ksar # graphical interface

Debugging of Binaries

To debug binaries that misbehave and dont give obvious logs try the following:

  • Look in the system messages log (/var/log/messages)
  • Use strace and look out for file permission errors which might cause the binary to stop 
    strace -pid <PID> -f
  • Debug the binary with gdb (u are going pretty deep in here)

Monitor a process that times out

strace -f pid-file (child-threads followen) (not confirmed)

Short of that, you can always look in the process init script. For instance, the SSH daemon is started with the script in /etc/init.d/sshd. Sometimes the PID will be defined there (search for pid, PID, PIDFILE, PID_FILE, etc.). For anything that sources /etc/init.d/functions, the PID will live in /var/run/*.pid.