SadServer solution for https://sadservers.com/scenario/melbourne

Reviewing the code shows me that part Content-Length contains 0 size. So I set the proper size of the code (that is "Hello, World\n" size). And then "sudo systemctl restart gunicorn.service" and "sudo systemctl restart gunicorn.socket" The final code looks like this:

Next issue is with Nginx - inside of "default" config we have file that ends with .socket, and in /run/gunicorn.sock - it use .sock Replace this and restart Nginx "sudo systemctl restart nginx"

Final test:

SadServer - Oaxaca solution for https://sadservers.com/scenario/oaxaca

So, the process is started with the current bash - and it has FD (file descriptor) 77. In the picture, you can see "FD/77w". File descriptor usually usage is: 0 for stdin, 1 for stdout, 2 for stderr, and 255 for bash shell - it points to /dev/tty or /dev/pts/N - where is N number. The main process is bash shell and the subprocess is this FD/77 - so by killing the main process we are doomed to destroy our connections. if we run "lsof somefile" - it will show us our bash shell and under /proc/[PID of shell]/fd/77 we have a symbolic link to /home/admin/somefile.

to release the file descriptor we need to re-use the same number with the command: eval "exec 77>&-"

.bashrc contains the running command:

symbolic link to file:

Unusual task with no points - are you in VM or Docker shell? Simple solution could be this:

SadServer Salt solution URL: https://sadservers.com/scenario/salta

After logging into a server - notice that port 8888 is used. Missing tool lsof, I install with "sudo apt install lsof" and review what process is using port 8888. Nginx was used so I stopped the process with "sudo systemctl stop nginx"

Inside of Dockerfile - found missing proper port 8888 (it was written 8880) and for CMD there was "serve.js" instead of "server.js" - a local file in the same directory.

When the fix was solved, the docker container is built with cmd: "sudo docker build -t sampleapp:v1 . "

running app with "sudo docker run -p 8888:8888 sampleapp:v1" and the task is done

Solution for Cape Town task from URL: https://sadservers.com/scenario/capetown

After logging into the server, there is no working nginx.

Examine details on why nginx does not work show me the first line containing ";". So I removed ";" from the nginx file and nginx does not yet work.

After examining the error log - I was able to see and spot file limits.

After viewing /proc/[pid]/limits - I spot this (Max open files 10)

I did check limits for user www-data, as well as other things (fs.file-max, others)

In the end - Maybe systemD has some limitations per process.

After reading the .service file from systemD

Add # on the start of the line, reload the system daemon, and restart nginx and it works!

For this task, it takes 30 min to solve

The task was to find out how to run a web server on port 80 and to check if is working properly. URL: https://sadservers.com/scenario/tokyo

After checking logs (apache, systems, etc) and finally running iptables - saw HTTP blocked in iptables. So running iptables -F solve this issue (I also added a path for apache /var/www/html/ inside of apache.conf FYI)

SadServer Manhattan - medium, url: https://sadservers.com/scenario/manhattan My first medium task and solution. When I log in - running just

sudo systemctl restart postgresql@14-main.service

The issue is in these lines: "no space left on device"

After running df -h, notice 100% usage for /opt/pgdata/ Removing files that do not need it at all - solved this issue.

Solution

SadServer Apia task url: https://sadservers.com/scenario/apia

Simple file-level task. Using md5sum and diff only.

Compare all files, find unusual md5 hash:

md5sum files

Use diff command, add word in solution, md5sum to check: final commands

SadServer task Bilbao url: https://sadservers.com/scenario/bilbao

After login and inspection of the issue, we get this picture:

Pod status

After googling I found an issue with nodeSelector - so I removed it from the manifest.yml file:

Remove nodeSelector

After removing - it needs to delete all pods, re-run manifest, and check with curl: Delete pods & run yaml file

Solution for Bucharest

Bucharest - url for task: https://sadservers.com/scenario/bucharest File edit:

If we run command:

PGPASSWORD=app1user psql -h 127.0.0.1 -d app1 -U app1user -c '\q'

We could see issue with access and file named "pg_hba.conf " If we open file, and we could see next lines:

host    all             all             all                     reject
host    all             all             all                     reject

If we dig manual for file - we could find that "reject" do that - reject any connection. So replacing with md5 word reject (use sudo for editing file and restarting services), and restart services with

sudo systemctl restart postgresql@13-main.service

LHasa solution for url: https://sadservers.com/scenario/lhasa

In this case we could use awk to sum all numbers in second columns and divide by number of entries ( 100 of them) So 520.4 / 100 -> 5.20

awk '{sum+=$2;} END{print sum;}' scores.txt

Task at: https://sadservers.com/scenario/taipei

Inside of this task we have port knocking, very famous solution to bring protection to access for certain port. More information on this: https://wiki.archlinux.org/title/Port_knocking

One simple way to unlock port 80 is to use nmap 2-3 times on all ports to open port 80:

nmap -Pn -p 1-65535 localhost

Famous port knocking with nmap tool

Solution for: https://sadservers.com/scenario/command-line-murders

File people contain all names, what we need is to find md5 hash of real name to add into mysolution file.

First, command to extract all people names:

awk '{print $1" "$2}' people >> find

After that we need to extract proper name with md5 hash from file "find", so use command:

while read -r col1 ; do
   echo $col1  "$(echo $col1 | md5sum )" | grep 9bba101c7369f49ca890ea96aa242dd5
done < find

and here you go your killer name:

Killer name

Solution for: https://sadservers.com/scenario/santiago

We have 3 files, each contains word "Alice". To count how many there is lines with word Alice

grep Alice *.txt | wc -l

After that we could find file with only once with:

grep Alice *.txt | cut  -d ":" -f 1 | sort | uniq -c | sort -r

Solution for Saskatoon Command:

awk '{print $1}' access.log | sort | uniq -c | sort -r | head -20

With awk, we pickup first field from the line - that is IP address. Second command - make sort this lines so next one command uniq could easy count (that is -c) and sort -r will set reverse list from highest to lower counts of IPs

Count of IPs from access log

For the task https://sadservers.com/scenario/saint-john there is easy solution by usage of tool called lsof (short for list list open files) where man pages https://man7.org/linux/man-pages/man8/lsof.8.html gives details of usage. Take notice here: I use command kill -9, but in case of important tasks like email servers and similar where are data valuable please use kill -15

Solution in one picture:

I'm solving SadServers Challenges!

I've decided to dive into the world of SadServers challenges (https://sadservers.com/scenarios)! This platform offers a variety of system administrator scenarios that test your troubleshooting skills and Linux knowledge.

With over 26 years of experience, I've encountered a wide range of Linux issues, including the infamous "Out of Memory" (OOM) problems with drivers.

Here's the exciting part:

I'll be tackling these challenges and sharing my solutions right here! The first one will be published today, June 7th, 2024, and I'll keep this page updated with links to all my future solutions.

Stay tuned for some in-depth troubleshooting and Linux problem-solving!

Easy

Saint John - Easy - solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-saint-john-task

Saskatoon - Easy solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-saskatoon

Santiago - Easy solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-santiago

Command line murderers - Easy solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-command-line-murderes

Taipei - Easy solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-taipei

Lhasa - Easy math solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-lhasa

Bucharest - Easy, Postgres solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-bucharest

Bilbao - Easy Kubernetes issue solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-biblao-k8s-task

Apia - Easy file tools usage solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-apia-task

Medium

Manhattan - medium, solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-manhattan-medium

Tokyio - medium, solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-tokyio-solution

Cape town - medium, solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-cape-town

Salta - medium, solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-salta-solution

Venice - medium, solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-venice

Oaxaca - medium, solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-oaxaca

Melbourn - medium, solution: https://www.vladimircicovic.com/2024/06/sadserver-solutions-melbourne

Optimizacija i ubrzavanje bloga uz pomoć Geo DNS i reverznih proksija

digitalocean image

Vikend eksperiment jeste postavljanje više PoP (Point of Presence. Više na https://www.cachefly.com/news/why-points-of-presence-pops-are-pivotal-for-optimal-cdn-performance/) servera kako bi lag/kašnjenje bilo manje.

Koristio sam gotove servise: EasyDNS i Linode Cloud.

EasyDNS nudi GeoDNS za 9$ mjesečno. Moguće je prema geografskoj lokaciji odrediti koji DNS zapis tj povratni odgovor na DNS upit (od A recorda pa dalje). Sa Linodom je moguće "kopirati" na više lokacija širom svijeta (paralelno) kada se sredi prvi box/server. Linode mjenja IP adresu servera koji se klonira a sve ostalo ostaje kako jeste (config, šifre, ključevi, SSL sertifikati, ostalo)

Tako da podesimo 1 server i uradimo kloniranje na više mjesta i samim tim napravimo PoP (https://www.cachefly.com/news/why-points-of-presence-pops-are-pivotal-for-optimal-cdn-performance/)

prije proxy servera

poslije dodavanja Geo DNS i reversnih proxy servera

Testiranje uz pomoc: https://www.dotcom-tools.com/website-speed-test

Geo DNS Pool

Koraci koje trebamo uraditi su sljedeći:

• Kreirati bazni reverse Nginx proxy
• Klonirati na različite lokacije
• Testirati uz pomoć Curl-a
• Podesiti GeoDNS na EasyDNS
• Testirati DNS propagaciju i web prisutnost

Nginx reverse proxy

Konfiguracija za revezni proxy (neću davati druge nepotrebne informacije):

http {

   # putanja za /cache mora biti kreirana i podešena za nginx usera/grupu www-data

   # levels - do kojeg nivoa će ići poddirektorijumi za kesiranje

  # Broj objekata koje će sačuvati u kešu - 10m

  # maksimalna veličina fajla koji može biti keširan - 1g

proxy_cache_path /cache levels=1:2 keys_zone=m_cache:10m max_size=1g;

   server {

           location / {

                proxy_cache m_cache;

                proxy_cache_valid 200 302 120m;   
                # vremenski koliko dugo - 120 minuta

                proxy_cache_valid 404 1m;     

                proxy_pass https://9.8.7.1;   
                # ovdje je 9.8.7.1 izvorni sajt

             }

}

Kako testirati da li je proxy podešen

Ako imamo Linux komandnu liniju onda izvršite komandu:

curl -H "host: www.vladimircicovic.com" -k https://172.232.148.193/

Ovdje koristimo host header i IP adresu gdje Curl upucujemo da ignorise SSL/TLS sertifikat i validnost (porediće IP adresu i domenu u SSL sertifikatu, i zatim odbiti da pošalje zahtjev, zato dodajemo -k opciju) Sa ovom komandom bi trebalo da vidimo početnu stranicu.

Kako testirati DNS za odredjene zemlje

Jedan od bržih načina da se vidi DNS propagacija (https://www.digicert.com/faq/dns/what-is-dns-propagation) jeste sajt: https://dnschecker.org/#A/www.vladimircicovic.com

Drugi način ako imate Linux komandnu liniju:

dig +short A www.vladimircicovic.com @118.127.62.178 

172.105.181.107

Gdje je javni DNS server za Australiju dostupan sa: https://public-dns.info/nameserver/au.html

IP adresa 172.105.181.107 je za Australiju i druge zemlje Okeanije.

Kako testirati web pristup za odredjene zemlje

Najbolje je uz pomoć sajta https://www.dotcom-tools.com/website-speed-test ali postoje i slični gdje je moguće testirati.

Optimizacija web stranice i ograničavanje sadržaja radi bržeg učitavanja

Npr možete prikazati 5, 10 zadnjih postova na svom sajtu. Tako učitavanje bude jako teško za klijenta. Jedan od načina jeste ograničavanjem količine broja postova na prvoj strani.

Alat za testiranje: https://pagespeed.web.dev/

Prije ograničavanja sadržaja:

Poslije ograničavanja sadržaja: