Recently I had a co-worker do an oopsie on a server he was running a script on. He, actually wisely, executed a script on a screen within his terminal while connected to a production server. Now, this is all standard, what isn’t standard is that he somehow managed to overwrite the script (while in execution) on another terminal with a one-liner.

This script took a while to run so it was still running after he reported the issue to me. This is atleast 2 days of work from him, a guy that actually knows what he’s doing (unlike me :D ).

So I remembered from my old days that you could output a script from memory using GDB as debugger, so after a few man searches and a few hairs pulled out of my head (maybe a few from his head) and found a very useful command from within GDB to help us with this.

I can’t give you the actual script recovery, however I can replicate it.

Testing script

Assuming you have GDB we can replicate this procedure, if you don’t, you’re dead on the water. Just kidding, you can install it on whatever distro with the package gdb.

Let’s start with a very simple script within a screen:

#!/bin/bash

while true; do
    sleep 1; echo "Running..."
done

In the meantime we can leave the screen with Ctrl+A+D and then proceed to do some process debugging.

PIDs everywhere

Let’s start with getting the PID for the process:

ps -ef | grep bash

In this case we find the following output:

root       436   410  0 19:14 pts/0    00:00:00 /bin/bash ./while.sh

Debugging our way into the script

We want the PID number which is 436, so we attach ourselves to the process with our favorite open source debugger:

gdb -p 436

Now what we’re attached we enter the gdb prompt that will pause the process excecution while we’re attached, take this into account as it prevents a lot of stress thinking the process is going to end and lose everything we’re trying to rescue. Also, generate a core file:

(gdb) generate-core-file

Which will generate, you guessed it, a core file, with the output: Saved corefile core.436. This core file has everything in-memory regarding the process in execution, including our bash script. However we don’t want to suddenly stop the execution of the process so we detach from it using the command (insert obviousness here):

(gdb) detach

And we exit the debugger:

(gdb) quit

Analizing

This output is a binary file that isn’t stored as clear text, if we try to use cat to outpit it’s contents the system will say something like cat: write error: Input/output error so we use strings to make it usable as clear text (don’t have it? install it):

strings core.436 > rescuescript.sh

And within the dump we can find our script in clear text, however don’t be so hopeful and happy yet, core files can have a lot of data that isn’t useful for us, so be patient!. Anyway here’s the script we found on the core file (on line 10509, hence the need for patience):

#!/bin/bash
while true; do
    sleep 1; echo "Running..."
done

Et voilà!

Hope you don’t ever need to use this. Good luck if you do!