15. Debugging
Most unexpected behavior can be debugged by using the parameter --debug
.
If you ever encounter a situation in which:
THOR doesn't produce an alert on a known malicious element
THOR exits with an error
THOR takes a long time or unexpected short time on elements
15.1. Collecting a Diagnostcs Pack
THOR Util comes with the functionality to collect a diagnostics pack for THOR scans. This is helpful if a scan is taking more time as expected or if THOR exits unexpectedly. More detail can be found in the diagnostics section of THOR Util.
15.2. Debugging Examples
Then try scanning that specific element with the --debug
parameter set.
To run only a certain module use:
C:\nextron\thor>thor64.exe -a Mutex
C:\nextron\thor>thor64.exe -a FileScan
C:\nextron\thor>thor64.exe -a Eventlog
You can try to reduce the scope of a module even further by using lookbacks
C:\nextron\thor>thor64.exe -a Eventlog --lookback 3
C:\nextron\thor>thor64.exe -a FileScan -p C:\Windows\System32 --globallookback --lookback 1
To find out why a certain file couldn't be detected, use
--debug
with --printall
and try to switch into --intense
mode.
C:\nextron\thor>thor64.exe -a Filescan -p C:\testfolder --debug --printall
C:\nextron\thor>thor64.exe -a Filescan -p C:\testfolder --debug --printall --intense
If it has been detected in --intense
mode but not in default mode,
the file extension or the magic header is most likely the problem.
You can adjust max_file_size
in ./config/thor.yml
or add a
magic header in ./signatures/misc/file-type-signatures.cfg
.
15.3. Finding Bottlenecks
You may get the error message MODULE: RuntimeWatcher MESSAGE: Maximum runtime has exceeded, killing THOR
or encounter very slow or never-ending scans.
You can check the statistics table in thor10.db
on that end
system after a scan to determine the last element or elements that took
a long time to process.
We recommend using: https://sqlitebrowser.org/
The THOR DB is located at: C:\ProgramData\thor\thor10.db
.

15.4. Most Frequent Causes of Missing Alerts
15.4.1. THOR didn't scan file due to file size restrictions
Solution: Use --max_file_size
parameter or set permanently in
config file ./config/thor.yml
. Also note that in lab scanning
mode the default value is much bigger (--max_file_size_intense
)
15.4.2. THOR didn’t scan the file due to a skipped deeper inspection
This can be caused by two reasons:
The magic header of that file is not in the list of interesting magic
headers (see ./signatures/misc/file-type-signatures.cfg
) AND file
doesn't have a relevant file extension (.asp, .vbs, .ps, .ps1, .rar,
.tmp, .bas, .bat, .chm, .cmd, .com, .cpl, .crt, .dll, .exe, .hta, .js,
.lnk, .msc, .ocx, .pcd, .pif, .pot, .pdf, .reg, .scr, .sct, .sys, .url,
.vb, .vbe, .vbs, .wsc, .wsf, .wsh, .ct, .t, .input, .war, .jsp, .php,
.asp, .aspx, .doc, .docx, .pdf, .xls, .xlsx, .ppt, .pptx, .tmp, .log,
.dump, .pwd, .w, .txt, .conf, .cfg, .conf, .config, .psd1, .psm1,
.ps1xml, .clixml, .psc1, .pssc, .pl, .www, .rdp, .jar, .docm, .ace,
.job, .temp, .plg, .asm)
Solution: Use an intense scanning mode for that folder (--intense
)
or add the magic header to file-type-signatures.cfg
Warning
This file gets overwritten with an update; Intense scanning mode threatens the scan and system stability!
15.4.3. THOR fails to initialize custom rules with the correct type
It happens very often that users that prepare custom IOCs or YARA rules forget to include the correct keyword in the filename of the IOC or YARA rule file.
The correct use of keywords is described in the chapters Simple IOCs for IOCs and Specific YARA Rules for YARA rules.
A wrong or missing keyword leads to situations in which a file that contains YARA rules that are meant to be applied to log files, doesn't contain a "log" keyword in it's name.
You can review a correct initialization in the command line output or log file.
Info Adding rule set from my-log-rules.yar as 'log' type
Using the keyword c2 for C2 IOCs in a filename should result in a line like the following:
Info Reading iocs from /tmp/thor10/custom-signatures/iocs/my-c2-iocs.txt as 'domains' type
15.5. Most Frequent Causes of Frozen Scans
Whenever THOR stops / pauses without any traceback or panic message and no error
Usually the following sources are responsible (descending order, by frequency):
Antivirus or EDR suspends the THOR process (>98%)
A frozen command line window due to Windows "Quick Edit Mode" (<1%)
A constant high system load that causes THOR to stay back and wait for an idling CPU (<1%)
The sensation of a stalled scan that is actually running (<1%)
15.5.1. Antivirus or EDR suspends THOR
In more than 98% of the cases, an Antivirus or EDR is responsible for a stalled process. Especially McAfee AV/EDR is a well-known source of issues. This is caused by the different dialogues in which exceptions have to be defined and the fact certain kinds of blocks cannot be found in any logs.
If a THOR scans stalls in one of these modules, a Antivirus or EDR interaction is highly likely:
Mutex
Events
NamedPipes
ShimCache
ProcessCheck
Solution: Review all possible exclusions in your AV / EDR and add the THOR folder to the exclusion list
15.5.2. Windows Quick Edit Mode
Since Windows 10, the Windows command line window has the so-called "Quick Edit Mode" enabled by default. This mode causes a behavior that can lead to a paused THOR scan process. Whenever a user switches the active windows from the cmd.exe to a different application, e.g. Windows Explorer, and clicks back into the command line window, the running process automatically gets suspended. A user has to press "Enter" to resume the suspended process. As the progress indicator of THOR isn't always changing, this could lead to the impression that the scan paused by itself.
See this StackOverflow post for more details.
Solution: Press "Enter" in the command line window
15.5.3. Constant High System Load
Since THOR automatically sets a low process priority a scan can slow down to a level that appears to be paused or suspended on systems that are under a constant high load.
Solution: You can avoid this behaviour by using the --nolowprio
flag. Be aware
that scans on a system with a constant high CPU load take longer than on other systems
and could slow down the processes that would otherwise take all the CPU capacity.
15.5.4. The Sensation of a Stalled Scan
Under certain circumstances the scan may appear stalled but is still running. You can always interrupt a scan using CTRL+C that brings THOR into the interrupt menu in which you can see the currently scanned element. In case of the "FileScan" module, this is a file or folder. In case of the "EventLog" module, this is an event with an ID. If you resume the scan by pressing "C" and interrupt it again a few minutes later, you should see another element in the interrupt menu.
If THOR still processes the same element for several hours, we recommend checking that element (size, format, access rights, location).
Solution: Check progress using the interrupt menu (CTRL+C)
15.6. Most Frequent Causes of Failed Scans
15.6.1. External Processes Terminating THOR
Whenever THOR dies without any traceback or panic message and no error message in the log file, an external process terminated the THOR process.
Usually the four following sources are responsible (descending order, by frequency):
Antivirus or EDR killed the THOR process
A user killed the THOR process
A management solution that noticed a high CPU load caused by the THOR process killed it
Attackers killed the THOR process
Note: A process termination that always happens at the same element is a sign for an Antivirus or EDR detection.
15.6.2. Insufficient Free Memory
fatal error: out of memory
Probable causes:
Other processes consume a lot of memory
THOR's scanning of certain elements requires a lot of memory
You've set ulimit values that are too restrictive
You are using the wrong THOR version for your architecture
You've activated a feature that consumes a lot of memory (e.g.
--mft
or--intense
)
Whenever THOR recognizes a low amount of free memory, THOR checks the top 3 memory consumers on the system and includes them in the log message that it writes before exiting.
You could try running THOR in Soft Mode (--soft
) in which modules and
features that require a lot of memory are deactivated.
Using the 32bit binary of thor named thor.exe
on a 64bit system
can lead to interrupted scans with this error message. The 32bit binary
isn't able to address as much memory as the 64bit version. Always make
sure to use the correct THOR version for the respective architecture.
Several ulimits might cause THOR to terminate if they are too restrictive, including:
locked-in-memory size
address space
number of open file descriptors
maximum data size
If you are certain your machine has sufficient RAM, check your ulimits with
ulimit -a
and try to rerun the scan with increased limits, if necessary. The man page for the ulimits configuration size gives a full overview over the limits and how to persistently modify them.
15.7. Help Us With The Debugging
If you cannot find the source of a problem, please contact us using the support@nextron-systems.com email address.
You can help us find and debug the problem as quickly as possible by providing the following information.
15.7.1. Which THOR version do you use?
Tell us which THOR version you are using:
For which operating system (Windows, Linux, macOS, AIX)
For which architecture (32bit, 64bit)
Run thor --version
and copy the resulting text into the email.
On Windows:
C:\thor>thor64.exe --version
THOR 10.6.6
Build bea8066 (2021-04-27 14:32:40)
YARA 4.0.5
PE-Sieve 0.2.8.5
OpenSSL 1.1.1j
Signature Database 2021/05/03-150936
Sigma Database 0.19.1-1749-g2f12c5c5
On Linux:
user@desktop:~$ ./thor-linux-64 --version
THOR 10.6.6
Build bea8066 (2021-04-27 14:32:40)
YARA 4.0.5
PE-Sieve 0.2.8.5
OpenSSL 1.1.1j
Signature Database 2021/05/03-150936
Sigma Database 0.19.1-1749-g2f12c5c5
On Mac OS:
user@macos:~$ ./thor-macosx --version
THOR 10.6.6
Build bea8066 (2021-04-27 14:32:40)
YARA 4.0.5
PE-Sieve 0.2.8.5
OpenSSL 1.1.1j
Signature Database 2021/05/03-150936
Sigma Database 0.19.1-1749-g2f12c5c5
This should produce a message like this:
15.7.2. What is the target platform that THOR fails on?
Please provide the output of the following commands.
On Windows:
C:\Users\user>systeminfo > systeminfo.txt
On Linux:
user@desktop:~$ uname -a
On Mac OS:
user@macos:~$ system_profiler -detailLevel mini > system_profile.txt
15.7.3. Which command line arguments do you use?
Please provide a complete list of command line arguments that you've used when the error occurred.
C:\thor>thor64.exe --quick -e D:\logs -p C:\Windows\System32
15.7.4. Provide the log of a scan with --debug flag
The most important element is a scan log of a scan with set --debug
flag.
The easiest way is to run the scan exactly as you've run it when the
problem occurred adding the --debug
command line flag.
E.g.
C:\thor>thor64.exe --quick -e D:\logs -p C:\Windows\System32 --debug
If you're able to pinpoint the error to a certain module, you could limit the scan to that module to get to the problematic element more quickly, e.g.
C:\thor>thor64.exe -a Rootkit --debug
After the scan you will find the normal text log (*.txt) in the program folder. It is okay to replace confidential information like the hostname or IP addresses.
Note: The debug log files can be pretty big, so please compress the file before submitting it to us. Normal log files have a size between 1 and 4 MB. Scans started with the --debug flag typically have sizes of 30-200 MB. The compression ratio is typically between 2-4%, so a compressed file shouldn't be larger than 10 MB.
15.7.5. Provide a Screenshot (Optional)
Sometimes errors lead to panics of the executables, which causes the situation that the relevant log lines don't appear in the log file. In these cases, please also create a screenshot of a panic shown in the command line window.
15.7.6. Provide the THOR database (Optional)
The THOR DB helps us debug situations in which
the THOR scan timed out or didn't complete at all. It contains statistics on the
run time of all used modules and the durations of all folders up to the second
folder level from the root of a partition. (e.g. C:\Windows\SysWow64
).
The default location of that file is:
Windows:
C:\ProgramData\thor\thor10.db
Linux/macOS:
/var/lib/thor/thor10.db
Please provide that file in situations in which:
THOR exceeded its maximum run time
THOR froze and didn't complete a scan for days
THOR scans take too long for the selected scan targets
15.7.7. Further Notes
If the files are too big to send, even after compression, please contact us and you'll receive a file upload link that you can use
If a certain file or element (eventlog, registry hive) caused the issue, please check if you can provide that file or element for our analysis