Kernel rootkits are the tools that run in the kernel, hence making it really hard to detect. The entire operating system would be altered in the process, which would help in the process of hiding the fact that the system is compromised.
Kernel rootkits are of the most malicious nature -- they completely undermine a systems integrity and capacity for self-diagnosis. Essentially any tool that could be used for detecting such a rootkit is susceptible to false results due to syscall hijacking. Some tools however does not have to run in memory of the target system itself may benefit from running on a remote machine to avoid certain syscalls (i.e over NFS) -- this can be explained in more detail sometime. There are a many ways to subvert.
Here are some of them:
-- Directly modifying the syscall table
-- Syscall jump
-- IA32 Debug
To understand the working of kernel rootkit, it is important to understand how a kernel works. Though we talk a lot more about Linux related samples at most cases, all kernels behave the same way. Kernel is a mediator between user apps and the hardware. Hence, kernel should interact/talk-to both the user-mode apps and the hardware. To do this in a secure way, operating systems has ensured that all processes communicate with the kernel through system libraries and the system libraries sends system calls to the kernel mode, which reaches the system call table. The system call table then interacts with the text or data segment of the kernel depending on the nature of Syscall or the function to be performed. Once that is processed in the kernel, the kernel then interacts with the hardware accordingly. The following figure shows a simple interaction diagram:
In the ring terminology, kernel rootkits run on ring0. Since, ring0 is under all the other layers, monitoring kernel rootkits is almost impossible if the monitoring software is run on the same system, except for certain detection techniques that compares the different states or interacts with kernel using different stimulus signals and compares the response. Though once again, since the kernel rootkit modifies the entire operating system properties, functions and anything and everything possible to deceive a user from noticing the compromise by completely hiding itself, detection is quite hard in case of kernelland rootkits.
As said in the ring terminology section, ring0 has the highest privilege and ring3 has the least privilege. Most anti-rootkit tools that are created to detect kernel rootkits, really do not do the stuff that they were created for if the kernel rootkit does its job as it was meant to.
Kernelland rootkits are known for overwriting the entire operating system. They do this by modifying the kernel. Loadable Kernel Module [LKM] is used to expand the kernel for adding new hardware or Filesystem without a system reboot. LKM is used only when required and when they are used, they run in the memory and when done they are removed. In this process, an attacker/rootkit can run the infected process that would load the EVIL_LKM to modify the kernel. LKM runs in ring0 of the system as it is running on the kernel itself, which means that it is running on the highest privilege. To do this, the EVIL_PROCESS loads the EVIL_LKM. There are several ways to do this and there are several things that can be done once this is done. Syscall hijack is one way that we talk about in our other paper and also the Syscall modification too. Once this is loaded, the rootkits can alter the Syscall to EVIL_SYSCALL or intercept the good Syscall with EVIL_FUNCTION, depending on the technique that the rootkit chooses to use. Once this is done, the kernel access is obtained by running the code that the rootkit tries to access in the Text segment.
Another way to subvert the kernel is to alter kernel memory. /dev/kmem is used for mapping the kernel memory at run-time in Linux. In Windows systems, Hence to change the kernel at runtime, direct memory I/O could alter kmem. Even though there are patches to protect kmem, there are other ways to override the patches for the rootkits to do what they were meant to.
The syscall table is generally an array of function pointers; these function pointers can be redirected to point to functions implemented in the rootkit, often times an LKM. This type of kernel rootkit is easy to detect compared to other kernel rootkits. A hands on approach is to analyze /proc/kcore with gdb and verify the syscalls against the System.map file (A file created at kernel compilation time for debugging purposes). A simpler approach is to run a tool like ktraq, which also has the benefit of disabling this type of rootkit by resetting the syscall function pointers.
This is a more stealth method of syscall hijacking without having to directly modify the syscall table; instead the first 7 bytes of the syscall are overwritten with a jump to the new code -- i.e "movl $0x0, %eax; jmp *%eax". Keep in mind the original 7 bytes are saved and copied back into the original syscall so that it may be called within the evil syscall. This type of syscall hijacking cannot be detected by tools like ktraq, however a tool like ElfStat which analyzes the text segment of the running kernel and compares it to vmlinux can detect this method. This method is also used in kernel function hijacking (Not only syscalls).
NOTE: Our tools are listed in many sites and torrents, which makes it hard for us to track all downloads. Hence, we are listing only the total installations from our website.