Detection of Hooked Syscalls in ntdll.dll

Introduction

Detecting modifications in critical system functions, such as syscalls, is a valuable technique for identifying potential interference from security solutions like EDRs (Endpoint Detection and Response) or even malware trying to alter system behavior. Syscalls (System Calls) are functions that allow programs to request services from the kernel, such as file access, process control, and networking. These functions typically reside in the ntdll.dll library on Windows.

Advanced security solutions often intercept or redirect these calls by using hooks, a technique that modifies a function to redirect its execution elsewhere before or during its normal operation. A hook can be introduced using instructions like jmp or call, which alter the flow of execution to custom code.

In this article, we will discuss how to detect hooked syscalls in ntdll.dll by analyzing the assembly instructions at the beginning of functions. This will allow us to verify whether functions have been altered or remain in their original state.

How Syscall Hooking Works

Hooks usually work by redirecting a function’s flow, inserting a jump (jmp) or call (call) instruction at the beginning of the original function. This is done to monitor or modify the behavior of the function without the original process being aware of the change.

In the context of Windows, syscall functions begin with a well-defined prologue, a sequence of bytes that start the execution of a syscall. This standard prologue can be used as a reference to check if the function has been altered. The common syscall prologue in ntdll.dll begins with the following bytes:

4c 8b d1 b8

Any deviation from this pattern, especially with a jmp or call instruction, may indicate that the function has been hooked.

Code to Detect Hooked Syscalls

Below is a C++ code snippet that detects modifications to exported functions in ntdll.dll without relying on the psapi.h library. The code looks for functions that start with the typical syscall prologue and flags functions that might have been modified (hooked) by instructions such as jmp or call.

C++ Code:

#include <iostream>
#include <Windows.h>

bool IsHookedFunction(PVOID functionAddress)
{
    // Syscall stubs start with these bytes in ntdll
    unsigned char syscallPrologue[4] = { 0x4c, 0x8b, 0xd1, 0xb8 };

    // Check if the first few bytes match the expected prologue
    if (memcmp(functionAddress, syscallPrologue, sizeof(syscallPrologue)) == 0)
    {
        return false; // Function is not hooked, matches syscall stub
    }
    
    // If it's a JMP or CALL instruction, likely a hook
    if (*(unsigned char*)functionAddress == 0xE9 || *(unsigned char*)functionAddress == 0xE8)
    {
        return true; // Function appears to be hooked
    }

    return true; // If it doesn't match the prologue or has an unexpected instruction, it's potentially hooked
}

int main()
{
    PDWORD functionAddress = nullptr;
    
    // Get ntdll base address
    HMODULE libraryBase = LoadLibraryA("ntdll");

    PIMAGE_DOS_HEADER dosHeader = (PIMAGE_DOS_HEADER)libraryBase;
    PIMAGE_NT_HEADERS imageNTHeaders = (PIMAGE_NT_HEADERS)((DWORD_PTR)libraryBase + dosHeader->e_lfanew);

    // Locate export address table
    DWORD_PTR exportDirectoryRVA = imageNTHeaders->OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_EXPORT].VirtualAddress;
    PIMAGE_EXPORT_DIRECTORY imageExportDirectory = (PIMAGE_EXPORT_DIRECTORY)((DWORD_PTR)libraryBase + exportDirectoryRVA);

    // Offsets to list of exported functions and their names
    PDWORD addressOfFunctionsRVA = (PDWORD)((DWORD_PTR)libraryBase + imageExportDirectory->AddressOfFunctions);
    PDWORD addressOfNamesRVA = (PDWORD)((DWORD_PTR)libraryBase + imageExportDirectory->AddressOfNames);
    PWORD addressOfNameOrdinalsRVA = (PWORD)((DWORD_PTR)libraryBase + imageExportDirectory->AddressOfNameOrdinals);

    // Iterate through exported functions of ntdll
    for (DWORD i = 0; i < imageExportDirectory->NumberOfNames; i++)
    {
        // Resolve exported function name
        DWORD functionNameRVA = addressOfNamesRVA[i];
        DWORD_PTR functionNameVA = (DWORD_PTR)libraryBase + functionNameRVA;
        char* functionName = (char*)functionNameVA;
        
        // Resolve exported function address
        DWORD_PTR functionAddressRVA = addressOfFunctionsRVA[addressOfNameOrdinalsRVA[i]];
        functionAddress = (PDWORD)((DWORD_PTR)libraryBase + functionAddressRVA);

        // Only interested in Nt|Zw functions
        if (strncmp(functionName, "Nt", 2) == 0 || strncmp(functionName, "Zw", 2) == 0)
        {
            if (IsHookedFunction(functionAddress))
            {
                printf("Hooked or modified: %s : %p\n", functionName, functionAddress);
            }
            else
            {
                printf("Not hooked: %s : %p\n", functionName, functionAddress);
            }
        }
    }

    return 0;
}

Code Explanation

  1. Loading ntdll.dll: The code starts by loading the ntdll.dll library using the LoadLibraryA function. This library contains the implementations of syscalls.

  2. Locating the Export Table: We use the IMAGE_DOS_HEADER and IMAGE_NT_HEADERS structures to locate the export table in ntdll.dll, where all the exported functions are listed, including the syscalls we are interested in.

  3. Iterating Through Exported Functions: The code iterates over all exported functions and focuses on those that start with "Nt" or "Zw", which are the syscalls in Windows.

  4. Hook Detection: For each syscall function, the code checks if the first bytes match the typical syscall prologue (4c 8b d1 b8). If not, it checks if the function starts with a jmp or call instruction, which could indicate that the function has been redirected (hooked).

  5. Detection Output: If a function is detected as hooked, the program prints a message indicating that the function has been altered. Otherwise, it indicates that the function remains intact.

Result:

Advantages and Limitations

Advantages:

  • Simple and Direct: This code is straightforward and does not rely on external libraries for hook detection.

  • Robust Detection: It detects common redirection instructions like jmp and call, and also verifies the typical syscall prologue.

  • Applicable to Critical Functions: The focus on syscalls is crucial for detecting changes to sensitive system functions, often targeted by security solutions and malware.

Limitations:

  • Limited to Syscall Prologues: The code only checks the first few bytes of the function to detect hooks. More advanced techniques, such as hooks inserted deeper into the function, are not detected by this simple approach.

  • Does Not Analyze Deep Modifications: If a hook is more subtly inserted by modifying instructions within the function and not at the beginning, this method may fail to detect it.

Conclusion

Detecting hooks in syscall functions is a valuable technique for ensuring the integrity of system behavior and identifying the presence of security solutions that might interfere with software execution. The presented code offers a simple and effective solution to detect changes in syscalls by checking the prologue and common redirection instructions like jmp and call. It can serve as a starting point for more advanced solutions that detect tampering in system calls.

As defensive and offensive techniques evolve, it is essential to continue improving these methods by incorporating more advanced code and memory analysis techniques to ensure even more robust detection of hooks and other modifications to syscall behavior.

PreviousWindows API Hashing to MalwareNextPrivilege Escalation

Last updated