Probabilistic Call Stack: A Deep Dive into Non-Deterministic Execution Paths

Abstract

This article explores the concept of Probabilistic Call Stacks (also known as Non-Deterministic Call Stacks), a technique that creates varying execution paths to reach the same final payload. By randomizing the call chain at runtime, each execution produces a unique stack signature, making pattern-based detection more challenging. This document provides a comprehensive analysis of the implementation, discusses the underlying concepts, examines code patterns, and suggests potential improvements for both offensive research and defensive detection strategies.

PoC: https://github.com/CyberSecurityUP/Probabilistic-Call-Stack-PoC

---

Table of Contents

1. Introduction

2. Understanding Call Stack Fingerprinting

3. The Probabilistic Approach

4. Architecture Overview

5. Implementation Deep Dive

6. Path Analysis

7. Stack Trace Capture Mechanism

8. Potential Improvements

9. Detection Strategies

10. Conclusion

---

1. Introduction

Endpoint Detection and Response (EDR) solutions employ various techniques to identify malicious behavior. One such technique involves analyzing the call stack at the moment suspicious API calls are made. The call stack reveals the sequence of function calls that led to a particular point in execution, providing valuable context about the code's behavior.

Traditional malware often exhibits predictable call patterns. When a payload executes, the stack trace typically shows a consistent path from the entry point to the malicious function. EDRs can fingerprint these patterns and flag executions that match known signatures.

Probabilistic Call Stacks aim to break this determinism by introducing randomness into the execution path. While the final behavior remains identical (the same payload executes), the journey to reach that payload varies with each execution, producing different stack signatures.

Use Cases

• EDR Development: Testing detection capabilities against polymorphic execution patterns

• Red Team Operations: Understanding how call stack analysis works and its limitations

• Security Research: Studying the relationship between execution paths and detection mechanisms

• Malware Analysis: Recognizing this technique when analyzing samples

---

2. Understanding Call Stack Fingerprinting

What is a Call Stack?

The call stack is a data structure that stores information about the active subroutines (functions) in a program. When function A calls function B, the return address (where to continue in A after B finishes) is pushed onto the stack. This creates a chain of return addresses representing the execution path.

Example Call Stack:
[0] payload()           <- Current function
[1] wrapper_level2()    <- Called payload()
[2] wrapper_level1()    <- Called wrapper_level2()
[3] main()              <- Called wrapper_level1()
[4] __libc_start_main() <- Runtime entry

EDR Call Stack Analysis

EDRs hook critical Windows APIs (e.g., VirtualAlloc, CreateRemoteThread, NtWriteVirtualMemory) and capture the call stack when these functions are invoked. The analysis typically involves:

1. Return Address Validation: Checking if return addresses point to legitimate code sections

2. Module Attribution: Identifying which modules (DLLs, EXEs) appear in the stack

3. Pattern Matching: Comparing the stack structure against known malicious patterns

4. Depth Analysis: Examining the call depth and presence of expected intermediate functions

The Fingerprinting Problem

When malware uses a consistent execution flow, the call stack becomes a reliable fingerprint:

Consistent Pattern (Easily Fingerprinted):
main() -> initialize() -> load_payload() -> execute() -> VirtualAlloc()

Every execution produces the same stack, making detection straightforward. The EDR simply needs to flag this specific pattern.

---

3. The Probabilistic Approach

Core Concept

Instead of a single deterministic path, we create multiple equivalent paths that all lead to the same destination. At runtime, one path is selected randomly:

              ┌─> path_A() ──────────────────┐
              │                              │
main() ──────┼─> path_B() -> helper() ──────┼──> payload()
              │                              │
              ├─> path_C() -> aux() -> ... ──┤
              │                              │
              └─> path_D() ──────────────────┘

Each path may have:

• Different nesting depths

• Different intermediate function names

• Different auxiliary operations (timing, memory, system queries)

• Different call mechanisms (direct, pointer-based, recursive)

Mathematical Perspective

If we have N distinct paths, each with potentially different characteristics, the probability of seeing the exact same call stack twice is significantly reduced. For our implementation with 13 paths, where some paths (like Path K and M) have internal variability:

• Path K: 4 possible depths (1-4 recursion levels)

• Path M: 4 possible routes (2 branch points with 2 options each)

This creates effectively: 7 fixed paths + 4 (Path K variants) + 4 (Path M variants) = 15+ unique stack signatures

---

4. Architecture Overview

Component Diagram

┌─────────────────────────────────────────────────────────────────┐
│                        MAIN ENTRY                               │
│  - Initialize RNG                                               │
│  - Configuration                                                │
│  - Run demonstration loop                                       │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                    PATH SELECTION                               │
│  - Random selection from g_wrappers[] array                     │
│  - Execute selected wrapper function                            │
└─────────────────────────────────────────────────────────────────┘
                              │
          ┌───────────────────┼───────────────────┐
          ▼                   ▼                   ▼
    ┌──────────┐        ┌──────────┐        ┌──────────┐
    │ Path A   │        │ Path H   │        │ Path M   │
    │ (Direct) │        │ (Tower)  │        │(Branch)  │
    └────┬─────┘        └────┬─────┘        └────┬─────┘
         │                   │                   │
         │              ┌────┴────┐         ┌────┴────┐
         │              ▼         │         ▼         ▼
         │         [Level 1]      │    [Branch L]  [Branch R]
         │              │         │         │         │
         │              ▼         │         └────┬────┘
         │         [Level 2]      │              │
         │              │        ...            ...
         │             ...        │              │
         │              │         │              │
         └──────────────┴─────────┴──────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                    AUXILIARY FUNCTIONS                          │
│  - aux_small_delay()      - aux_heap_operation()                │
│  - aux_get_time()         - aux_query_perf()                    │
│  - aux_thread_info()                                            │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                       PAYLOAD                                   │
│  - execute_payload()                                            │
│  - Stack trace capture (optional)                               │
│  - MessageBoxA (benign demonstration)                           │
└─────────────────────────────────────────────────────────────────┘

File Structure

probabilistic_callstack/
├── probabilistic_callstack.cpp   # Main implementation (660 lines)
├── Makefile                      # Build automation
├── build.bat                     # Windows build script
└── article_probabilistic_callstack.md  # This document

---

5. Implementation Deep Dive

5.1 Configuration and Global State

// Configuration
#define NUM_WRAPPERS 13
#define MAX_STACK_DEPTH 64
#define ENABLE_STACK_TRACE 1

// Forward declarations
typedef void (*PayloadFunc)();
void execute_payload();
void capture_and_print_stack(const char* context);

// Global execution counter for demonstration
static int g_execution_id = 0;

Analysis:

• NUM_WRAPPERS: Defines the number of available paths. Increasing this value adds more randomization potential.

• MAX_STACK_DEPTH: Maximum frames to capture during stack trace. 64 is sufficient for most scenarios.

• ENABLE_STACK_TRACE: Compile-time toggle for stack visualization. Disable in production for stealth.

• PayloadFunc: Function pointer typedef enables indirect calling mechanisms.

5.2 The Payload Function

void execute_payload() {
    printf("\n[PAYLOAD] Executing final payload (execution #%d)\n", g_execution_id);

    // Benign payload: Display a message box
    MessageBoxA(NULL,
                "Payload executed successfully!\n\nCall stack was randomized.",
                "Probabilistic Call Stack PoC",
                MB_OK | MB_ICONINFORMATION);

    printf("[PAYLOAD] Payload completed\n");
}

Analysis:

• This is the convergence point - all paths lead here

• Uses MessageBoxA as a benign, visible indicator of execution

• In real scenarios, this could be any operation: shellcode execution, API call, etc.

• The payload itself is deterministic; only the path to reach it varies

5.3 Auxiliary Functions

Auxiliary functions serve two purposes:

1. Add stack depth - More functions in the call chain

2. Add behavioral noise - Different API calls create different patterns

void aux_small_delay() {
    Sleep(rand() % 10 + 1);
}

void aux_get_time() {
    SYSTEMTIME st;
    GetSystemTime(&st);
    printf("  [AUX] System time: %02d:%02d:%02d.%03d\n",
           st.wHour, st.wMinute, st.wSecond, st.wMilliseconds);
}

void aux_heap_operation() {
    HANDLE heap = GetProcessHeap();
    void* mem = HeapAlloc(heap, HEAP_ZERO_MEMORY, 64);
    if (mem) {
        printf("  [AUX] Heap allocated at: 0x%p\n", mem);
        HeapFree(heap, 0, mem);
    }
}

void aux_query_perf() {
    LARGE_INTEGER freq, counter;
    QueryPerformanceFrequency(&freq);
    QueryPerformanceCounter(&counter);
    printf("  [AUX] Performance counter: %lld (freq: %lld)\n",
           counter.QuadPart, freq.QuadPart);
}

void aux_thread_info() {
    DWORD tid = GetCurrentThreadId();
    DWORD pid = GetCurrentProcessId();
    printf("  [AUX] PID: %lu, TID: %lu\n", pid, tid);
}

API Diversity Benefits:

Function	Windows API Used	Purpose
aux_small_delay	Sleep()	Timing variation
aux_get_time	GetSystemTime()	System interaction
aux_heap_operation	HeapAlloc/HeapFree	Memory operations
aux_query_perf	QueryPerformanceCounter	High-precision timing
aux_thread_info	GetCurrentThreadId/ProcessId	Process context

5.4 Path Selection Mechanism

typedef void (*WrapperFunc)();

WrapperFunc g_wrappers[NUM_WRAPPERS] = {
    wrapper_path_A_direct,
    wrapper_path_B_nested,
    wrapper_path_C_deep,
    wrapper_path_D_indirect,
    wrapper_path_E_entry,
    wrapper_path_F_heavy,
    wrapper_path_G_virtual,
    wrapper_path_H_tower,
    wrapper_path_I_deep6,
    wrapper_path_J_chain,
    wrapper_path_K_mixed,
    wrapper_path_L_staircase,
    wrapper_path_M_branching
};

void execute_random_path() {
    int selected = rand() % NUM_WRAPPERS;

    // Execute the selected wrapper
    g_wrappers[selected]();
}

Analysis:

• Function pointer array enables runtime path selection

• rand() % NUM_WRAPPERS provides uniform distribution

• The indirection through function pointers adds a layer of obfuscation

• Easy to extend: just add new functions to the array

---

6. Path Analysis

6.1 Path A: Direct (Minimal Depth)

void wrapper_path_A_direct() {
    printf("[PATH A] Direct execution path\n");
    aux_thread_info();
    execute_payload();
}

Stack Depth: 1 wrapper level Call Stack:

execute_payload()
wrapper_path_A_direct()
execute_random_path()
main()

Characteristics:

• Simplest path with minimal overhead

• Single auxiliary call for slight variation

• Useful as a baseline for comparison

---

6.2 Path B: Nested (2 Levels)

void wrapper_path_B_inner() {
    printf("  [PATH B] Inner wrapper\n");
    aux_get_time();
    execute_payload();
}

void wrapper_path_B_nested() {
    printf("[PATH B] Nested execution path\n");
    aux_small_delay();
    wrapper_path_B_inner();
}

Stack Depth: 2 wrapper levels Call Stack:

execute_payload()
wrapper_path_B_inner()
wrapper_path_B_nested()
execute_random_path()
main()

Characteristics:

• Introduces basic nesting concept

• Different auxiliary functions at each level

• Demonstrates how depth increases stack uniqueness

---

6.3 Path C: Deep (3 Levels)

void wrapper_path_C_level3() {
    printf("    [PATH C] Level 3\n");
    execute_payload();
}

void wrapper_path_C_level2() {
    printf("  [PATH C] Level 2\n");
    aux_query_perf();
    wrapper_path_C_level3();
}

void wrapper_path_C_deep() {
    printf("[PATH C] Deep nested path\n");
    aux_heap_operation();
    wrapper_path_C_level2();
}

Stack Depth: 3 wrapper levels Characteristics:

• Linear descent through three named levels

• Memory and timing operations mixed in

---

6.4 Path D: Indirect (Function Pointer)

void wrapper_path_D_indirect() {
    printf("[PATH D] Indirect execution via function pointer\n");
    aux_get_time();

    PayloadFunc ptr = execute_payload;
    printf("  [PATH D] Calling through pointer: 0x%p\n", (void*)ptr);
    ptr();
}

Stack Depth: 1 wrapper level (but with indirection) Characteristics:

• Stores payload address in function pointer before calling

• The call through pointer may appear differently in some analysis tools

• Demonstrates indirect execution patterns

---

6.5 Path E: Recursive (Variable 1-3 Levels)

void wrapper_path_E_recursive(int depth) {
    printf("  [PATH E] Recursion depth: %d\n", depth);

    if (depth <= 0) {
        aux_thread_info();
        execute_payload();
    } else {
        aux_small_delay();
        wrapper_path_E_recursive(depth - 1);
    }
}

void wrapper_path_E_entry() {
    printf("[PATH E] Recursive execution path\n");
    int recursion_depth = rand() % 3 + 1;  // 1-3 levels
    wrapper_path_E_recursive(recursion_depth);
}

Stack Depth: Variable (2-4 wrapper levels) Characteristics:

• First source of internal variability

• Same path selection can produce different depths

• Recursion creates multiple instances of the same function on the stack

• Stack shows repeated wrapper_path_E_recursive entries

---

6.6 Path H: Tower (5 Levels)

void wrapper_path_H_level5() {
    printf("          [PATH H] Level 5 - Final\n");
    aux_thread_info();
    execute_payload();
}

void wrapper_path_H_level4() {
    printf("        [PATH H] Level 4\n");
    aux_query_perf();
    wrapper_path_H_level5();
}

void wrapper_path_H_level3() {
    printf("      [PATH H] Level 3\n");
    aux_heap_operation();
    wrapper_path_H_level4();
}

void wrapper_path_H_level2() {
    printf("    [PATH H] Level 2\n");
    aux_get_time();
    wrapper_path_H_level3();
}

void wrapper_path_H_level1() {
    printf("  [PATH H] Level 1\n");
    aux_small_delay();
    wrapper_path_H_level2();
}

void wrapper_path_H_tower() {
    printf("[PATH H] 5-Level Tower path\n");
    wrapper_path_H_level1();
}

Stack Depth: 6 wrapper levels Call Stack:

execute_payload()
wrapper_path_H_level5()
wrapper_path_H_level4()
wrapper_path_H_level3()
wrapper_path_H_level2()
wrapper_path_H_level1()
wrapper_path_H_tower()
execute_random_path()
main()

Characteristics:

• Deep linear path with unique function at each level

• Different auxiliary operation per level creates rich behavioral pattern

• Significantly different stack signature from shallow paths

---

6.7 Path I: Deep (6 Levels)

void wrapper_path_I_level6() {
    printf("            [PATH I] Level 6 - Terminus\n");
    execute_payload();
}

void wrapper_path_I_level5() {
    printf("          [PATH I] Level 5\n");
    HANDLE heap = GetProcessHeap();
    void* m = HeapAlloc(heap, 0, 32);
    wrapper_path_I_level6();
    if (m) HeapFree(heap, 0, m);
}

// ... levels 4, 3, 2, 1 ...

void wrapper_path_I_deep6() {
    printf("[PATH I] 6-Level Deep path\n");
    wrapper_path_I_level1();
}

Stack Depth: 7 wrapper levels Characteristics:

• Deepest linear path (excluding variable paths)

• Inline API calls rather than auxiliary function calls

• Memory is allocated and held across the nested call

---

6.8 Path J: Function Pointer Chain (5 Links)

typedef void (*ChainFunc)();

void wrapper_path_J_final() {
    printf("          [PATH J] Chain end\n");
    aux_heap_operation();
    execute_payload();
}

void wrapper_path_J_link4() {
    printf("        [PATH J] Link 4\n");
    ChainFunc next = wrapper_path_J_final;
    aux_small_delay();
    next();
}

void wrapper_path_J_link3() {
    printf("      [PATH J] Link 3\n");
    ChainFunc next = wrapper_path_J_link4;
    aux_query_perf();
    next();
}

// ... links 2, 1 ...

void wrapper_path_J_chain() {
    printf("[PATH J] Function Pointer Chain (5 links)\n");
    ChainFunc start = wrapper_path_J_link1;
    start();
}

Stack Depth: 6 wrapper levels Characteristics:

• Each level calls the next through a function pointer

• Combines depth with indirection

• Local function pointer variables add to stack frame complexity

---

6.9 Path K: Mixed Recursion + Nesting (4-7 Levels)

void wrapper_path_K_nested_inner() {
    printf("        [PATH K] Nested inner\n");
    aux_get_time();
    execute_payload();
}

void wrapper_path_K_nested_outer() {
    printf("      [PATH K] Nested outer\n");
    aux_heap_operation();
    wrapper_path_K_nested_inner();
}

void wrapper_path_K_recursive(int depth) {
    printf("    [PATH K] Recursive level: %d\n", depth);
    aux_small_delay();

    if (depth <= 0) {
        wrapper_path_K_nested_outer();
    } else {
        wrapper_path_K_recursive(depth - 1);
    }
}

void wrapper_path_K_mixed() {
    printf("[PATH K] Mixed Recursion + Nesting path\n");
    int depth = rand() % 4 + 1;  // 1-4 recursion levels + 2 nested
    wrapper_path_K_recursive(depth);
}

Stack Depth: Variable (4-7 wrapper levels) Characteristics:

• Combines two patterns: recursion followed by fixed nesting

• Random recursion depth (1-4) plus 2 fixed nested levels

• Creates 4 distinct possible stack depths

• Most complex variable-depth path

---

6.10 Path L: Staircase (7 Levels) - System Info Path

void wrapper_path_L_level7() {
    printf("              [PATH L] Level 7 - Summit\n");
    execute_payload();
}

void wrapper_path_L_level6() {
    printf("            [PATH L] Level 6\n");
    char buf[64];
    GetEnvironmentVariableA("PATH", buf, 10);
    wrapper_path_L_level7();
}

void wrapper_path_L_level5() {
    printf("          [PATH L] Level 5\n");
    DWORD size = MAX_PATH;
    char compname[MAX_PATH];
    GetComputerNameA(compname, &size);
    wrapper_path_L_level6();
}

void wrapper_path_L_level4() {
    printf("        [PATH L] Level 4\n");
    MEMORYSTATUSEX memstat;
    memstat.dwLength = sizeof(memstat);
    GlobalMemoryStatusEx(&memstat);
    wrapper_path_L_level5();
}

void wrapper_path_L_level3() {
    printf("      [PATH L] Level 3\n");
    SYSTEM_INFO sysinfo;
    GetSystemInfo(&sysinfo);
    wrapper_path_L_level4();
}

void wrapper_path_L_level2() {
    printf("    [PATH L] Level 2\n");
    FILETIME ft;
    GetSystemTimeAsFileTime(&ft);
    wrapper_path_L_level3();
}

void wrapper_path_L_level1() {
    printf("  [PATH L] Level 1\n");
    aux_thread_info();
    wrapper_path_L_level2();
}

void wrapper_path_L_staircase() {
    printf("[PATH L] 7-Level Staircase path\n");
    wrapper_path_L_level1();
}

Stack Depth: 8 wrapper levels (deepest fixed path) API Calls:

Level	API
1	GetCurrentThreadId, GetCurrentProcessId
2	GetSystemTimeAsFileTime
3	GetSystemInfo
4	GlobalMemoryStatusEx
5	GetComputerNameA
6	GetEnvironmentVariableA
7	(none - final)

Characteristics:

• Deepest fixed-depth path

• Uses diverse system information APIs

• Creates a rich behavioral profile alongside deep stack

---

6.11 Path M: Branching (5 Levels, 4 Routes)

void wrapper_path_M_terminus_alpha() {
    printf("          [PATH M] Terminus Alpha\n");
    aux_get_time();
    execute_payload();
}

void wrapper_path_M_terminus_beta() {
    printf("          [PATH M] Terminus Beta\n");
    aux_query_perf();
    execute_payload();
}

void wrapper_path_M_branch_level4() {
    printf("        [PATH M] Level 4 - Branch point\n");
    if (rand() % 2) {
        wrapper_path_M_terminus_alpha();
    } else {
        wrapper_path_M_terminus_beta();
    }
}

void wrapper_path_M_level3() {
    printf("      [PATH M] Level 3\n");
    aux_heap_operation();
    wrapper_path_M_branch_level4();
}

void wrapper_path_M_level2_left() {
    printf("    [PATH M] Level 2 - Left\n");
    aux_small_delay();
    wrapper_path_M_level3();
}

void wrapper_path_M_level2_right() {
    printf("    [PATH M] Level 2 - Right\n");
    aux_thread_info();
    wrapper_path_M_level3();
}

void wrapper_path_M_level1() {
    printf("  [PATH M] Level 1 - Initial branch\n");
    if (rand() % 2) {
        wrapper_path_M_level2_left();
    } else {
        wrapper_path_M_level2_right();
    }
}

void wrapper_path_M_branching() {
    printf("[PATH M] Branching Deep path (5 levels, 4 possible routes)\n");
    wrapper_path_M_level1();
}

Stack Depth: 6 wrapper levels (fixed) Possible Routes:

Route 1: level1 -> level2_left  -> level3 -> branch_level4 -> terminus_alpha
Route 2: level1 -> level2_left  -> level3 -> branch_level4 -> terminus_beta
Route 3: level1 -> level2_right -> level3 -> branch_level4 -> terminus_alpha
Route 4: level1 -> level2_right -> level3 -> branch_level4 -> terminus_beta

Characteristics:

• Most complex path structure

• Two independent branch points create 4 possible stack signatures

• Same depth, different function sequences

• Particularly effective against signature-based detection

---

7. Stack Trace Capture Mechanism

Implementation

void capture_and_print_stack(const char* context) {
#if ENABLE_STACK_TRACE
    void* stack[MAX_STACK_DEPTH];
    USHORT frames;

    frames = RtlCaptureStackBackTrace(0, MAX_STACK_DEPTH, stack, NULL);

    printf("\n[STACK TRACE] %s (depth: %d frames)\n", context, frames);

    HANDLE process = GetCurrentProcess();
    SymInitialize(process, NULL, TRUE);

    char symbol_buffer[sizeof(SYMBOL_INFO) + 256];
    SYMBOL_INFO* symbol = (SYMBOL_INFO*)symbol_buffer;
    symbol->MaxNameLen = 255;
    symbol->SizeOfStruct = sizeof(SYMBOL_INFO);

    for (USHORT i = 0; i < frames && i < 15; i++) {
        DWORD64 address = (DWORD64)stack[i];

        if (SymFromAddr(process, address, NULL, symbol)) {
            printf("  [%2d] 0x%016llX %s\n", i, address, symbol->Name);
        } else {
            printf("  [%2d] 0x%016llX <unknown>\n", i, address);
        }
    }

    SymCleanup(process);
#endif
}

Key Components

1. RtlCaptureStackBackTrace: Windows API that captures return addresses from the stack

   • Parameter 1: Frames to skip (0 = start from current)

   • Parameter 2: Maximum frames to capture

   • Parameter 3: Output array for addresses

   • Returns: Number of frames captured

2. SymInitialize / SymFromAddr / SymCleanup: Debug Help Library functions

   • Initialize symbol handler for the process

   • Resolve addresses to function names

   • Requires dbghelp.lib and debug symbols for full resolution

Example Output

[STACK TRACE] Before wrapper execution (depth: 12 frames)
----------------------------------------
  [ 0] 0x00007FF6A1B01234 capture_and_print_stack
  [ 1] 0x00007FF6A1B01567 execute_random_path
  [ 2] 0x00007FF6A1B01890 run_demonstration
  [ 3] 0x00007FF6A1B01ABC main
  [ 4] 0x00007FF6A1B01DEF __scrt_common_main_seh
  ...
----------------------------------------

---

8. Potential Improvements

8.1 Cryptographically Secure Randomness

Current:

srand((unsigned int)time(NULL));
int selected = rand() % NUM_WRAPPERS;

Improved:

#include <bcrypt.h>
#pragma comment(lib, "bcrypt.lib")

int secure_random(int max) {
    ULONG random_value;
    BCryptGenRandom(NULL, (PUCHAR)&random_value, sizeof(random_value),
                    BCRYPT_USE_SYSTEM_PREFERRED_RNG);
    return random_value % max;
}

Benefit: Unpredictable selection even if execution time is known.

---

8.2 Dynamic Wrapper Generation

Instead of static wrapper functions, generate them at runtime:

// Conceptual approach
typedef struct {
    int depth;
    void (*aux_funcs[8])();
    int aux_count;
} DynamicPath;

void execute_dynamic_path(DynamicPath* path) {
    for (int i = 0; i < path->aux_count; i++) {
        path->aux_funcs[i]();
    }
    execute_payload();
}

// Generate random path configuration at runtime
DynamicPath* generate_random_path() {
    DynamicPath* p = malloc(sizeof(DynamicPath));
    p->depth = rand() % 5 + 1;
    p->aux_count = rand() % 4;
    // ... populate aux_funcs randomly
    return p;
}

Benefit: Infinite path variations without code bloat.

---

8.3 JIT Wrapper Compilation

For advanced scenarios, generate wrapper code in executable memory:

void create_jit_wrapper() {
    void* mem = VirtualAlloc(NULL, 4096, MEM_COMMIT | MEM_RESERVE,
                             PAGE_EXECUTE_READWRITE);

    // Write machine code for a simple wrapper
    unsigned char code[] = {
        0x48, 0x83, 0xEC, 0x28,  // sub rsp, 28h
        0x48, 0xB8,              // mov rax, <address>
        // ... payload address bytes ...
        0xFF, 0xD0,              // call rax
        0x48, 0x83, 0xC4, 0x28,  // add rsp, 28h
        0xC3                     // ret
    };

    memcpy(mem, code, sizeof(code));
    // Patch in payload address
    // Execute
}

Benefit: Stack shows addresses in non-module memory, harder to attribute.

---

8.4 Timing-Based Path Selection

Add temporal variation:

void execute_time_based_path() {
    SYSTEMTIME st;
    GetSystemTime(&st);

    // Different path based on milliseconds
    int time_seed = st.wMilliseconds;
    int path_index = (time_seed * 7 + st.wSecond * 13) % NUM_WRAPPERS;

    g_wrappers[path_index]();
}

Benefit: Even with fixed RNG seed, paths vary based on execution timing.

---

8.5 More Auxiliary Diversity

Add more system API calls to auxiliary functions:

void aux_registry_query() {
    HKEY key;
    if (RegOpenKeyExA(HKEY_LOCAL_MACHINE, "SOFTWARE\\Microsoft",
                      0, KEY_READ, &key) == ERROR_SUCCESS) {
        RegCloseKey(key);
    }
}

void aux_file_query() {
    WIN32_FIND_DATAA findData;
    HANDLE h = FindFirstFileA("C:\\Windows\\*.dll", &findData);
    if (h != INVALID_HANDLE_VALUE) {
        FindClose(h);
    }
}

void aux_network_check() {
    WSADATA wsaData;
    if (WSAStartup(MAKEWORD(2, 2), &wsaData) == 0) {
        WSACleanup();
    }
}

Benefit: More diverse behavioral patterns, harder to correlate.

---

8.6 Anti-Analysis Timing

Add variable delays to frustrate dynamic analysis:

void wrapper_with_anti_analysis() {
    // Random delay 0-100ms
    Sleep(rand() % 100);

    // Check if debugger attached
    if (IsDebuggerPresent()) {
        // Take different (decoy) path
        decoy_path();
    } else {
        real_path();
    }
}

---

8.7 Path Weight Configuration

Allow unequal probability distribution:

typedef struct {
    WrapperFunc func;
    int weight;  // Higher = more likely
} WeightedPath;

WeightedPath g_weighted_paths[] = {
    { wrapper_path_A_direct, 1 },     // Rare
    { wrapper_path_H_tower, 5 },      // Common
    { wrapper_path_L_staircase, 10 }, // Very common
    // ...
};

void execute_weighted_random_path() {
    int total_weight = 0;
    for (int i = 0; i < NUM_WRAPPERS; i++) {
        total_weight += g_weighted_paths[i].weight;
    }

    int r = rand() % total_weight;
    int cumulative = 0;

    for (int i = 0; i < NUM_WRAPPERS; i++) {
        cumulative += g_weighted_paths[i].weight;
        if (r < cumulative) {
            g_weighted_paths[i].func();
            return;
        }
    }
}

Benefit: Favor deeper paths to maximize stack signature variation.

---

9. Detection Strategies

Understanding how to detect this technique is crucial for EDR developers.

9.1 Behavioral Clustering

Instead of exact stack matching, cluster similar behaviors:

Observation: All paths eventually call MessageBoxA
Detection: Track API call sequences regardless of intermediate stack

9.2 Statistical Analysis

Monitor for unusual stack depth variance:

Normal application: Consistent stack depth for same operation
This technique: High variance in stack depth for identical outcomes

Detection heuristic: Flag processes where the same final API is reached via significantly different stack depths across multiple observations.

9.3 Return Address Module Analysis

// EDR-side detection concept
bool analyze_return_addresses(void** stack, int depth) {
    for (int i = 0; i < depth; i++) {
        HMODULE module;
        GetModuleHandleExA(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS,
                          (LPCSTR)stack[i], &module);

        // Flag if return address is in:
        // - Non-backed memory (JIT)
        // - Unknown/suspicious module
        // - Unusually deep user-mode stack
    }
}

9.4 Function Name Pattern Detection

The wrapper function names follow patterns:

wrapper_path_*
wrapper_path_*_level*
wrapper_path_*_link*

Detection: Flag modules containing multiple functions matching wrapper-like naming patterns.

9.5 Call Graph Analysis

Build a call graph and identify:

• Multiple entry points to same sensitive function

• Unusual fan-in patterns

• Functions that only exist to call other functions

9.6 Entropy-Based Detection

Measure randomness in execution patterns:

High entropy in call path selection +
Consistent final behavior =
Potential probabilistic evasion

---

10. Conclusion

Probabilistic Call Stacks represent an interesting approach to evading stack-based detection mechanisms. By introducing randomness at the execution flow level, attackers can generate unique signatures for each run while maintaining identical malicious behavior.

Key Takeaways

1. Effectiveness varies: This technique is most effective against signature-based detection that relies on exact stack matches. Behavioral analysis remains effective.

2. Complexity trade-off: More paths and deeper nesting increase evasion potential but also increase code size and complexity.

3. Not a silver bullet: This is one layer in a defense-evasion strategy, not a complete solution.

4. Detection is possible: Statistical analysis, behavioral clustering, and call graph analysis can identify this pattern.

For EDR Developers

• Move beyond exact stack matching

• Implement statistical analysis of stack variance

• Focus on behavioral endpoints, not intermediate paths

• Consider entropy-based anomaly detection

For Security Researchers

• This technique demonstrates the cat-and-mouse nature of security

• Understanding evasion techniques improves detection capabilities

• The code provides a foundation for further research and testing

---

References

• Windows API Documentation: RtlCaptureStackBackTrace

• DbgHelp Library: Symbol Handler Functions

• MSDN: Structured Exception Handling and Stack Unwinding

• Academic Papers on Control Flow Integrity