PicoCTF 2025 Reverse Engineering—Perplexed Writeup

Back again with another writeup. This time, I’m breaking down Perplexed, a Reverse Engineering challenge from picoCTF. It’s one of those puzzles that look simple on the surface but demand a sharp eye and solid fundamentals in binary analysis. I’ll walk through how I approached the challenge, what stood out, and the techniques I used to get to the flag. If you’re into reverse engineering or just looking to sharpen your skills, this one’s worth dissecting.

At first, the lack of a description or hint made me think this would be a tough one—something along the lines of Homework. But surprisingly, it turned out to be much more straightforward than expected. Let's get started!

Let's execute the code first to see its behavior

It's simply asking for the password, nothing more. We didn't gather enough information by running the binary, so let's open up Ghidra to dive deeper and analyze what's going on underneath.

After decompiling, this is the main function

bool main(void)

{
  bool bVar1;
  char local_118 [268];
  int local_c;
  
  local_118[0] = '\0';
  local_118[1] = '\0';
  local_118[2] = '\0';
  ....
  local_118[0xfe] = '\0';
  local_118[0xff] = '\0';
  printf("Enter the password: ");
  fgets(local_118,0x100,stdin);
  local_c = check(local_118);
  bVar1 = local_c != 1;
  if (bVar1) {
    puts("Correct!! :D");
  }
  else {
    puts("Wrong :(");
  }
  return !bVar1;
}

This main function initializes a 256-byte buffer, prompts the user to enter a password, and reads the input using fgets. It then passes the input to a function called check, and based on the result, prints either "Correct!! :D" if the password is valid or "Wrong :(" if it's not. The return value reflects whether the correct password was entered.

Let's visit the check function

undefined8 check(char *param_1)

{
  size_t sVar1;
  undefined8 uVar2;
  size_t sVar3;
  char local_58 [36];
  uint local_34;
  uint local_30;
  undefined4 local_2c;
  int local_28;
  uint local_24;
  int local_20;
  int local_1c;
  
  sVar1 = strlen(param_1);
  if (sVar1 == 0x1b) {
    local_58[0] = -0x1f;
    local_58[1] = -0x59;
    local_58[2] = '\x1e';
    local_58[3] = -8;
    local_58[4] = 'u';
    local_58[5] = '#';
    local_58[6] = '{';
    local_58[7] = 'a';
    local_58[8] = -0x47;
    local_58[9] = -99;
    local_58[10] = -4;
    local_58[0xb] = 'Z';
    local_58[0xc] = '[';
    local_58[0xd] = -0x21;
    local_58[0xe] = 'i';
    local_58[0xf] = 0xd2;
    local_58[0x10] = -2;
    local_58[0x11] = '\x1b';
    local_58[0x12] = -0x13;
    local_58[0x13] = -0xc;
    local_58[0x14] = -0x13;
    local_58[0x15] = 'g';
    local_58[0x16] = -0xc;
    local_1c = 0;
    local_20 = 0;
    local_2c = 0;
    for (local_24 = 0; local_24 < 0x17; local_24 = local_24 + 1) {
      for (local_28 = 0; local_28 < 8; local_28 = local_28 + 1) {
        if (local_20 == 0) {
          local_20 = 1;
        }
        local_30 = 1 << (7U - (char)local_28 & 0x1f);
        local_34 = 1 << (7U - (char)local_20 & 0x1f);
        if (0 < (int)((int)param_1[local_1c] & local_34) !=
            0 < (int)((int)local_58[(int)local_24] & local_30)) {
          return 1;
        }
        local_20 = local_20 + 1;
        if (local_20 == 8) {
          local_20 = 0;
          local_1c = local_1c + 1;
        }
        sVar3 = (size_t)local_1c;
        sVar1 = strlen(param_1);
        if (sVar3 == sVar1) {
          return 0;
        }
      }
    }
    uVar2 = 0;
  }
  else {
    uVar2 = 1;
  }
  return uVar2;
}

The function clearly enforces that the password must be exactly 27 characters long—if not, it immediately exits without doing any further validation. The local_58 array acts like a reference pattern, and each bit of the input is compared against it through a series of bit-level operations. This is essentially a custom verification mechanism that checks the input against a hidden "bitmask hash." If even a single bit doesn't align during the comparison, the function returns early, marking the input as incorrect.

Rather than analyzing the exact bit logic in depth, a more practical route was attempted: starting with a test string that followed the picoCTF{...} format and padding the rest with filler characters like 'A'. Observing how far the check progressed gave hints that some parts were correct. From there, a brute-force approach was taken—testing characters one by one and monitoring if the function moved on to the next comparison byte—eventually reconstructing the full flag by confirming each correct character incrementally.

To solve the challenge, we can brute-force the correct password one character at a time by exploiting how the check() function compares each bit of the input against a hardcoded reference array. Since the function checks bit-by-bit and stops at the first mismatch, we can modify or replicate the function to track how many characters have been successfully matched before a failure occurs. By testing each possible printable character at each position and observing which one allows the function to progress further, we can gradually reconstruct the entire 27-character password.

I implemented the solution using the C.

#include <stdio.h>
#include <string.h>

#define MAX_PASS_LEN 28

typedef unsigned char u8;
typedef unsigned int u32;

char attempt_password[MAX_PASS_LEN];
char solved_password[MAX_PASS_LEN];

int solved_length = 0;

typedef struct {
    char reference_key[36];
    int char_index;
    u8 bit_progress;
} BreakerContext;

// Initializes the reference key and context for password checking
void configure_reference_key(BreakerContext *ctx) {
    const char key[] = {
        -31, -89, 30, -8, 'u', '#', '{', 'a',
        -71, -99, -4, 'Z', '[', -33, 'i', 210,
        -2, 27, -19, -12, -19, 'g', -12
    };
    memcpy(ctx->reference_key, key, sizeof(key));
    ctx->char_index = 0;
    ctx->bit_progress = 0;
}

// Validates a password attempt bit-by-bit against the reference key
u8 check_password_attempt(const char *attempt, BreakerContext *ctx) {
    // Ensure password length is exactly 27 characters
    if (strlen(attempt) != 27) {
        printf("ERROR: Password length must be 27 characters, got %zu\n", 
               strlen(attempt));
        return 1;  // Error
    }

    // Loop through the first 23 bytes of the reference key
    for (u32 key_idx = 0; key_idx < 23; key_idx++) {
        // Compare each bit (8 bits per byte)
        for (u32 bit_idx = 0; bit_idx < 8; bit_idx++) {
            if (!ctx->bit_progress) {
                ctx->bit_progress = 1;  // Start bit progress from 1
            }

            // Create bit masks to isolate the bit being checked
            u32 attempt_mask = 1 << (7 - ctx->bit_progress);
            u32 key_mask = 1 << (7 - bit_idx);

            // Compare corresponding bits between attempt and reference key
            if (((attempt[ctx->char_index] & attempt_mask) != 0) !=
                ((ctx->reference_key[key_idx] & key_mask) != 0)) {
                // If mismatch, store how many characters were correct
                solved_length = ctx->char_index;
                return 1;  // Password attempt failed
            }

            // Move to next bit
            ctx->bit_progress++;
            if (ctx->bit_progress == 8) {
                ctx->bit_progress = 0;
                ctx->char_index++;  // Move to next character
            }

            // Stop if we've checked the whole password
            if ((size_t)ctx->char_index == strlen(attempt)) {
                return 0;  // Success
            }
        }
    }
    return 0;  // All checks passed
}

// Attempts to brute-force the password character by character
int break_password(void) {
    BreakerContext breaker;
    int prev_solved_count = 0;
    u8 breaker_status = 1;
    configure_reference_key(&breaker);

    // Continue brute-forcing until the password is solved
    while (breaker_status) {
        // Try all printable ASCII characters (space to ~)
        for (char c = 0x20; c < 0x7f; c++) {
            // If some characters are already confirmed, use them as prefix
            if (solved_length > 0) {
                memcpy(attempt_password, solved_password, solved_length);
                printf("Continuing with prefix: %.*s\n", 
                       solved_length, solved_password);
            }

            // Fill the remaining with 'A' except the character being tested
            memset(attempt_password + solved_length, 'A', 
                   MAX_PASS_LEN - solved_length - 1);
            attempt_password[solved_length] = c;
            attempt_password[MAX_PASS_LEN - 1] = '\0';
            printf("Testing: %s\n", attempt_password);

            // Reset context for each attempt
            breaker.char_index = 0;
            breaker.bit_progress = 0;

            // Check if current attempt matches the key pattern
            breaker_status = check_password_attempt(attempt_password, &breaker);

            // If more characters have been confirmed, save them
            if (solved_length > prev_solved_count) {
                printf("ADVANCE: Confirmed %d characters\n", solved_length);
                memcpy(solved_password, attempt_password, solved_length);
                prev_solved_count = solved_length;
            }

            // If password ends with '}', it’s likely fully recovered
            if (solved_length > 0 && solved_password[solved_length - 1] == '}') {
                solved_password[solved_length] = '\0';
                printf("\nPassword Retrieved: %s\n", solved_password);
                return 0;
            }
        }
    }
    return 0;
}

int main(void) {
    memset(attempt_password, 0, MAX_PASS_LEN);
    memset(solved_password, 0, MAX_PASS_LEN);
    return break_password();
}

Compile then execute and it will give us the flag!

We've successfully solved Perplexed!!