Grep Is Bad — Use AST Search: Harmful Advice

# "Grep Is Bad, Use AST Search": Bad Advice for Large Codebases

## What They Say at Meetups

https://youtu.be/Nm3MsnngCJg?si=4QWKn2MKNIyEsEGg

Sheiko says in his lecture:

> "Even though agents are very good at using grep, it doesn't work on colossal projects because every grep dump produces a wall of matches. Instead of using grep, which throws out this garbage, you can use AST Search."

Sounds logical. Sounds like an expert opinion. The audience nods. Everyone thinks: "Yes, grep is primitive, AST Search is the right way."

This is **bad advice**. Because it confuses cause and effect. The problem isn't with grep. The problem is **the lack of indexing and filtering**. And AST Search on large codebases is **also a non-trivial problem** that Sheiko doesn't mention a word about.

---

## The Problem Isn't Grep. The Problem Is How People Use It

When they say "grep throws out a wall of matches" — it means that a person (or agent) ran `grep -r "someFunction" .` across the entire project without any filtering. And got 5000 matches in node_modules, .git, vendor, build artifacts, generated files.

This is **not a grep problem**. This is a problem of **missing**:

1. `.gitignore`-like filtering (excluding node_modules, vendor, build)

2. Indexing (searching only relevant files)

3. Contextual output (not just a match line, but the code around it)

grep itself is a **phenomenally fast tool**. It's written in C, optimized over decades, works at the byte level. Combined with shell scripts, it becomes a powerful search pipeline that processes gigabytes of text in milliseconds.

Killing grep because someone doesn't know how to use it is like killing a hammer because someone hits their fingers with it.

---

## grep + Indexing + Shell = Phenomenal Speed

Here's what **proper** search in a large codebase looks like:

**Step 1: Indexing.**

A project file index is created: all source files, headers, configs. Excluded: node_modules, vendor, build, .git, generated files. The index stores a list of files with their paths and metadata (language, size, modification date).

**Step 2: Static library indexing.**

Connected libraries (especially large ones like UE5, Boost, Qt) are indexed **once** upon inclusion. The index contains: headers, function signatures, key classes. This allows searching the library's API without scanning all source files.

**Step 3: Dynamic incremental indexing on update.**

When a library is updated, the index is rebuilt **incrementally**: only changed files. Not a full rebuild. This takes seconds, not minutes.

**Step 4: Search via shell script.**

```bash

#!/bin/bash

# search.sh - search through project index

QUERY="$1"

INDEX_FILE=".search_index"

# Search through index (only relevant files)

grep -n "$QUERY" $(cat "$INDEX_FILE") \

  --include="*.cpp" --include="*.h" --include="*.py" \

  --exclude-dir={node_modules,vendor,build,.git} \

  -A 5 -B 2  # Context: 5 lines after, 2 before

```

Result: not a "wall of matches", but **contextual output** with filtering. And the speed — milliseconds.

**Step 5: Search via skill or MCP tool.**

The agent doesn't run bare `grep`. It calls a **configured search tool** (skill or MCP tool) that internally does:

1. Read the index

2. Filter by file type and directory

3. grep the filtered list

4. Format the result with context

5. Return a compact answer to the agent

The agent gets not a "5000-line wall", but **5-10 relevant matches with context**. That's what's needed for work.

---

## AST Search on Large Codebases Is Also Non-Trivial

Sheiko says: "Use AST Search, it works on large projects." But doesn't say **what's behind AST Search on large codebases**.

**1. Building an AST is expensive.**

For each file, you need to parse the syntax tree. For a project with 10k files, that's minutes for initial build. For UE5 with millions of lines of code — it's **hours**. And it's not a one-time operation: every file change requires AST rebuild.

**2. Multi-language support.**

AST Search requires a parser for each language. C++, Python, JavaScript, TypeScript, Rust — each has its own parser, its own rules, its own edge cases. grep works with any text. AST Search only works with what has a parser.

**3. AST storage and maintenance.**

AST takes memory. For a large project — gigabytes. You need to store it, update it, invalidate on changes. That's infrastructure you need to maintain.

**4. AST Search doesn't solve the filtering problem.**

Even with AST, if you search for `someFunction` across the whole project, you'll get hundreds of matches. AST helps understand the **context** of a match (is it a call, a definition, a reference), but doesn't help **filter out** irrelevant files. For that, you still need indexing.

AST Search is a **complement** to grep, not a replacement. It's useful for specific tasks: "find all calls to function X that aren't wrapped in try-catch." But for basic codebase search, grep with indexing is faster and simpler.

---

## Personal Experience: UE5, Search via Index < 50ms

Here's a concrete example. The UE5 engine codebase — millions of lines of code, thousands of files, dozens of modules. Searching such a base with bare grep without indexing takes seconds and produces a wall of matches.

But with proper indexing:

**What was done:**

- Index of UE5 headers and source files (static indexing upon engine inclusion)

- Index of project modules (dynamic indexing on changes)

- Shell script for indexed search with contextual output

- MCP tool for the agent that calls this script

**Result:**

- Index query: **< 50ms**

- Output: a code chunk with context immediately (function, class, signature)

- No "wall of matches." Only relevant results.

This isn't AST Search. This is **grep + indexing + shell**. And it works faster and simpler than AST Search on such a codebase.

---

## Don't Believe Them When They Say "Grep Is Bad"

When an "expert" says "grep is bad, use AST Search" — it means that they:

1. **Don't know how to set up indexing.** And instead of setting it up, say "the tool is bad."

2. **Don't know how to write shell scripts.** And instead of building a pipeline, look for "the right tool."

3. **Don't understand that AST Search also requires infrastructure.** And think it's a "plug and play" solution.

grep is a **phenomenal tool**. It's existed for decades, optimized to the limit, works on any platform, requires no dependencies. Combined with indexing and shell scripts, it solves 90% of codebase search tasks.

AST Search is a **specialized tool** for tasks that require syntactic analysis. It doesn't replace grep. It complements it.

---

## What to Do for a Large Codebase

**Step 1: Create a file index.**

A list of all relevant project files. Exclude garbage (node_modules, vendor, build, .git). Store in a `.search_index` file.

**Step 2: Index connected libraries.**

For large libraries (UE5, Boost, Qt), create a separate index of headers and signatures. Index once upon inclusion.

**Step 3: Set up dynamic indexing.**

When a library is updated or project files change, rebuild the index incrementally. Only changed files.

**Step 4: Write a shell script for search.**

grep over index + filtering + contextual output. Not bare `grep -r`, but a configured pipeline.

**Step 5: Wrap in a skill or MCP tool.**

The agent calls a search tool, not bare grep. The tool handles indexing, filtering, formatting internally. The agent gets a compact, relevant answer.

**Step 6: AST Search only for specific tasks.**

If you need syntactic analysis ("find all calls without try-catch", "find all subclasses of class X") — then AST Search. For basic search — grep with indexing.

---

## Conclusion

"Grep is bad, use AST Search" is bad advice. It confuses cause and effect. The problem isn't with grep. The problem is the lack of indexing and filtering.

grep is a phenomenally fast tool. Combined with indexing, shell scripts, and MCP tools, it solves 90% of codebase search tasks. Faster and simpler than AST Search.

AST Search on large codebases is also a non-trivial problem. Building ASTs, supporting parsers, storage, maintenance. That's infrastructure you need to maintain. And it doesn't replace indexing.

Those who say "grep is bad" simply don't know how to set up indexing. And that's their problem. But they shouldn't blame the tool that's worked for decades.