AI doesn't remember your project, Markdown does

In the previous post, Have You Built an Agent Harness Yet?, we built a tiny harness in Swift. We gave it a loop, a prompt, and some tools, and with that it already had most of the features we are used to with modern AI. No magic. Just autocomplete, and a harness deciding what the model gets to see and do.

But there is another bit of magic we didn’t touch on and we should demystify. The appearance of memory. You know that moment when you open a brand new conversation and the agent already seems to know your project.

It doesn’t.

LLMs have no memory between fresh conversations. They just autocomplete from what they have in the current context. So if your AI tool of the week seems to remember your repo conventions, folder structure, or coding rules, what is happening is much simpler.

The industry has come to agree on the AGENTS.md standard, although some companies think they are special and use other file names. This is where you can put information about the project for the AI to learn.

But remember, there is no magic in AI. No memory. Just autocomplete. So how does this even work? If you’ve read the first post in this series, you already know the answer. The harness is doing the work.

The trick is that the harness automatically loads that file when the conversation starts and injects it into the context.

That’s it. That is the whole trick.

It is not a secret memory layer. It is not special agent magic. It is just more text in the context.

Let’s add it to the little harness from last post so you can see how simple the trick really is.

The Agent Only Knows What We Send

At the end of the previous article, our system prompt was still very simple:

self.systemPrompt = """
    You are a helpful assistant with access to tools.

    The workspace root is:
    \(workspaceRoot.path)

    Tools:
    \(toolsPrompt)

    When you want to use the tool, reply with exactly one line in this format and nothing else:
    TOOL_CALL {"name":"tool_name","arguments":{"path":"some/path"}}

    After receiving a TOOL_RESULT message, continue the task.
    If no tool is needed, answer normally.
    """

That is enough for the agent to use tools and know where the workspace root is. But it still doesn’t know anything specific about the project itself.

So if we start a fresh conversation, the agent won’t know anything about our project and will have to go discover it.

<Turn 1>
You: What do you know about this project?
<Step 1>
Assistant: I don’t know much yet beyond the project name and whatever is in the package manifest.
From the workspace path, it appears to be a Swift project named **swiftagentharness**.
I’ve requested the file listing and the `Package.swift` manifest so I can summarize:
- what kind of Swift package it is
- its targets/products
- dependencies
- likely purpose
If you want, I can continue by inspecting the repository structure and give you a concise overview of the project.
tool-call: TOOL_CALL {"name":"read_file","arguments":{"path":"\/Users\/me\/code\/swiftagentharness\/Package.swift"}}
tool-success: read_file -> 22 lines
<Step 2>
Assistant: Based on `Package.swift`, here’s what I know:
...

That is exactly what AGENTS.md is for.

Let’s Load the File

The implementation is delightfully boring.

Before building the system prompt, we try to read an AGENTS.md file from the workspace root:

let agentsMD = try? String(
    contentsOf: workspaceRoot.appendingPathComponent("AGENTS.md"),
    encoding: .utf8
)

let projectInstructions = if let agentsMD {
    """
    Project instructions in AGENTS.md:
    \(agentsMD)
    """
} else {
    ""
}

And once we have that text, we add it to the system prompt:

self.systemPrompt = """
    You are a helpful assistant with access to tools.

    The workspace root is:
    \(workspaceRoot.path)

    \(projectInstructions)

    Tools:
    \(toolsPrompt)

    When you want to use the tool, reply with exactly one line in this format and nothing else:
    TOOL_CALL {"name":"tool_name","arguments":{"path":"some/path"}}

    After receiving a TOOL_RESULT message, continue the task.
    If no tool is needed, answer normally.
    """

That’s it.

Now the “project knowledge” is available to the LLM. Not in some mystical way, but just by being in its context. The harness literally copies the Markdown into the context it sends to the model.

That’s all this “project conventions” memory is.

Making the Trick Visible

Because one of the goals of this whole exercise is to break the illusion, I also made the harness print whether it loaded the file:

let agentsMD = try? String(contentsOf: workspaceRoot.appendingPathComponent("AGENTS.md"), encoding: .utf8)
self.agentsMDLineCount = agentsMD.map { $0.components(separatedBy: .newlines).count }

...

print("Swift Agent Harness")
print("Model: \(client.model)")
if let lineCount = agentsMDLineCount {
    print("Loaded AGENTS.md (\(lineCount) lines)")
}
print("Ctrl+C to quit")

If the harness says it loaded 2 lines, that means 2 lines of Markdown are about to shape the model’s behavior. Nothing more. Nothing less.

For my demo project, the file is tiny:

This is a test AGENTS.md.
If user says "Hello Swift Agent Harness", reply acknowledging you are that Swift agent harness and that you have read this AGENTS.md.

And once the harness injects that into the system prompt, the effect is immediate:

Swift Agent Harness
Model: gpt-5.4
Loaded AGENTS.md (2 lines)
Ctrl+C to quit
<Turn 1>
You: Hello Swift Agent Harness
<Step 1>
Assistant: Hello! I’m the Swift Agent Harness, and I’ve read the AGENTS.md.

Of course it can answer that way. We literally gave it those instructions in the context.

It Is Just Markdown

This is the part I really want people to internalize.

AGENTS.md is not a magical new memory subsystem for AI. It is not some privileged channel into the model’s brain. It is not a special capability only the big fancy tools have.

It is just Markdown added into the context.

Useful Markdown, yes. Important Markdown, maybe. But still just text that your harness decided to prepend before the conversation starts. You could literally start each conversation by pasting that instruction yourself, and it wouldn’t be much different.

Look what happens if we do that before any of the changes in this post.

<Turn 1>
You: This is a test AGENTS.md. 
If user says "Hello Swift Agent Harness", reply acknowledging you are that Swift agent harness and that you have read this AGENTS.md.
<Step 1>
Assistant: Understood. How can I help with the AGENTS.md test?
<Turn 2>
You: Hello Swift Agent Harness
<Step 1>
Assistant: Hello! I’m the Swift Agent Harness, and I’ve read this AGENTS.md.
<Turn 3>

Of course, here we’re just trying to learn how things work and demystify them. Production-grade tools can go further. They may walk parent directories, support nested instruction files, merge global and local rules, decide when to refresh or skip them, even summarize huge files to fit the token budget.

All of that is just more real engineering work (and learning the particularities of the autocomplete machines).

But the core idea is still exactly this one. Somewhere in the code there is a boring piece of logic reading a file and sticking its contents into the prompt.

That is the “memory”.

That is the “project awareness”.

That is the “magic”.

One More Layer Off the Illusion

I like this example because it teaches a few things.

The model never knows your project by default. If it knows something, your harness probably put it there.

The conventions around agent tooling are much more ordinary than they sound. AGENTS.md is not a sacred artifact. It is just a contract between your harness and your repo. If this file exists, load it and send it too.

And honestly, that is reassuring. Once you see it clearly, the whole thing becomes easier to reason about. Easier to debug. Easier to improve. Easier to distrust when the marketing gets too theatrical.

It’s just models and context, with a bit of Markdown sprinkled on top.