Recaps, Another Hidden AI Furnace

In the previous post we looked at hidden furnaces in agentic products. The example was the most common one, title generation. You type the first message, and the product quietly asks another model to name the conversation. Small and useful.

This time I want to add recaps to our tiny harness. Not because they are flashy, but because they are a genuinely useful hidden furnace.

The human memory problem

The interface for most agentic tools is still a long-ass chat history. That is fine while you are inside the flow. You ask something, the agent replies, it calls tools, you steer it, the work keeps moving.

But the better you get at agentic engineering, the longer the turns become, which means you are free to jump between threads, or even jump between projects. Sometimes I leave a coding agent halfway through a task, come back later, and the only thing waiting for me is a giant wall of text.

Of course the history is there. I can scroll. I can reconstruct in my mind what happened. But that is exactly the problem. I do not want to recreate my own memory every time I come back to a conversation.

So a recap feature is a very natural hidden furnace. After the conversation has been idle for a bit, the harness can ask a smaller model to produce a short reminder of where we are. Not a full summary. Just enough to refresh my memory.

Something like this:

> Recaps are useful in agent chats because they preserve shared context, reduce
  drift, and make it easier to continue complex work across long or interrupted
  conversations.

gpt-5.4 | #2 | Recaps help… | read_file,+2 | done | idle
You:

That came from a real run of the harness. The important part is not the terminal UI. The important part is that the recap did not come from the assistant turn itself. It came from another model call the harness decided to run.

So yes. Another furnace.

Use the same secondary model

In the title post, we separated the main model from the model used for small background jobs. That paid off immediately. A title does not need the same model that is doing the actual work.

Recaps fit that same bucket.

The harness now has a primary client and a secondary client:

struct Agent {
    let client: OpenAICompatibleClient
    let secondaryClient: any LLMClient
    let secondaryClientDescription: String
    let recapConfiguration: RecapConfiguration?
}

The main client handles the real conversation. The secondaryClient handles background work like titles and recaps.

This is a small naming change, but it is the right mental model. The title model was not really a title model anymore. It was the model for hidden furnaces.

Wait until things get quiet

The next question is when to generate the recap. For the title we had a clear place, but this is a bit more involved since we want to keep the recap fresh, but only do it when it is useful.

Doing it after every keystroke would be useless. Doing it while the assistant is still calling tools would also be wasteful. As long as the user or the agent are active, there is no point in summarizing the conversation yet.

What we want is to debounce and only trigger it when appropriate. After a turn finishes, schedule a recap. If the user starts typing, restart the timer. If the user submits a new prompt, cancel it. Only when the conversation has been quiet for a little while do we burn the extra tokens.

At the end of each turn, the harness does this:

if let recapConfiguration {
    let transcript = Self.presentationTranscript(from: conversation)
    recapCoordinator.schedule(
        debounceSeconds: recapConfiguration.debounceSeconds,
        presentationTranscript: transcript,
        client: secondaryClient,
        ui: ui
    )
}

That little block is the moment the furnace is lit. The assistant turn is done. We build the version of history the recap model is allowed to see, then hand it to the coordinator.

The user activity hook is where the debounce gets its shape:

let event = try ui.readInput(
    status: inputStatus,
    onActivity: { @Sendable in
        recapCoordinator.restartFromActivity()
    }
)

When the user is typing, we treat it as activity that needs to restart the pending summary countdown.

But if a recap has already been generated for the current history, typing should not start a new timer. Otherwise the harness would keep summarizing the same conversation again and again. So the coordinator keeps a tiny state machine:

private enum Phase {
    case idle
    case pending
    case completed
}

Activity only restarts the timer while the phase is pending. Once the recap is completed, the harness waits for the next submitted prompt to change the conversation before scheduling another one.

The coordinator owns the timing

The coordinator is a class that remembers the latest request, cancels the old task when needed, and starts a new countdown. It contains the core logic for the feature and is quite isolated from the harness and the UI.

final class RecapCoordinator: Sendable {
    private struct Request: Sendable {
        let debounceSeconds: Int
        let presentationTranscript: String
        let client: any LLMClient
        let ui: SynchronizedHarnessUI
    }

    private struct State {
        var task: Task<Void, Never>?
        var request: Request?
        var phase: Phase = .idle
    }

    private let state = Mutex(State())
}

The core of the task is very plain:

private func start(_ request: Request) {
    state.withLock { state in
        state.task?.cancel()
        state.request = request
        state.phase = .pending
    }

    let newTask = Task {
        do {
            for remaining in stride(from: request.debounceSeconds, through: 1, by: -1) {
                request.ui.setRecapStatus("\(remaining)s")
                try await Task.sleep(for: .seconds(1))
            }

            request.ui.setRecapStatus("generating")
            let recap = try await Agent.generateRecap(
                from: request.presentationTranscript,
                client: request.client
            )

            try Task.checkCancellation()
            request.ui.setRecap(recap)
            request.ui.setRecapStatus("done")

            state.withLock { state in
                state.task = nil
                state.phase = .completed
            }
        } catch is CancellationError {
        } catch {
            request.ui.setRecapStatus("error")
        }
    }

    state.withLock { state in
        state.task = newTask
    }
}

There is no clever scheduling system here. Just a cancellable Swift task, a sleep, and a model call. Swift concurrency takes care of the rest.

In the terminal, we update the status line to show the timer ticking and the status of the summary:

gpt-5.4 | #2 | Recaps help… | read_file,+2 | 2s | idle
gpt-5.4 | #2 | Recaps help… | read_file,+2 | 1s | idle
gpt-5.4 | #2 | Recaps help… | read_file,+2 | generating | idle

> Recaps are useful in agent chats because they preserve shared context, reduce
  drift, and make it easier to continue complex work across long or interrupted
  conversations.

gpt-5.4 | #2 | Recaps help… | read_file,+2 | done | idle

What does the recap model see?

This is the most interesting design choice.

The recap model should not receive the whole internal conversation exactly as the main model sees it. The system prompt or the skill catalog are not an active part of the thread, so adding them all would skew the summary to the wrong conclusion. Full tool output can be huge and often not useful for this little feature, so I decided to cut that as well.

This is how the harness builds a presentation transcript.

static func presentationTranscript(from messages: [Message]) -> String {
    var lines = [String]()

    for message in messages {
        switch message.role {
        case "system":
            continue
        case "user":
            if let toolResult = presentationToolResult(from: message.content) {
                lines.append(toolResult)
            } else {
                lines.append("User: \(message.content)")
            }
        case "assistant":
            if let parsedResponse = try? parseAssistantResponse(from: message.content) {
                if parsedResponse.userFacingText.isEmpty == false {
                    lines.append("Assistant: \(parsedResponse.userFacingText)")
                }
                if let invocation = parsedResponse.invocation {
                    lines.append("Assistant called tool \(invocation.name).")
                }
            }
        default:
            continue
        }
    }

    return lines.joined(separator: "\n")
}

Of course a production-grade summarization context would need more analysis and research to find what exactly works best for the specific model used. For me, the goal was just to include the conversation the user would recognize, and compress the noisy parts.

Then the actual recap prompt is also tiny:

let prompt = """
    Write a very short recap of where this agent conversation stands.

    Rules:
    - Reply with one compact paragraph of 1 or 2 sentences.
    - Aim for 40 to 80 words.
    - Do not use bullets.
    - Do not include line breaks.
    - Do not retell the whole conversation.
    - Focus only on what the user needs to remember when returning to the terminal.
    - Mention the current goal and the next useful step.
    """

Again, no magic. We build a little transcript. We ask a smaller model for a small paragraph. We display it outside the main conversation.

A useful little burn

I like this feature much more than titles. Don’t get me wrong, titles are nice, but recaps actually help me work. They solve a real annoyance of agentic tools, the feeling of returning to a long chat and needing to rebuild your own mental thread before you can type the next useful prompt.

But the lesson is the same as last time. The recap is not free. It is another inference call, started by the harness, with context selected by the harness, using a model selected by the harness. So if your harness supports a similar feature, always keep in mind what it will cost you.

That is not a reason to avoid it. It is a reason to make it visible and design it deliberately.