A daily job that scraped a few hundred articles, summarized each with an LLM, ranked them, and published the top ones. It worked on my machine with ten articles. In production, on day three, it died two-thirds of the way through, and cheerfully started the next run by re-summarizing everything it had already done.
TL;DR: A non-conversational LLM job is a data pipeline, not a chat session. The instinct to model it as “one clever prompt in a loop” gives you the three classic failures of a bad batch job: no resumability (a crash re-does everything), no idempotency (re-runs duplicate work and cost), and no separation of concerns (one call both generates and judges, and you can’t debug either). Model it as ETL instead (discrete stages, a checkpoint after each unit, and idempotent writes keyed by stable identity) and it becomes delightfully boring in the way good infrastructure is boring.
The one-big-prompt trap
The first version was the obvious version. A function that, for each article, called the model with a prompt that said, roughly: summarize this, score how interesting it is, and tell me its topic. Loop over the articles, collect the results, sort by score, publish the top ten. One prompt doing everything, wrapped in a for.
It demoed beautifully. Ten articles, thirty seconds, a clean ranked list. The demo proved the prompt worked. Production proved the architecture didn’t.
Production is where “a few hundred articles” and “a flaky external scrape” and “a model endpoint that occasionally times out” all live. The run died partway. And because the whole thing was one in-memory loop, dying meant:
- Everything it had summarized was gone, held in a list that evaporated with the process.
- The next scheduled run started from zero and paid, again, to summarize the same articles it had already summarized yesterday.
- When a ranking looked wrong, I had no way to inspect the summary separately from the score, because a single opaque call had produced both.
Three different problems, and they all trace back to one modeling mistake: I’d built a conversation (stateful, in-memory, all-or-nothing) when what I needed was a pipeline (staged, persisted, resumable).
Strip the LLM away and what remained was almost mundane: batch data movement with a very expensive map in the middle. Every hard-won lesson about batch jobs applied; I’d just skipped all of them because there was a model involved and it felt like something new.
Stages, not a monolith
The first move is to stop asking one call to do three jobs. Scraping, summarizing, and ranking are different operations with different failure modes, different costs, and different reasons to re-run. Split them into stages that each take an input and produce a durable output:
Each stage reads the previous stage’s persisted output and writes its own; the job is the seam between them, not a single call:
Scrape Summarize Rank Publish ──────── ─────────── ────── ───────── fetch raw → LLM: 1 article → LLM: score → pick top N articles → 1 summary a batch write feed │ │ │ │ ▼ ▼ ▼ ▼ [raw store] [summary store] [ranked store] [live feed]
Two things fall out of this immediately.
Generation and judgment come apart. Summarizing an article (“what does it say?”) and ranking it (“how much does it matter, relative to the others?”) are genuinely different tasks. A summary is an independent transform; ranking is a comparative operation. Fuse them into one call and you force the model to invent a global judgment from local context: to rank something while it can only see one of them. They contaminate each other, and you can’t even tell whether a bad result was a bad summary or a bad score. Split, the summary stage is a pure per-item transform, and the ranking stage sees the whole batch at once, which is the only way to rank anything sensibly. (Ranking one item in isolation is just scoring, and scoring-in-isolation is exactly how you get ten articles all rated “8/10, highly relevant.”)
Each stage becomes independently re-runnable. The scrape failed but the summaries are fine? Re-run scrape only. The ranking prompt improved? Re-run ranking over yesterday’s summaries without paying to regenerate a single one. Improving the ranking prompt shouldn’t invalidate thousands of perfectly good summaries, and with stages, it doesn’t. A monolith can only be re-run whole.
Checkpoint after every unit
Stages give you where to resume. Checkpointing gives you the state to resume from. The rule is small and it changes everything: persist the output of each unit of work the instant it’s produced, before moving to the next. Not at the end of the batch, at the end of each item.
Concretely, the summarize stage doesn’t build a list in memory and save it at the end. It writes each summary as it completes:
async def summarize_stage(articles, store):
for article in articles:
if await store.has_summary(article.id): # already done on a prior run, skip
continue
summary = await llm.summarize(article.text)
await store.put_summary(article.id, summary) # durable BEFORE the next itemNow a crash at article 200 of 300 costs you article 200, not articles 1–200. The next run’s very first act is to skip everything already in the store. Resumability stops being a feature you build and becomes a property of the shape: the store is the checkpoint. Frameworks like LangGraph formalize this with a checkpointer/saver that snapshots graph state to a durable backend, so a long agent run can crash and resume mid-graph, but the idea predates any framework. It’s just: write it down before you take the next step.
Idempotency is what makes re-runs free
The has_summary check above is doing quiet, load-bearing work, and it’s worth naming what it is: idempotency keyed by stable identity. Run the pipeline once or five times over the same input, and the result (and the cost) is the same, because completed work is recognized and skipped rather than redone.
The key has to be a stable identity, not a position. That identity might be a URL, a content hash, or an upstream document ID, anything that names the same item across runs. Never key on “the 4th article in today’s list,” because tomorrow’s list is different and position means nothing across runs. Get identity right and two things you were dreading become non-events:
- The daily overlap. Most days, most of the “new” articles overlap with yesterday’s crawl. With content-keyed idempotency, the overlap is free, recognized and skipped. Without it, you re-summarize (and re-pay for) the same articles every single day.
- The retry. A flaky external dependency stops being scary. Just run the job again. Idempotency guarantees the retry only does the work the first attempt didn’t finish.
The same 300-article day, before and after; the win is entirely in the work you don’t repeat:
Naive loop re-summarizes all 300 every run, loses everything on a crash Staged + idempotent summarizes only the ~40 genuinely new, resumes a crash for free
Stages are observable for free
There’s a fourth property that falls out of staging, and it’s the one you’ll be grateful for at 2am: a staged pipeline can tell you where it is. Every stage answers three questions independently, how many items entered, how many succeeded, how many failed. A monolithic prompt-in-a-loop can only ever report “the run failed.” A staged pipeline reports “scraping completed with 312 articles, summarization reached article 183 and then the model endpoint started timing out, ranking never started.” That’s the difference between debugging a process and debugging a black box, and it’s the same lesson as reading logs: you can only reason about the steps you can actually see.
What was left over: the actual chatbot
Here’s the irony that named the post. After splitting all this out, there was still a genuinely conversational piece of the product, a chat agent users could ask questions of. And it wanted the opposite properties: stateful, turn-by-turn, memory across a session. That’s a real chatbot, and it deserved real conversational machinery (a graph with tool nodes, per-session checkpointing so a conversation survives a redeploy).
The mistake was never “using an LLM in a loop.” It was letting the chatbot shape (stateful, all-at-once, one clever prompt) leak into the batch work, where it’s precisely wrong. The daily job isn’t a conversation that happens to run on a schedule. It’s ETL that happens to call a model in its transform step. Once I let it be a data pipeline, everything it had been bad at (crashing, repeating, hiding its own reasoning) it simply stopped doing.
The one line to remember
Reach for a model and it’s easy to think everything downstream is a conversation. Most of it isn’t. Most LLM work in a real product is batch data movement with a very expensive map in the middle, and every hard-won lesson about batch jobs still applies, doubly so when each step costs money.
If it runs on a schedule and processes a batch, it’s a data pipeline. Stage it, checkpoint every unit, key idempotency on content, and save the conversation for the part that’s actually a conversation.