I built a job discovery agent. Here is what actually happened.

cv-tailor already let you build a profile and generate a clean ATS-proof CV. But finding the right listings to apply to was still manual. So I built the agent to fix that.

The idea was simple. Every day at 17:00 the agent scrapes job boards, sends each listing through an AI model to score how well it matches a profile, saves the good matches to a database, and sends a formatted email digest. Listings already seen are tracked so they never surface twice.

Simple enough in theory. Here is what actually broke.

I deleted data before saving the new data

The CV import feature let users upload their existing CV to auto-populate their profile. The original implementation deleted all existing profile entries first, then inserted the parsed ones. If the insert failed halfway through, the user was left with an empty profile and no way to get it back.

I reversed the order. Insert first, delete after. Not a perfect solution but a much safer one. If the insert fails, the old data is still there.

The 504

This is the interesting one.

The agent processed listings one by one. Fetch a listing, call the AI, wait for the response, move to the next one. With 30 to 50 listings per run and each AI call taking a couple of seconds, the total execution time pushed well past 60 seconds. Vercel serverless functions time out at 60 seconds. The function was being killed before it finished.

The fix was parallelism. Instead of waiting for each listing to finish before starting the next, the agent now runs five analyses simultaneously. Total execution time dropped significantly.

I also capped the number of listings analysed per run and pre-marked all unseen listings as seen before starting analysis. That way listings beyond the cap do not resurface in the next run even if they were never analysed.

Why Redis

The natural question is: why Redis and not just a database column in Postgres?

Three reasons.

The first is automatic expiry. Seen-listing records in Redis expire after 30 days automatically. After that, old listings can resurface if they are still being posted. A database column would need a cleanup job. Redis handles it natively.

The second is the distributed lock. When the agent runs it sets a lock in Redis. If a second run is triggered while the first is still executing, the second run sees the lock and exits immediately. The lock also has a timeout, so if the agent crashes without clearing it the lock heals itself.

The third is where this is going. The next feature is preference learning. Every time a user discards a listing, saves one, or marks one for later, that action feeds a signal into Redis. Over time the agent builds a weighted preference model from those signals and uses it to adjust future scoring. Redis is the right place for this kind of fast, structured, ephemeral state. It is not just deduplication storage — it is the agent’s memory.

The honest workarounds

A few things in the current implementation are fragile and I know it.

The progress bar showing the agent working through four stages is theatre. The stages fire on a timer regardless of what the server is actually doing. It looks like real-time feedback. It is not. It will get replaced with proper server-driven updates eventually.

The CV adaptation download uses pattern matching to apply AI suggestions to the PDF. It works until the AI changes how it formats its output. It will break eventually.

Both are known. Both will get fixed when they cause actual problems.

What I actually learned

Sequential AI calls do not scale on serverless functions. This was entirely predictable in retrospect. Parallelism with a concurrency limit is the pattern and I should have started there.

Being your own first user in production surfaces problems fast. Ship something that works, then look at what broke. The current implementation has workarounds. They are documented, understood, and will get fixed when they matter.

That is what building actually looks like.