We Built a CV Scanner That Compares Skills Against Real Jobs. Then Production Exposed a Bottleneck.

Building an AI-powered application is one thing. Getting it to behave reliably when someone actually uses it is another.

I've been building Glassbox, a tool that compares the skills on a CV against skills mentioned in real tech job descriptions.

The idea is straightforward: instead of assigning someone an unexplained career score, show them where their documented skills overlap with what employers are asking for.

But the engineering turned out to be much more interesting than the original idea.

How the pipeline works

At a high level, Glassbox:

1. Reads the uploaded CV and extracts its skills.
2. Retrieves relevant tech job descriptions.
3. Normalizes and compares the extracted skills against the job market.
4. Uses title embeddings to help match relevant roles.
5. Produces an analysis of skills in demand, overlaps and potential gaps.

The application uses a Next.js frontend, a Python backend, PostgreSQL, and an ONNX-based embedding component.

One design principle has been important throughout: the results need to be explainable. Seeing that six of the ten most-requested skills for a role are already on your CV is more useful than receiving a mysterious 60% hiring score.

When production started struggling

A recent production investigation uncovered a particularly expensive part of the pipeline.

We measured two live scan requests:

PDF extraction was almost negligible.

The surprising part wasn't the AI model itself. It was the amount of repeated work being performed around it.

Embedding thousands of job titles for each comparison could consume more time than extracting CV skills and scoring results combined.

Earlier, we had also investigated a memory-sensitive embedding path and introduced bounded inference batches of 32 inputs.

Introducing a bounded embedding cache

We then implemented a cache for job-title vectors.

The reasoning was simple: job titles frequently repeat across requests. If the same title has already been embedded by the same model, we shouldn't perform the inference again.

We deliberately kept the cache bounded at 4,096 entries instead of letting process memory grow indefinitely.

Importantly, CV-derived profiles remain outside that cache.

We benchmarked the change against the same synthetic workload containing 4,000 jobs.

The fixture-backed responses produced identical JSON hashes before and after the change, including the job ordering and the 3,893 jobs retained by the existing role filter.

That mattered. A performance improvement that silently changes the analysis is not necessarily an improvement.

The important caveat

These were local synthetic benchmarks, not production end-to-end measurements after deployment.

Only three runs were performed, so they aren't enough to establish reliable latency percentiles.

And there's another constraint: the backend can scale to zero, meaning an in-memory cache is lost whenever the container shuts down.

Cold scans still need embedding, and job retrieval alone was taking roughly 16 seconds in the measured production requests.

So caching helps, but it doesn't finish the optimization story.

What building Glassbox has taught me

Three lessons stand out:

Measure stages separately. An endpoint taking 50 seconds tells you very little until you know where those seconds are going.

Preserve behavior when optimizing. Identical output and bounded memory mattered just as much as lower latency.

Building a working AI demo isn't the same as building a usable product. Loading states, production failures, long waits and recovery paths are part of the user experience.

We're still improving Glassbox, and I'm interested in hearing how other developers approach caching, CPU inference and latency in resource-constrained applications.

If you're curious about the product itself, you can try it here:

Glassbox

And if you've built a similar pipeline, I'd especially love to know: how do you balance cold-start costs against keeping an inference service warm?