Fixing Serverless Function Timeouts When Polling YouTube Creator Analytics for D2C Campaigns

The Problem

When building post-campaign influencer monitoring for Indian D2C brands at PulseROI, cron jobs fetching live YouTube creator engagement metrics hit Vercel's 300-second serverless execution limit and time out halfway through processing 50+ active creator profiles.

What I Tried First (and Why It Failed)

My initial approach was a straightforward sequential for...of loop running inside a Next.js API route triggered by cron-job.org:

// The naive approach that timed out on creator #23 export async function GET(req: Request) { const creators = await db.query('SELECT * FROM active_creators'); for (const creator of creators) { // Each YouTube Data API v3 call takes ~800ms to 1.5s const metrics = await fetchYouTubeMetrics(creator.channel_id); await db.query('UPDATE creators SET stats = $1 WHERE id = $2', [metrics, creator.id]); } return Response.json({ success: true }); } 

This failed because sequential polling of 50 creators meant total execution time regularly breached 45 seconds, and when YouTube's API throttled us with a 429 status, the entire batch failed, rolling back nothing and leaving telemetry half-updated.

The Actual Fix

To fix this without upgrading to expensive dedicated worker servers, I refactored the polling logic into parallel chunks using Promise.all coupled with an explicit Promise.race timeout wrapper (enforcing a hard 25-second limit per batch) and comma-separated API key rotation for Gemini and YouTube data fetches.

// The production fix: batched parallel processing with a 25s timeout guard export async function GET(req: Request) { const creators = await db.query('SELECT * FROM active_creators LIMIT 50'); const batchSize = 10; for (let i = 0; i < creators.rows.length; i += batchSize) { const batch = creators.rows.slice(i, i + batchSize); const promises = batch.map(async (creator) => { const timeoutPromise = new Promise((_, reject) => setTimeout(() => reject(new Error('YouTube API timeout')), 5000) ); const fetchPromise = fetchYouTubeMetricsWithRotation(creator.channel_id); return Promise.race([fetchPromise, timeoutPromise]); }); try { const results = await Promise.allSettled(promises); // Process results and bulk update Neon Postgres using tagged templates await persistBatchMetrics(results); } catch (err) { console.error(`Batch ${i} failed:`, err.message); } } return Response.json({ status: 'completed_with_batches' }); } 

What Surprised Me

Using Promise.allSettled instead of Promise.all was the turning point. When one single creator channel returned a 404 or private status, Promise.all threw an uncaught error that aborted the entire cron run. Promise.allSettled let us log the specific failed channel ID to our Turso state store while successfully committing metrics for the other 9 creators in the batch.

The Result

Cron job completion time for 50 creator profiles dropped from a failing ~52 seconds (or outright timeout) down to a stable ~4.2 seconds. Database connection pressure on Neon serverless also decreased because updates were bundled into single-transaction bulk queries rather than 50 separate round-trips.

Where to Go From Here

Next, I am working on adding automated Slack alerts whenever a creator's engagement drops below baseline post-campaign thresholds. If you want to see how this telemetry integrates into our full automated monitoring stack, check out PulseROI.