Processing Millions without Memory Crashes: Laravel Lazy Collections

The Out-of-Memory Collection Wall

When developing an enterprise B2B SaaS platform at Smart Tech Devs, processing large arrays of internal data is inevitable. Imagine an analytics routine or a custom reporting system that loops through 500,000 user logs, matches specific metadata filters, and map-transforms the arrays into a formatted payload structure. The standard Laravel approach is to use standard Eloquent collections: $logs = ActivityLog::all();.

For small database tables, this works perfectly. But at enterprise scale, this approach is an operational hazard. When you invoke a standard collection, Laravel attempts to load all 500,000 records into system RAM simultaneously, wrapped in heavy object containers. Your PHP process will instantly hit its physical boundaries, throwing a fatal Allowed memory size exhausted error. To process massive datasets cleanly, you must decouple your collection processing arrays from immediate RAM allocations using Lazy Collections.

The Power of PHP Generators

Introduced in Laravel 6 and powered natively by PHP Generators under the hood, Lazy Collections allow you to manipulate millions of rows while keeping your application memory consumption virtually flat at a few megabytes.

Instead of compiling a massive dataset completely in memory before executing loops, a Lazy Collection yields a single item at a time. The rest of the data sits quietly inside the database stream, loading just-in-time and dropping immediately out of RAM once processed.

Step 1: Swapping Query Builders to Cursors

To convert an expensive database extraction into a memory-safe stream, simply chain the cursor() method onto your query. Laravel will instantly return a LazyCollection instance.

namespace App\Http\Controllers\Api;

use App\Http\Controllers\Controller;
use App\Models\ActivityLog;
use Illuminate\Http\Request;

class LogProcessingController extends Controller
{
    public function processTelemetry(Request $request)
    {
        // ❌ THE ANTI-PATTERN: Pulls 500k rows into memory at once
        // $logs = ActivityLog::where('status', 'error')->get();

        // ✅ THE ENTERPRISE PATTERN: Keeps memory flat at less than 5MB!
        $logs = ActivityLog::where('status', 'error')
            ->cursor(); // Returns a LazyCollection

        // We can cleanly chain array operations without ever loading the full dataset into RAM
        $processedData = $logs->filter(function ($log) {
            return $log->is_critical_breach;
        })->map(function ($log) {
            return [
                'event_id'  => $log->uuid,
                'client_ip' => $log->ip_address,
                'logged_at' => $log->created_at->toIso8601String(),
            ];
        });

        // The database connection streams rows on-demand only when the loop starts
        foreach ($processedData as $item) {
            app('telemetry.dispatcher')->send($item);
        }

        return response()->json(['status' => 'pipeline_complete']);
    }
}

The Engineering ROI

By implementing Lazy Collections across your heavy calculation systems, you achieve predictable system scaling. Memory allocation curves shift from a steep exponential spike down to a flat line. Your server can comfortably parse large analytics tables back-to-back, leaving your hardware resources unbothered and available to handle standard API endpoints without service interruptions.