Wrapping a 100k-row CSV or API sync inside a single DB::transaction() is a trap—one malformed row at 95% either rolls back the entire run or leaves your database half-imported with no clean way to resume.
I built laravel-data-pipeline to handle streaming ETL in Laravel without custom state-tracking boilerplate in every job:
• Streaming ETL & Batch Validation to keep worker memory flat
• In-flight Deduplication before hitting the database
• Transactional Checkpoints so progress commits safely in chunks
• Quarantine Recovery to isolate bad rows and replay them later without killing the run
Repo: https://github.com/robelsust/laravel-data-pipeline
How are you currently handling poisoned rows in bulk Laravel imports—dead-letter tables, Bus::batch(), or custom chunking?
GitHubGitHub - robelsust/laravel-data-pipeline: Streaming ETL pipelines for Laravel with batch validation, transactional checkpoints, deduplication, and quarantine recovery.Streaming ETL pipelines for Laravel with batch validation, transactional checkpoints, deduplication, and quarantine recovery. - robelsust/laravel-data-pipeline
commit 9b14759