Rebuilding a 13-stage analytics pipeline
2025 – 2026Technical lead · single-family, then multifamily
94 modules, about 16,500 lines, and a CSV file written to disk between every one of thirteen stages — so multi-gigabyte files were re-read and re-parsed roughly twenty-one times per run. Six hours per ticker. But the disk round-trips were only half of it. One algorithm was doing far more arithmetic than the data justified:
One home’s rent over a year
0 rows evaluated
6 h → 1.5 minCompute per ticker
6 GB → 18 MBRaw storage
94 → 42Modules
195× / 90×Rows / speed