TruckOff

Challenges

Each truck retailer used different page structures, labels, and content formats. Automating the data scraping process required building site-specific extraction logic and handling inconsistencies in vehicle details like pricing, features, and availability.

Another challenge was that the client wanted a modern eCommerce-style interface with sorting, filtering, and responsive design—on top of a large and growing dataset. Maintaining snappy UX while applying complex filtering logic on nested fields posed a significant challenge.

Solutions

To automate data collection, I used Puppeteer to simulate user interaction with each retailer's site. I first scraped truck listing URLs, then visited each listing page to extract fields like title, price, specs, and images. I built normalization logic to clean and structure the data, then stored it in MongoDB.

For filtering, I engineered a client-side filtering engine using optimized JavaScript array methods and MongoDB queries for server-side efficiency. I designed a responsive UI using Next.js, ensuring the filtering interface remained intuitive and performant—even with large datasets and deeply nested attributes.

Notable features

• Automated data pipeline for 27 truck retailers using Puppeteer
• Advanced filter engine with sort, category toggles, and search
• Mobile-first, pixel-perfect responsive design
• MongoDB integration for dynamic inventory