TruckOff

Challenges

The project posed two substantial challenges. Firstly, I extracted truck data from 27 different truck retailers' websites. Each of these websites had a unique structure and provided different sets of information, making it a complex data extraction process.

The second challenge was to build modern and user-friendly filters and sort functionalities. This required developing complex sorting and filtering algorithms along with meticulous use of HTML and CSS to build the UI, providing a seamless user experience.

Solutions

To overcome the first challenge of extracting truck data from retailers' websites, I used Puppeteer. By utilizing this headless browser automation tool, I was able to programmatically access and extract data from each webpage.

My approach was to collect the links to truck listings on each page and store them in an array. Next, I looped through the array, visiting each truck page to extract the relevant data. Following the extraction, I processed the information to ensure it conformed to the required standards and then stored it in a MongoDB database.

Regarding the second challenge of building the filter and sort algorithm and its interface, I used different array methods provided by JavaScript and conducted thorough research to build the user interface using HTML and CSS.

Process

For this project, I started with backend development. Initially, I was provided with a list of 27 truck websites, which served as the primary data sources. The client's objective was to extract data from each of these websites twice a week, store it in a database, and then present this data on the website.

To achieve this goal, I utilized TypeScript, ExpressJS, MongoDB, and Puppeteer. I developed 27 data extraction scripts using Puppeteer, ensuring that each extractor successfully retrieved data without errors and saved it to the database. To keep the data updated, I set up a cron job to schedule the extractors to run as per the client's requirements. Additionally, I created an Express route to facilitate data retrieval from the database.

Next, I started working on the frontend using NextJS to build the user interface of the application. After the initial frontend build, I solicited feedback from the client, diligently addressing and implementing his suggestions. Once both the client and I were satisfied with the user interface, I proceeded to integrate it with the backend.

Upon completing the integration, I hosted the frontend application on Vercel and shared the application link with the client. The client conducted thorough testing of the site and he was extremely happy with the final product. Following his confirmation, I transferred the code to his repositories, effectively concluding the project on a mutually satisfactory note.

Feature highlights

• Pixel-perfect user interface and experience
• Implementation of the latest security standards
• 100% performance and 93% structure optimized (GTmetrix)
• Responsive to mobile, tablet, laptop, and desktop screens

Key functionalities

• Truck search
• Advanced sort and filter
• Animated trucks logos
• Dynamic categorization