Automotive Data.

Case Study

Learn More

Sign up for a free 30 min consultation meeting.

Objective:
Generate Data For DMS (Dealership Management System)

Extract data from multiple XML files and load it to a data warehouse. The data comes from multiple dealers, and it contains vehicle sales and vehicle repair data (VW, Lexus, etc.).

The Challenge

  • Daily XML files (source) are relatively large in size.
  • The data inside the files contain multiple child-parent relationship, some of which are not properly structured.
  • Loading a large historical data with over 100 GB in size.

The Solution

  • Using ETL component that would flatten the XML files and map it against the table schema before the load process.
  • Creating optimized queries for faster transfer and load (indexes added).
  • Store the in-process data at the memory location instead of a physical location before loading it to the data warehouse.

Features Added

  • Error handling with log registration.
  • Notification alert for successful or failed run.
  • Automating the ETL to run periodically.

Newsletter

Subscribe For Awesome Articles