The long-standing employees provide interesting insights into their projects and place the architecture aspect in the context of their projects. The interviews illustrate in-factory's high level of methodological and implementation expertise in the field of data architectures and data modeling of data warehouse systems in project practice.
In the various projects, we encounter different situations in terms of architecture ("DataVault vs. classic system") and different process models ("agile vs. waterfall") and development processes in the field of data warehousing. Our colleagues report on the individual projects as follows.
Carsten Schlotmann - Project in the automotive sector
Carsten Schlotmann - Senior Consultant
Carsten, can you say something about yourself?
I have been with in-factory for more than 12 years and have been involved in data integration and business intelligence in the broader sense for over 20 years. I currently work primarily in the area of data and process architecture and am - hopefully in personal union - a "busy" SCRUM master in my project.
What project is this and what is your use case?
We are working on the development of a central integration platform for customer data for a large German automotive group, which is based on a classic data warehouse architecture. The use case is the receipt, integration and distribution of customer data to support the sales process. This central integration platform is constantly being expanded and adapted. Projects with clear framework conditions, such as pre-defined tasks with start and end dates, form the basis for this.
Ultimately, driven by prioritized use cases, we replaced a legacy system for campaign support and are now continuing to develop the new solution in order to support innovative sales processes (lead management) in a timely manner. The legacy system will continue to be operated in parallel.
Which architectural approach did you choose and why?
The legacy system had weaknesses in terms of scalability. This is primarily expressed in a non-linear increase in complexity with new use cases. The Data Vault approach counteracts this effectively; new use cases can generally be implemented quickly without having to adapt existing logics. In addition, the model remains explainable, a key factor in quality assurance in requirements engineering. It is striking that the architectural approach has carried us unchanged through a complete refocusing of the project, from a batch-oriented campaign backend to a near/realtime data hub for lead processing. This saves both time and money.
Specifically, how do you adapt your data architecture when integrating new use cases?
In the simplest case, these are attribute additions to existing satellites. However, this is not always the case. As a rule, additional satellites are required at the hubs and occasionally at the link tables. Lineage is an important issue for us, so the additions are mirrored in the integration layer and in staging in order to keep transformation steps reproducible. Existing ETL routes generally remain unchanged, new ETL routes can be derived from the existing ones. Processes are organized according to vault object classes and are therefore easy to adapt.
How can I imagine the coexistence of legacy and new systems?
The new system was originally implemented for markets that were not covered by the old system. This has changed. Currently, the delimitation is more use case-driven. Lead management is carried out in the new system. Campaigns are still largely managed in the legacy system, but here too, conversion is currently taking place, driven by the introduction of a new campaign tool.
What is special about you?
We have been agile since the beginning and have been very successful with it. The Data Vault approach optimally supports the agile approach and is also forgiving of the odd adjustment in the requirements. The stable momentum and high project transparency that we gain from the combination of SCRUM and Data Vault make the project stand out time and again.
