In our “BigData” project at an automotive manufacturer, deployment became an increasingly important point in the development of the software. Due to the increased size of the packages and the diverse environments, combined with the requirement to be able to build an environment from scratch in one day, a SCRUM story “Deployment 2.0” was created.
What did the previous deployment process look like?
The project involves diverse technologies, the major areas were:
– Oracle Database
– Informatica workflows (IICS)
– Shell scripts for control and e.g. address validation
– APEX as Oracle GUI
– The scheduling software Control-M
The deployment of individual components was previously done with a shell script, since it’s a Unix environment. The configuration took place in a control file, and the startup script had to be started manually. Also the execution had to be monitored live.
This was error prone and also very limited in parallel execution. Troubleshooting was equally difficult. But most importantly, the process could only be monitored by one person on one machine.
Our solution approach
With the change of the scheduling software from Control-M to Automic (UC4), the idea came up to automate the whole deployment process at the same time as well and to adapt it to the new requirements (e.g. GIT as versioning tool).
The packages did not have to be created and adapted manually after each development step, the sprint deployment (keyword SCRUM) should be completely controlled by tags.
Monitoring should be executable by multiple people and notification of errors should also be automatic.
First, the job network was roughly divided on the drawing board as a template. In the following picture, which is the final version of the complete implementation of the deployment, you can see the division of the respective steps.
First, a script was written to handle the control of the respective packages to be rolled out, the “preparation” group in the picture. For this the GIT API, which provides exactly such scenarios and handles the respective steps, was used. In this way, packages could be built in sprints and deployed with repeatable accuracy. In addition, all information relevant for the respective deployment was collected.
The second step was to schedule the updating of the job nets, group “automic” in the picture.
With the information now available, the packages were extracted from GIT and each split for the upcoming technology strands (group “packages” in the image).
The packages were each scheduled to run in parallel. On the one hand, this saved time, and on the other hand, in the event of an error, it was ensured that the complete deployment did not have to be repeated, but only the faulty part.
In the last group “finish” the environments were cleaned up, the logs were moved to the respective archive and the deployment was marked as successful in the database.
The control was not done via files, but via a table created for this purpose in the project database. This way, the information could also be extracted and displayed in a GUI – for a quick overview of the respective environments (DEV / TEST / PROD) with the corresponding software versions as well as other desired information.