Unification of systems, processes and data following a complex acquisition

In large organisations, especially those that have grown through acquisitions, it is common to encounter legacy systems, disparate platforms and duplicate processes. 

The parent company faces a lack of standardisation in its practices, which hinders information traceability. This leads to redundancies, errors and coordination costs. Everything becomes even more complicated when there are different regulatory frameworks, formats and internal practices.

This was the case for our client who, after integrating several companies with different structures, needed to implement a single corporate web platform common to all of them. Not only that, but this platform should only contain data from each of the acquired companies that the parent company considered relevant, complete and consistent. In other words, a selective migration of relevant data. A complex migration that ensured the new platform would integrate seamlessly with their infrastructure and allow for adaptation to future requirements and functionalities.

The challenge: unifying processes and data following a company acquisition

The project involved the development of a single enterprise web platform that would ensure an efficient transition from the previous system and allow implementation without disrupting the normal operations of each company. The application would have to support technical teams working in the field, recording data in different environments and with high documentation requirements. Main requirements:

• Handling large volumes of data, ensuring fast, real-time access.
• Adaptation to multiple heterogeneous regulatory environments, each with different local needs, regulations and formats.
• Migration from old systems without a standardised structure.
• Unification of processes to improve quality and reduce operational errors.
• Guaranteed real-time performance and synchronisation for more than 2,000 concurrent users.
• Flexibility to integrate with other systems and incorporate future functionalities.
• Field data capture with different devices, signature capture and voice dictation.

Solution and technology

To respond to these challenges, Xeridia developed a modular and scalable web application. Capable of adapting to the particularities of each company in the group without losing consistency in data and processes:

• Responsive web platform for use from any device.
• Enhanced security: penetration testing and regulatory compliance.
• Specialised modules for field data capture, including voice-to-text and multimedia files.
• Customised migration plan, compatible with disorganised data structures and inconsistent relationships.

A heterogeneous data migration with YAML and Python

The source data was unstructured, without foreign relationships and with dirty data. The main challenge was to develop a solution that not only cleaned and structured this information but was also flexible enough to adapt to the constant changes that arose as the client’s specifications for the final system evolved.

To achieve this, we created an innovative migration architecture based on a YAML file system. We created our own declarative language within these files that allowed us to generate SQL queries dynamically. This approach gave us the ability to quickly define and modify relationships between multiple source tables, select specific fields, apply filters and groupings, all without having to rewrite complex SQL queries from scratch.

The great advantage of this system is its agility. We could alter both the source and destination fields directly in the YAML files, allowing for almost immediate adaptation to new requirements. In addition, to ensure data quality, we integrated the ability to apply custom Python functions (called adapters and transformers) at the column or table level. This allowed us to implement data cleaning, transformation, and validation logic in a modular and reusable way, ensuring a complete and high-quality end result.