TTL: A transference, transformation and loading approach for active monitoring
Date of this Version
In Data Warehouse (DW) environments, operational processes move data from sources to the warehouse. This includes data export, preparation, and loading usually performed using Extraction, Transformation and Loading (ETL) tools. Past research has treated DW ”as collections of materialized views” whose data is regularly refreshed and locally stored . Requirements have changed and real time transactions are required to support on-line operational decision making. Traditional DW systems may impose unacceptable delays due to their batch nature. ETL techniques are difficult to scale up to address the challenge of data loading, performance and low latency to provide real-time decision support. We propose a new approach for designing real-time DW in which traditional ETL does not apply. Data is pre-analysed by agents in each data source before being pushed as needed to the DW. The approach has been evaluated in a simulated environment and some of the results are discussed here.
This document has been peer reviewed.