Generally, with the development and expansion of business, there are more and more production lines and more and more data are generated. These data collection methods, original data formats, data volumes, storage requirements and usage scenarios are quite different.
Data extraction refers to extracting ODS source data into DW, pulling the data and displaying it after cleaning. Generally, the cleaned data is loaded into mysql, and then used in various systems, or directly displayed to relevant personnel using Tableau. Metadata management system is necessary for data warehouse, and relevant personnel must maintain it regularly. If the changes of metadata and data warehouse are not synchronized, then the metadata system is useless.
Extended data
The processes described by ETL generally include ETL or ELT, which are mixed. Generally, the larger the data volume, the more complex the conversion logic and the stronger the destination computing power. In order to take advantage of the parallel processing power of the destination database, the more inclined it is to use ELT. The purpose is to integrate scattered, messy and inconsistent data in enterprises and provide analytical basis for enterprise decision-making, which is an important link of business intelligence projects.
References:
Baidu encyclopedia -ETL