文档介绍：etl开发原则(ETL development principles)
1 ETL development principle
The development of ETL currently using IBM Datastage Designer tool, the client developers on the client development tools, development method using remote login to the server does not allow the development of the principle of ETL involves many aspects, this document according to the classification of DataStage objects, from large to small Project, Job, Stage etc. a few blocks to describe:
the use of ETL database system account permissions
ETL relates to the source system and the target system, when using parallel JOB development, the database system account permissions have certain requirements, according to the requirements of the Datastage document, the Oracle account needs at least the following system permissions:
- DBA_EXTENTS
- DBA_DATA_FILES
- DBA_TAB_PARTITONS
- DBA_TAB_SUBPARTITONS
- DBA_OBJECTS
- ALL_PART_INDEXES
- ALL_PART_TABLES
- ALL_INDEXES
$INSTANCE (Only if Oracle Parallel Server is used)
Project establish principles and essential content
Project Job and Routine DataStage is stored in the object, an application may have multiple Project. In principle, a Project and a corresponding target system, all Job for unified management of loading of the target, but considering the server performance and the total number of Job and other special circumstances, in some cases for a target system to load the job can also be split into multiple Project.
In order to facilitate unified management, must have the following contents in each Project:
DayOpen & DayOff
Each Project should have a set of DayOpen job and a set of job DayOff, mon thing is that IShDayOpen calls IMaDayOpen, and IShDayOff calls IMaDayOff job. The operation of DayOpen marks the beginning of the day Job, the main task is to do some initialization operations, such as: create the day's rundate directory, files, modify the parameters in and so on. The operation of DayOff represent