DBI_System_Log

Overview

Almost every event that occurs in this system is logged into this MySQL metadata table.  There are typically 3 events for most processes to denote the following: when the process is Initialized, Started, and Ended.  Several other specific metadata information is captured and further described below

Table Details

  • Event_TS – The data and time that the event descibed in this detail record occured
  • DBI_Process_Name – This is the DBI process name that is executing
  • Table_Job_Name – This field is used for either the table or job name that is being processed
  • File_Step_Name –  This field provide the file being processed, or if the DBI_Activate process, it includes the step code (linux command) that is being executed.
  • Message – This is a message from the system that type identifies when a process is Initialized, Started or Ended.  This also includes the type of process that is executing.
  • Rows – The number of rows being processed.  This will not appear in all of the metadata records, but specific ones where available.  Query the data to determine which ones.
  • Run_Time – The time that the process took to execute.  This is typically only available on the ‘Ended’ record.
  • Grouping_Name – This is an optional name that is provided for reporting purposes.  If you want specific records to be tagged, the grouping name will need to be configured in the scripts application .ini.
  • Batch – This is a sequental number that provide sequencing of a process.
  • Cluster_Name – This provide the name of a Redshift cluster where the processing occurs.  This is valuable when multiple Redshift clusters are being used.
  • Size – This is the size of the file in bytes as related to the file that was being processed.

Reporting

The metadata can be used in creative ways.  The following is a pre-built report that will show when an DB extract has started, and loaded into Redshift.  These are events coming from both the DBI_DB_Extractor and the DBI_RS_Loader.

Advertisements