DBI_System_Manager

Overview

In order for any of the DBI operations to execute automically, the system manager allows for services to be started and managed.  Several other utilities are part of the menu structure, but if you only know the first 3, then this will provide you 90% of the this scripts purpose.

DBI is typically configured with 3 services to be executed, and this is driven by related parameters in an .ini file.  The services include the following:

  • DBI_Activate_Repeater_Service – This service allows any script or linux command to be executed on a repeating schedule.  Once this service is started, and the services that it starts are found and configured in a metadata table ‘dbi.dbi_activate_repeater’ in MySQL.  Once any parameter is changed in this table, they are dynamic to the process.  Therefore, if you set the ACTIVE_FLAG to ‘N’, that process will stop.  If you change the Interval, it will subsequently become effective.  Several of the basic services on install are as follows:
    • DBI_ACTIVATE
    • DBI_ACTIVATE_RESTART
    • DBI_FILE_COPY
    • DBI_SYSTEM_MONITOR_DISK_SPACE
    • DBI_SYSTEM_MONITOR_FILE_AGE
    • DBI_SYSTEM_MONITOR_FILE_DEPTH
    • DBI_SYSTEM_MONITOR_S3_AGE
    • DBI_SYSTEM_MONITOR_SERVICES_CHECK
    • DBI_SYSTEM_MONITOR_SYSTEM_LOAD
  • DBI_RS_Loader_Service – This services allows for object in the AWS S3 Inbox to be processed in Redshift.  This service will look for objects that have not been processed and a like object that is not already being processed.  If true, the object is copied to the subfolder ‘processing’, and an instance of the DBI_RS_Loader.sh script is called to process this object.  Once the object processing completes, if successful, it will be copied into the subfolder ‘processed’.  But if there was an error, it will be copied to the subfolder ‘reject’.
  • DBI_Services_Manager – This service acts as an independent monitor for the above services.  If a failure exists, this service can be configure to automatically restart a critical service.  This typically only happens the server runs out of memory causing a process to fail.  This service would be best to run on a seperate EC2 instance.

Usage

Usage: DBI_System_Manager.sh -f <config file name> -t <Request Type> -s <Service Name> -i <Service INI Name> -p <Instance>

DBI_System_Manager.sh -f <config file name> -t <Request Type> -s <Service Name> -i <Service INI Name> -p <Instance>
Example A: DBI_System_Manager.sh -f DBI_System_Manager.ini -t disk_status
Example B: DBI_System_Manager.sh -f DBI_System_Manager.ini -t service_status -s DBI_RS_Loader_Service
Example C: DBI_System_Manager.sh -f DBI_System_Manager.ini -t service_status -s DBI_Activate_Repeater_Service -i DBI_Activate_Repeater_Service
Example D: DBI_System_Manager.sh -f DBI_System_Manager.ini -t 3 -s DBI_Activate_Repeater_Service -i DBI_Activate_Repeater_Service_Finance -p MAIN_AP_DB_Extractor

Below are the Request Type options. Use either the number or name in the above commands

This script provides the basic commands for managing the database. The following options are as follows:
1) service_start – Starts the services for managing processes by executing the associated scripts
2) service_stop – Stops the services that are running
3) service_status – Shows the status of the services
4) disk_status – Provides the disk status and NAS share folders largest in size on the local Linux server
5) code_compare – Shows the difference between code in dev and qa
6) dbtest – Executes a simple SQL query against the various Redshift databases for status
7) dependency_check – Checks .sql file for the existence of tables in the various clusters
8) query_status – Shows the active queries and the query queue for the various Redshift databases
9) linux_processes – Shows the specific applications running on the local Linux server
10) linux_process_orphans – Shows the Linux process that are deemed to be orphans
11) linux_process_cleanup – Kills all fo the Linux process identified using the above option
12) sessions – Shows the sessions connected to the various Redshift databases
13) versions – Displays the various shell scripts comparing timestamps
14) whatsup – Provides status of the DB and servers
15) google_sheet_start – Starts the service for executing Google Sheet integration services via Python script
16) google_sheet_stop – Stops the service for Google Sheet integration
17) google_sheet_status – Shows the active Google processes

Advertisements