DBI_File_Copy

Overview

The major benefit for this service is to replicate files or all of the files in a folder to another location without any scheduling.  Thus, this service will copy a file once it is found in the source location. This is achieved by configuring a single rule in the rules file, then placing a file in the designated source location for processing.  A pertinent use case for this service allows production files to be shared with other DB instances to allow for DB synchronization, or just sending files from one location to another as required.

Once the rules are entered into the rules file, the service continually looks for files to process, so anytime a file appears in a source directory, it is processed automatically. When copying a file between shares or servers, the MD5SUM checksum function always verifies the data integrity. All of the error handling and user notifications can be configured likewise.  Users appreciate the simplicity as you only need to define the rule, and then the service does all of the work dynamically. Additional functions with this service allow for a file to be time stamped, zipped, unzipped, encrypted, decrypted, renamed, and an exit call for custom functions.  Additional options for file archiving, retention management, and email/text messaging after completion is also available.

Automated rules driven file mover between FTP, S3, Google Drive, Google Team Drive and Linux.

  • Files are moved within minutes upon arrival  
  • Can be for specific files, or ALL files that are appear in a folder
  • Allows for applying encryption, decryption, gzipping, unzipping and timestamps

Table Rules

  • Rule_Name – This is a unique rule name that can be used to call this rule in batch mode
  • Service_Flag (Y/N) – This flag allows this rule to be executed automatically.  The service used to automate this processing is managed using DBI_System_Manager, and the process is defined in the MySQL metadata table RS_Activate_Repeater
  • Batch_Flag (Y/N) – This flag allows this rule to be called in batch directly using the script DBI_File_Copy.sh, or ultimately in the DBI_Activate process
  • Description – A general description of the processes purpose
  • Lower_Case_Flag (Y/N) – If this flag is ‘Y’, when a file is processed, the full file name will be changed to lowercase
  • Add_Datetime_Flag (Y/N) – If this flag is ‘Y’, when a file is processed, a datetimestamp will be added to the filename if it doestn’ already exist
  • Gzip_Gunzip (gzip/gunzip) – If this value is set to gzip or gunzip, the file will include the related processing if the file is not already processed accordingly
  • Zip_Unzip (zip/unzip) – If this value is set to zip or unzip, the file will include the related processing if the file is not already processed accordingly
  • Remove_Source_Flag (Y/N) – Once the file is pulled into the EC2 instance for processing, the source file will be removed if permissions allow
  • Encrypt_Decrypt (encrypt/decrypt) – If this value is set to encrypt or decrypt, the file will include the related processing
  • Source_Linux_Ftp_Aws_Drive (linux/ftp/aws/drive/sheet/http) – This defines the source system for the file
  • Dest_Linux_Ftp_Aws_Drive (aws/sftp/linux/drive) – This defines the dest system for the file
  • Email_Job_Log – If the job is successful, the email address in this field will receive an email
  • Source_Userid – If the source system requires a userid, it can be included here.  This is only typical for the SFTP connection
  • Source_Password – If the source system requies a password, it can be entered here.  This should only be used in a non-production environment.  In production, this password can be stored in a hidden .ini file
  • Source_Server – This maintains several variations of the destination server address.  They are defined as follows:
    • AWS – use the bucket name, ie. dbintegrator-system
    • SFTP – use the IP address or DNS
    • LINUX – use the IP address or DNS
    • DRIVE – use the Google Drive GUID
    • SHEET – use the Google Sheet GUID
    • HTTP – use the domain name
  • Source_Dir – This is the path on the associated server. ie data/dbi_file
  • Source_File – The source file name prefix. This is the name of the file to the first period, and then everything after this is irrelevant. Therefore, if you don’t include the first part in whole and it’s just a partial name, the action will not take place. In the event where you want every file from a folder to be copied, the option ‘#ALL#’ acts as the wildcard. If you specify a file explicitly while also using the wildcard option, the file is always processed using the explicit entry
  • Exclude_String – This allows for a string which will be compared to the file names in the source.  If there is a match, the file is not processed.  This is not applicable for DRIVE, SHEET and HTTP Source Servers.
  • Dest_Userid – If the destination system requires a userid, it can be included here.  This is only typical for the SFTP connection
  • Dest_Password – If the destination system requies a password, it can be entered here.  This should only be used in a non-production environment.  In production, this password can be stored in a hidden .ini file.
  • Dest_Server – This maintains several variations of the destination server address.  They are defined as follows:
    • AWS – use the bucket name, ie. dbintegrator-system
    • SFTP – use the IP address or DNS
    • LINUX – use the IP address or DNS
    • DRIVE – use the Google Drive GUID
  • Dest_Dir – This is the path on the associated server. ie data/dbi_file
  • Dest_File – This is an optional field that can be used to change the name of the file.  If this field is left blank, then the file name is not changed.  If you enter a new name, this will be the new name of the file at the destination location.  The last option allows you to include a prefix or a suffix to the file name.  Enter the reserved keyword #FILE# with the prefix before this keyword, or the suffix after this keyword.   
  • Grouping_Name – 
  • Pre_Exit_Call – This is an exit call that runs at the beginning of the process. If this field is populated with the path and script name, it will be executed for every file being processed
  • Post_Exit_Call – This is an exit call that runs at the end of the process. If this field is populated with the path and script name, it will be executed for every file being processed
  • Archive_Flag (Y/N) – If the source location allow the file being processed to be archived, this option move the file to a subdirectory and append a timestamp to the filename.

 

Advertisements