Check duplicate record in Hive

Today, I will discuss about ” How to automate the process where in you can check entire row duplicate record in hive”. As I have mentioned in all Automation blogs, I will share the pseudo code.STEP1: In hive , use “desc table_name” , this command will give you column names along with datatype and data length. Store the output of this command in a file , say HIVE_TABLE_DDL.txt STEP2 : Read the file HIVE_TABLE_DDL.txt using “cat” command. cat HIVE_TABLE_DDL.txt | awk ‘{print $1}’ ORS=’,’ | sed ‘s/,$//’ * awk'{print $1}’…

shell script to check whether logs have errors or not

Today, I will discuss about “how to check logs files at regular interval and identify if any errors are there or not and if yes, then send email to the concerned team members. As mentioned in my previous blogs , I will be sharing pseudo code only. STEP1 : Create a parameterized file which has all the details of log file names. For example, consider filename is (logs.config) and below is the content of the fileUSECASE_NAME|SERVER_LOGNAME_PATTERNRECHARGE_USSD|rechargeUssdMode STEP2 : Create the shell script in such a way that when you trigger…

check whether process is running or not

Today, I will discuss about existence of a process in the linux server through shell script. I will share the pseudo code for the same in the form of steps.* Create a shell script which runs in infinite loop using while 1==1 * Inside this while loop, check the existence of a process on every 15th minute which can be achieved by dividing the minutes part of current timestamp by 15 using MOD function. If it returns 0 , go to next statements else come out of If statement. *…

BEELINE COMMAND LINE IN HIVE

Today, I will discuss about the beeline command line which we use to call the SQL Query through Linux. But what if same thing needs to be called through Shell Scripting. First of all, we need to call Sql Query through Beeline command line inside shell Scripting using below command. beeline -u “jdbc:hive2://localhost:10000/default;principal=hive/localhost“ -n “username” -p “password” –hivevar var1=$col1_hive –hivevar var2=$schema_name –hivevar var3=$table_name –hivevar var4=$col1_value -f sql_script.sql > text.log Here $col1_hive is the column name of a table. $table_name is the table name. $schema_name is the Schema Name where that…

Remove the extra delimiters in the file using shell script

Today, i will discuss about the “How to remove extra delimiters in a file” using shell script. Below is the content of the file. a|b||c d||e|||f Post execution of shell script, the output will be a|b|c d|e|f Below is the script which is created to remove extra delimiter in a file. #file is assigned to variable INPUT . Here we used two files text1.txt and text2.txt to manipulate the source file data. INPUT=”Path of the file”/file.txt #Truncating text1.txt. Make sure this file exists. >text1.txt #Reading character by character using while…

Shell script to run for x hours using while Loop

Today i will discuss about the shell script code snippet which will run for x hours. Below is the code snippet which will help you to run code for x hours. ## Here I am using 10 seconds , in order to run for 1 hours, mention the same in seconds which is 3600. $vartime value will be set to 3600 initially. vartime=$((SECONDS+10)) echo $vartime ## $SECONDS tells you the time period for which script is running.Initially it will be 0 seconds when script starts. So, 0 will be compared…

Shell script to check files changed in last 24 hours

Today , i will discuss about the how to get all files which are changed in last  24 hours in shell script. Below  is the code for the same. ## STEP-1 first of all get the yesterday date using below command. DATE=`date -d “yesterday” ‘+%Y%m%d%H%M’` echo $DATE ## STEP-2 Below command will create file date.txt as per yesterday timings. touch -t $DATE /opt/A1/date.txt ## STEP-3 Below are the paths where we need to check the changed files.Here I have considered four directories. FILES_PATH_A=”/opt/A” FILES_PATH_B=”/opt/B” FILES_PATH_C=”/opt/C” FILES_PATH_D=”/opt/D” ## STEP-4 Using find…

Shell Script to Merge multiple files into one file and Archive the source files.

Today i will discuss about the shell script which will merge the multiple files into one file  and post merging , archive the source files. Below is the code for the same. Here, i am passing four arguments  namely source file location,Archive file location,merged file name pattern,source file name pattern. Using date_1 variable which converts  the date into format YYYYMMDD_HHMMSS. Using “cd” command to go source file location. Count number of files in that location. If count is equal to 1 , that means there is no need to merge.…