MySQL Error: Error_code: 1032

Here's something about MySQL replication error. The exact error could be as below:

Could not execute Update_rows event on table DBName.tableName; Can't find record in 'tableName', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.01, end_log_pos 218

If SQL Thread on one of your slave stops with this error code, that just mean Master and this Slave are not in sync!! There must be an inconsistency. 

In our case, slave got an Update event form master, but that particular record was not available in the slave. (Of course, that was deleted on Master n Slave separately)
You would get this error only if the binlog_format is set to ROW_BASED or MIXED mode.

So, now check that particular binlog at that position where replication stopped with this error. 
When you convert the binlog with mysqlbinlog command, you may see some junk characters  where you were expecting some DMLs which caused the error(In our case its an Update statement).

So you can use below command to translate binlog completely:


mysqlbinlog --base64-output=DECODE-ROWS --verbose  mysql-bin.01 >mysql-bin.01.txt

mysql: unknown variable 'bind=localhost'

If you are getting this error open /etc/mysql/my.conf and comment the line

bind =127.0.0.1

or

bind =localhost


what ever in your case and you will be able to get mysql prompt.

Changing Replication factor in hadoop on fly

Changing Replication factor in Hadoop on fly, the may be many time when you need to define replication factor of a specific file or directory on the fly, so following command will help you to change it.

The command used for this is "setrep"

How will you do it?

Solution :

Custom Checkpoint, Performing check pointing in hadoop hadoop, Edit log size hadoop, edit log size more hoadoop

Some time it may happen that checkpoint will not happen, and edit size becomes more, of if check point is not happening the is a chance of data loss, if something bad happens, for this

Verify Secondry namenode is up and running fine

• check for services ps -ef | grep secondary\
• check if log rotation of secondary name node is happening properly

 Solution :

Find and remove files older than 'n' days !

Sometimes we schedule a particular daily job in cron, which generates daily log files. Though the size is smaller, we can't keep those files. In that case we need a shell command to clean up and here it is:

To delete all the files whose data modified before 'n' days:

find -mtime +n -delete    


To delete all the files whose data modified in last 'n' days:

find -mtime -n -delete    


To delete files of a particular patterns, say 'archive_date.csv'

find -name 'archive_*.csv' -mtime +n -delete    


To list out all files older than 'n' days along with date and time, just like 'ls' command:

find -mtime +n -ls   


These are simple commands, but very useful. 
Modify as per your need.
Thanks!

Find and remove empty direcory

This command will find the empty directory and delete it recursively

find . -type d -empty -delete   --> Recursively

find . -empty -type d -exec rmdir {} +     --> From current Dir

find . -depth -type d -empty -exec rmdir -v {} +




Find all files of type <*.txt or what ever you like>

find . -name \*.txt -print    --> In place of *.txt you can specify extention                             
                                                       you like to search

java.net.SocketException Too many open files

Have you ever encountered this error? is so here is the solution.... :)

This may happen if there are many http request, or forgetting to close the open connection in time so for this what you can do is you can increase the number of open file in linux machine, as it is related to os issue mostly.

Open /etc/sysctl.conf and add the following :

fs.file-max = <65535>   <-- keep this number as you need

And for changing the effect of this use following command

sudo sysctl -p /etc/sysctl.conf

Find file of specific size Linux

Some time you many need to find all files greater than equal to specific size like 1MB 10 MB or 1GB, so you can use following command to get the files have length more than equal to you have specified:

find / -type f -size <GIVE SIZE HERE IN KB> -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'

Example:

find / -type f -size +1048576k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'


This will find files greater than size 1GB. so you can experiment with the size as you need.

Whitelist a node in hadoop

You have a cluster with black listed nodes, this is how you can make dem white listed nodes

<property>
  <name>dfs.hosts</name>
  <value>path to whitelisted node file</value>
</property>

then issue following command

./bin/hadoop dfsadmin -refreshNodes

Hadoop Eco System

HDFS: The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large data sets.

Search for a string in all files in a directory Linux

Grep command you can use to search a specified string in all files in the directory path you have given

grep "string to search" /var/*

so this will search string to search in /var/ directory files.

another variation :

find . -type f -exec grep -i "string to find" {} \; -print

or you can use

grep "string to search" *.htm    --> this will search for the string to search in all htm files


There can be more options as always :) try and find more.