Linux : Prefix a string to all rows in a file - sed

Consider, you have a file like below:

# cat test.txt
12324
234523
3431
343
34345
3434
43234

Now, if you want to prefix a particular string to each line in the file, how will you do it? This is a common challenge for DBA/Developers working with Linux environment.

Linux: Comparing 2 files - 'comm'

Many a times, i encountered the situation wherein I would be having 2 files with some values in each. And I need to get common records in both the files and also, unique records for each files.

So, this is how  I do - one of the easiest way.

Below are the files to be compared:

MySQL Replication: Exclude an SQL from replicating

If you're using MySQL Replication, in rare cases you may require to skip a particular SQL statement from replicating - which is to be executed on Master and should not be on the Slave. Though, its not a good practice.

However, here's how to do it. 

MySQL has a variable sql_log_bin, by default its value will be 'ON' or '1', when we enable replication. That means, it replicates all the SQL statements from Master to Slave.
So, if we want to disable this for a particular SQL, do as below:

SET SESSION sql_log_bin=0; sql statement;SET SESSION sql_log_bin=1;


Example:

 
SET SESSION sql_log_bin=0; DELETE FROM tab1 WHERE col1='NULL'; 
SET SESSION sql_log_bin=1;

 However, after MySQL 5.5.5 version, no need to use SESSION. By default its a SESSION variable and it won't affect other user's transactions. 

Note: Executing this on Master would cause data inconsistency. Don't try this unless you are sure about what you are doing!

Top Command Options for Admnistrators

There are lot of option with top command but some parameters which i found very useful are following :


1. Type top on terminal you will get interface something like :




So mostly What do we need to see??

Memory wise sorted processes
CPU utilization wise sorted process and 
The process name which is using the memory and the CPU Right?

Linux: Add numbers in a column in a file

In windows we use Excel to add all the numbers in a column. But how about Unix/Linux?
We encountered such an requirement and we got the answer too.
Actually solution in much more simpler than the problem..!

Mission accomplished by 'awk' command. Its merely a very powerful and useful command.
Here is an example for it:

Convert image to byte array and byte array to image file

This code will convert an image file to byte array and will write it into a binary file, and will read the binary file and converti it to image file.

import java.awt.image.BufferedImage;
import java.io.BufferedOutputStream;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.BufferedInputStream;
import javax.imageio.ImageIO;


public class conveter {

 /**
  * @param args
  */
 public static void main(String[] args) {
  try {
    
   byte[] imageInByte;
   BufferedImage originalImage = ImageIO.read(new File(
     "Chrysanthemum.jpg"));
 

MySQL Error: Error_code: 1032

Here's something about MySQL replication error. The exact error could be as below:

Could not execute Update_rows event on table DBName.tableName; Can't find record in 'tableName', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.01, end_log_pos 218

If SQL Thread on one of your slave stops with this error code, that just mean Master and this Slave are not in sync!! There must be an inconsistency. 

In our case, slave got an Update event form master, but that particular record was not available in the slave. (Of course, that was deleted on Master n Slave separately)
You would get this error only if the binlog_format is set to ROW_BASED or MIXED mode.

So, now check that particular binlog at that position where replication stopped with this error. 
When you convert the binlog with mysqlbinlog command, you may see some junk characters  where you were expecting some DMLs which caused the error(In our case its an Update statement).

So you can use below command to translate binlog completely:


mysqlbinlog --base64-output=DECODE-ROWS --verbose  mysql-bin.01 >mysql-bin.01.txt

mysql: unknown variable 'bind=localhost'

If you are getting this error open /etc/mysql/my.conf and comment the line

bind =127.0.0.1

or

bind =localhost


what ever in your case and you will be able to get mysql prompt.

Changing Replication factor in hadoop on fly

Changing Replication factor in Hadoop on fly, the may be many time when you need to define replication factor of a specific file or directory on the fly, so following command will help you to change it.

The command used for this is "setrep"

How will you do it?

Solution :

Custom Checkpoint, Performing check pointing in hadoop hadoop, Edit log size hadoop, edit log size more hoadoop

Some time it may happen that checkpoint will not happen, and edit size becomes more, of if check point is not happening the is a chance of data loss, if something bad happens, for this

Verify Secondry namenode is up and running fine

• check for services ps -ef | grep secondary\
• check if log rotation of secondary name node is happening properly

 Solution :

Find and remove files older than 'n' days !

Sometimes we schedule a particular daily job in cron, which generates daily log files. Though the size is smaller, we can't keep those files. In that case we need a shell command to clean up and here it is:

To delete all the files whose data modified before 'n' days:

find -mtime +n -delete    


To delete all the files whose data modified in last 'n' days:

find -mtime -n -delete    


To delete files of a particular patterns, say 'archive_date.csv'

find -name 'archive_*.csv' -mtime +n -delete    


To list out all files older than 'n' days along with date and time, just like 'ls' command:

find -mtime +n -ls   


These are simple commands, but very useful. 
Modify as per your need.
Thanks!