How to delete duplicate files in Linux

There are number of times you require to delete the files with the same names and same contents, normally you can’t create the file with the same name in a single directory. In this case you can use the a great tool called fdupes

How to install fdupes?

To install fdupes on Ubuntu:

[email protected]~]# apt-get install fdupes

To install fdupes on CentOS :

[email protected]~]# yum install fdupes

How to use fdupes?

In order to test, I create a folder test in /usr/local/src

[[email protected] /usr/local/src]# mkdir test

Using the command below, I created 10 duplicate files, which contains text Justgeek.net is a great website for Linux Users

[[email protected] /usr/local/src/test]# for i in {1..10}; do echo "Justgeek.net is a great website for Linux Users" > justgeek${i}.txt ; done

You can run ls command to see files

[[email protected] /usr/local/src/test]# ll
total 40
-rw-r--r-- 1 root root 48 May 26 10:48 justgeek10.txt
-rw-r--r-- 1 root root 48 May 26 10:48 justgeek1.txt
-rw-r--r-- 1 root root 48 May 26 10:48 justgeek2.txt
-rw-r--r-- 1 root root 48 May 26 10:48 justgeek3.txt
-rw-r--r-- 1 root root 48 May 26 10:48 justgeek4.txt
-rw-r--r-- 1 root root 48 May 26 10:48 justgeek5.txt
-rw-r--r-- 1 root root 48 May 26 10:48 justgeek6.txt
-rw-r--r-- 1 root root 48 May 26 10:48 justgeek7.txt
-rw-r--r-- 1 root root 48 May 26 10:48 justgeek8.txt
-rw-r--r-- 1 root root 48 May 26 10:48 justgeek9.txt

The above script create 10 files with the names justgeek1.txt, justgeek2.txt and so on with the same data.

To search for duplicate files:

[[email protected] /usr/local/src/test]# fdupes /usr/local/src/test
/usr/local/src/test/justgeek2.txt
/usr/local/src/test/justgeek1.txt
/usr/local/src/test/justgeek9.txt
/usr/local/src/test/justgeek5.txt
/usr/local/src/test/justgeek8.txt
/usr/local/src/test/justgeek4.txt
/usr/local/src/test/justgeek10.txt
/usr/local/src/test/justgeek7.txt
/usr/local/src/test/justgeek3.txt
/usr/local/src/test/justgeek6.txt

In order to search for the files recursively in folder you can use -r option

[[email protected] /usr/local/src]# fdupes -r /usr/local/src/test/
Progress [3012/6700] 44%

See the size of duplicates found within a folder using the -S option.

[[email protected] /usr/local/src/test]# fdupes -S /usr/local/src/test
48 bytes each:
/usr/local/src/test/justgeek2.txt
/usr/local/src/test/justgeek1.txt
/usr/local/src/test/justgeek9.txt
/usr/local/src/test/justgeek5.txt
/usr/local/src/test/justgeek8.txt
/usr/local/src/test/justgeek4.txt
/usr/local/src/test/justgeek10.txt
/usr/local/src/test/justgeek7.txt
/usr/local/src/test/justgeek3.txt
/usr/local/src/test/justgeek6.txt

To delete the duplicate files while preserving a copy you can use the option ‘-d’. Extra care should be taken while using this option else you might end up loosing necessary files/data and mind it the process is unrecoverable.

[[email protected] /usr/local/src/test]# fdupes -d /usr/local/src/test
[1] /usr/local/src/test/justgeek2.txt
[2] /usr/local/src/test/justgeek1.txt
[3] /usr/local/src/test/justgeek9.txt
[4] /usr/local/src/test/justgeek5.txt
[5] /usr/local/src/test/justgeek8.txt
[6] /usr/local/src/test/justgeek4.txt
[7] /usr/local/src/test/justgeek10.txt
[8] /usr/local/src/test/justgeek7.txt
[9] /usr/local/src/test/justgeek3.txt
[10] /usr/local/src/test/justgeek6.txt

Set 1 of 1, preserve files [1 - 10, all]:

It will ask you which files to preserve and which one to delete. You can enter the file number which you want to preserve.

It will then preserve 5th file.

[[email protected] /usr/local/src/test]# ll
total 4
-rw-r--r-- 1 root root 48 May 26 11:08 justgeek8.txt
[[email protected] /usr/local/src/test]#

How to create OpenVZ virtual machine?

How do I create OpenVZ virtual machine (VPS) to run CentOS or Debian as VPS?

OpenVZ comes with a template for each VPS or virtual machine. OpenVZ provides templates for all leading Linux distributions. You need to download those templates in order to create a VPS. Visit this page to grab templates for vps.

Download Ubuntu Linux VPS

Type the following commands to download precreated Ubuntu Linux template:

[[email protected] ~]# cd /vz/template/cache
[[email protected] ~]# wget http://download.openvz.org/template/precreated/ubuntu-9.04-x86_64.tar.gz

Download CentOS 64 bit template:

[[email protected] ~]# cd /vz/template/cache
[[email protected] ~]# wget http://download.openvz.org/template/precreated/centos-5-x86_64.tar.gz

Create VPS

Now you’ve download the template for your virtual machine. You can just start a VPS based on the template you have just downloaded, by typing the following commands: [10 is VPS ID you can use as per your requirement]

[[email protected] ~]# vzctl create 10 –ostemplate ubuntu-9.04-x86_64
[[email protected] ~]# vzctl set 10 –onboot yes --save

Set IP for VPS

[[email protected] ~]# vzctl set 10 –ipadd 192.168.1.5 --save

Set Nameservers IP for VPS

[[email protected] ~]# vzctl set 10 –nameserver 192.168.1.111 --save
[[email protected] ~]# vzctl set 10 –nameserver 192.168.1.111 --save

Set Hostname IP for VPS

[[email protected] ~]# vzctl set 10 server.justgeek.net --save

Set Disk quota for VPS (10G min [soft] and 11G max hard limit)

[[email protected] ~]# vzctl set 10 –diskspace 10G:11G --save

Start the vps

[[email protected] ~]# vzctl start 10

Set root user password for VPS

[[email protected] ~]# vzctl exec 10 passwd

vzctl is used to create and set various vps properties such as memory, disk usage and much more. Where,

* create 10 : Your VPS ID.
* –ostemplate ubuntu-9.04-x86_64 : VPS template.
* –config vps.ubuntu: Save configuration.
* set 10 : Set various option for VPS ID # 10.
* –onboot yes : Make sure VPS boots automatically after a reboot.
* –save : Save changes to config file.

Script to restore SQL backup

In the post here we have already backed up all the database on the server, now using the script below you can restore MySQL databases.

Steps to run Script.

01. Create a file restore.sh

02. Put the code below & make sure you have changed the path marked in red in the script to actual location where your backups are stored.

03. Now change permission of script using chmod +x restore.sh

04. Now run the script using command sh restore.sh

#!/bin/bash
echo "SCRIPT TO RESTORE MULTIPLE SQL FILE"

for i in `ls /backup/mysql-copied`
 do
 dbname=$(echo $i | cut -d. -f1)
 echo "Database $dbname is being restored" 
 mysql $dbname < /backup/mysql-copied/$i
 echo "Database $dbname is restored"
done