Dedup

DeDup is a Java command-line utilty for removing duplicate

lines in text files.

The dedup process compares adjacent lines only. It does not

sort first. The comparison is case sensitive. It removes

adjacent indentical lines. It replaces the original file, so

do a backup first. You can dedup 1 to N files in one

execution, just put their names on the command line.

DeDup deletes blank lines from both the beginning and end of

the file. It also deletes trailing blanks from each line. It

also cleans up line terminators \r \n \r\n and converts them

all to platform standard. If the file is already deduped,

it does not change the date.

Use it with:

java com.mindprod.dedup.DeDup myfile.txt anotherfile.txt yetanotherfile.txt

or

java -jar dedup.jar myfile.txt anotherfile.txt yetanotherfile.txt

or, if you have the jar association set up, even even

dedup.jar myfile.txt anotherfile.txt yetanotherfile.txt

It can handle a list of files, or an os-expanded wildcard.

It does not -s to process subdirectories or the use of

directory names to process entire directories.