If you’ve used Perl at all you are probably familiar with the simple oneliner to do a search and replace on a given string:
perl -p -i -e 's/oldstring/newstring/g' *
This will replace all occurrences of oldstring with newstring in all of the files in your current directory. Being able to do this quickly and easily is absolutely awesome, but what if you want to do this recursively across the current directory and all directories below that? I’m sure that there are plenty of ways to do this, the one below is the method that seems the easiest:
perl -p -i -e 's/oldstring/newstring/g' `find ./ -name *.html`
If you wanted to be more precise you could replace find with grep and only perform the search and replace on files that you know already contain oldstring, like this:
perl -p -i -e 's/oldstring/newstring/g' `grep -ril oldstring *`
This is one of those things that I don’t have to do very often, but when I do, this Perl oneliner is a real life saver.
May 16th, 2006 at 11:06 am klaus
wouldn’t it be better to pipe the “find” output to xargs and pass this to the perl expression?
May 16th, 2006 at 5:08 pm Nic
Couple problems. First, you have to quote ‘*.html’ or the shell will expand it and find will barf. Also, if there are spaces in any of the file names that find returns, perl will barf. So, as Klaus suggests, you should use
find . -name '*.html' -print0 | xargs -0 perl -pi -e 's/oldstring/newstring/g'May 16th, 2006 at 8:41 pm Josh
Klaus is right because the find clause might produce enough output that it will reach the limit for line length (input too long). Remarkably, this happens frequently enough. I remember having to debug somebody’s perl script that was written to search through 750gb of text files. One day, the number just managed to hit the max and it errored out with “input too long” messages.
find . | xargs perl -p -i -e ’s/something/else/g’
However, for smaller directories, I really like your second solution, grepping good matches out first. You should run some time trials to see if it’s a speed win to have grep do the detection and then only perl the ones you need. I bet there’s a break-even where less than some percentage of relevant files will give you a speed advantage.
-Josh-
February 15th, 2007 at 12:34 pm Michael’s Tech Blog » Blog Archive » Perl Oneliner: Recursive Search and Replace
[...] For example, my first Perl oneliner formats Apache log to be easy readable. Here is another good example of Perl onliner (it isn’t my actually). It makes a recursive search and replacement: perl -p [...]
February 21st, 2007 at 8:36 am GaryB
Just one more way to do:
find ./ -exec perl -p -i -e ’s/faltoo_string/aur_bhi_faltoo_string/g’ {} \;
March 20th, 2007 at 11:31 pm PrestonLee.com
Shell Magic: Globally Find/Replace Text Using RegExps Within Many Files…
Here’s one of my favorite Perl-based one-liners which I whip out when I need to find a bunch of files and replace text based on regular expressions at the command line.
find . -iname ‘*.js’ -exec perl -pi.bak -e ’s/2006/2007/g’…
June 18th, 2007 at 10:37 am Adam ruck
Hey guys, be careful with your symlinks! I used Nics method from post number 2, and it converted my symlink files to real files. That ended up breaking some stuff for me.
August 1st, 2007 at 2:35 am Flavien
Using grep to select the files is a performance killer : It means that the files matching the regexp will be scanned twice : Once by grep and once by perl. If the file is big, it’s a problem.
Not selecting the files that will be changed is a problem: Files that do not match will change (ie time, inode). Not good for keeping timestamps of unchanged files (think for example source files: make will rebuild the objects!)
/me is wondering if perl has an option “do not replace the file by a new one if it did not change, just keep it”.
February 22nd, 2008 at 10:39 am Carlos
I tried all these examples to no avail using Solaris9, just trying to find all files in filesystem with *.zip and change it to *.ZIP.
# perl -v
This is perl, v5.6.1 built for sun4-solaris-64int
(with 48 registered patches, see perl -V for more detail)
# find /usr/local/tmp/aaa -name “*.zip” -exec perl -p -i -e ’s/zip/ZIP/g’ {} \;
# ls -l /usr/local/tmp/aaa
total 6
-rw-r–r– 1 carlos staff 49 Feb 22 09:35 9_Recommended.zip
-rw-r–r– 1 carlos staff 49 Feb 22 09:35 DLG_V11_2.zip
-rw-r–r– 1 carlos staff 49 Feb 22 09:35 tzupdater-1_0_1.zip
#
Single quotes doesn’t work too
# find /usr/local/tmp/aaa -name ‘*.zip’ -exec perl -p -i -e ’s/zip/ZIP/g’ {} \;
# ls -l /usr/local/tmp/aaa
total 6
-rw-r–r– 1 carlos staff 49 Feb 22 09:37 9_Recommended.zip
-rw-r–r– 1 carlos staff 49 Feb 22 09:37 DLG_V11_2.zip
-rw-r–r– 1 carlos staff 49 Feb 22 09:37 tzupdater-1_0_1.zip
May 1st, 2008 at 10:28 am Nick
Carlos, perl -pi -e replaces the CONTENTS of the file. You are trying to RENAME the file. Two very different things. See if you have the rename command; it is a perl script and will let you rename a file using regex substitution.
May 12th, 2008 at 8:36 pm Mike Golvach
Hey There,
For “not replacing” files that already exist, I use this with that perl one-liner when I’m piping filenames to it - it’s not built into the options, but is pretty simple to type once :)
perl -p -i -e ’s/ThisThing/ThatThing/; rename $_, unless -e $_;’
It doesn’t work “out of context” because $_ requires a value, but you can use the rename half of my version of the one-liner to insert in the previous code to prevent against overwriting :)
Great post :)
, Mike
June 20th, 2008 at 3:58 pm Michael
hello all
I execute this:
# perl -p -w -e ’s/(?:\n)(.*)//g’ test1.txt
And get this:
Bareword found where operator expected at -e line 1, near “s/(?:\n)(.*)</collectioninfo”
Unquoted string “collectioninfo” may clash with future reserved word at -e line 1.
syntax error at -e line 1, near “s/(?:\n)(.*)</collectioninfo”
Execution of -e aborted due to compilation errors.
Any help ?
Thx !!
June 24th, 2008 at 2:48 am Mike Golvach
Hey There,
That error almost always has to do with unmatched quotation marks. Probably, the easiest way to figure out what’s wrong with it would be to turn your one-liner into a simple script. You’ll get waaaaaay too much information, but the answer will probably be in there.
If not, check test1.txt for any unmatched double quotes, single quotes, backticks, etc and that may lead you to the answer, as well.
Best wishes,
Mike
June 24th, 2008 at 2:50 am Mike Golvach
I should note, that, with the script, you’ll at least get a line number that probably won’t be 1 ;)