OSNews: http://www.osnews.com/story/21004/Awk_and_Sed_One-Liners_Explained Exploring the Future of Computing en-us Copyright 2001-2013, David Adams adam+nospam@osnews.com Wed, 19 Jun 2013 00:47:52 GMT http://www.osnews.com/images/osnews.gif OSNews.com http://www.osnews.com awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349794 http://www.osnews.com/thread?349794 My dad is always singing the praises of Sed and Awk but that's not the only reason I don't like them ;)

They encourage these one-liners, which just means removing the formatting from code. They use regular expressions, something I consider should be avoided at all costs on account of their impenetrable syntax. My general feeling is that in the time it takes to figure out how to do anything with these tools, you could have just written a Python script to do it.

I'm sure they were great tools in their day but I really think Python trumps them, replacing all their functionality and throwing in maintainability to boot.

All that said, these are excellent, useful and well written articles. If I ever find I'm forced to use these things, I shall be eternally greatful that something like this exists. Thu, 19 Feb 2009 12:42:00 GMT donotreply@osnews.com (Michael) Comments RE: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349799 http://www.osnews.com/thread?349799 Depends on the load you have to lift. If I wanted to perform a quick bucket-sort, then I’d use PHP, VBScript or any other non-compact scripting language. But if I had to make a syntax processor (like I have: http://camendesign.com/code/remarkable ), then there’s no way I’d do it without regex. I’d have to practically reimplement a hard-coded regex engine in the process of handling the byte-by-byte matching for all the use-cases.

Sure the likes of sed and awk are hard to use, but there’s wizards out there who can, and those of us who can’t -- it’s not ours to say that one tool is better than the other, when in the right hands.

Reminds me of this T-Shirt - http://www.thinkgeek.com/tshirts-apparel/unisex/frustrations/374d/?... - "Go away or I will replace you with a very small shell script."Edited 2009-02-19 13:08 UTC Thu, 19 Feb 2009 13:06:00 GMT donotreply@osnews.com (Kroc) Comments PDF http://www.osnews.com/thread?349802 http://www.osnews.com/thread?349802 Just liked. It would be nice to have a PDF version with all concatenated.

Thank you. Thu, 19 Feb 2009 13:19:00 GMT donotreply@osnews.com (dbolgheroni) Comments RE: PDF http://www.osnews.com/thread?349805 http://www.osnews.com/thread?349805 Yep. I will publish two free ebooks, one on Awk one-liners and one on Sed one-liners.

Ps. you can subscribe to my posts on my blog so that you don't miss them. Thu, 19 Feb 2009 13:25:00 GMT donotreply@osnews.com (pkrumins) Comments RE: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349810 http://www.osnews.com/thread?349810 First off, Python is not available anywhere, but awk and sed are (almost). Maybe perl is better than python, but anyway, that is not the point.
awk and sed scripts are not there to be maintained. They are not developer tools. They are administrator tools and they are there to make it easy for you to edit or search files quickly. sed and awk are used for one-shot commands in 95% of the cases. Once you have the result, there is no need to maintain the command at all.
How many lines do you have to write in python, just to open a file and read it? In sed or awk, that's 0. The file is open and parsed. There is no way you can make it faster in python.Edited 2009-02-19 14:22 UTC Thu, 19 Feb 2009 14:18:00 GMT donotreply@osnews.com (spiderman) Comments RE[2]: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349811 http://www.osnews.com/thread?349811 I've never considered awk hard to use. You can learn the basics in about thirty pages. Have you ever sat down and read "Effective GAWK Programming"?

http://www.gnu.org/software/gawk/manual/gawk.html

sed is not hard either. Python is great too, but for little quick jobs awk is great. awk can be just as maintainable if you write it cleanly (and I've seen python code that is a mess, because you can write messy Python too.) But for a one liner, who cares if it is maintainable? You use it once then throw it out! Thu, 19 Feb 2009 14:18:00 GMT donotreply@osnews.com (massysett) Comments Comment by timefortea http://www.osnews.com/thread?349815 http://www.osnews.com/thread?349815 Excellent. Thu, 19 Feb 2009 14:28:00 GMT donotreply@osnews.com (timefortea) Comments RE: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349818 http://www.osnews.com/thread?349818 Chinese (traditional/simplified) looks pretty incomprehensible to me but evidently a 1,000,000,000+ people manage to get by.

Without a doubt it's unwelcoming to those who don't understand it, but for those who do... Thu, 19 Feb 2009 14:58:00 GMT donotreply@osnews.com (JMcCarthy) Comments RE[2]: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349820 http://www.osnews.com/thread?349820 That's why I wrote the articles, to explain them.

It is unfortunate that only the first paragraph shows on hacker news website.

Click on Read More to find links to my articles! Thu, 19 Feb 2009 15:08:00 GMT donotreply@osnews.com (pkrumins) Comments My Favorite http://www.osnews.com/thread?349837 http://www.osnews.com/thread?349837 fretinator ~ $ sed "W+F rU $@y!n6" Thu, 19 Feb 2009 16:54:00 GMT donotreply@osnews.com (fretinator) Comments RE: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349838 http://www.osnews.com/thread?349838 Are you serious?!

Regular expressions are awesome, and definately a core technology, even when using Python.

You are either a troll or completely clueless. soz. Thu, 19 Feb 2009 17:02:00 GMT donotreply@osnews.com (appel) Comments pdf version? http://www.osnews.com/thread?349839 http://www.osnews.com/thread?349839 Very nice! But I'm going to wait for the PDF file that'll include all the tutorials (not that I don't like your blog or anything, hehe). Thu, 19 Feb 2009 17:02:00 GMT donotreply@osnews.com (maaxx) Comments RE: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349850 http://www.osnews.com/thread?349850 I wouldn't really say that AWK and SED encourage these one liners, rather I would say they enable one liners.

When you work with these tools to the comfort level that you can just spit out these one liners, that's where this facility becomes a powerful command line tool rather than a just a scripting language.

I don't think twice about just pounding out long pipelines in the shell, or short scripts. Similarly, if it can fit on one line, I'll consider doing the same with AWK or SED.

These are one off events that never last beyond perhaps the shell history. Thu, 19 Feb 2009 17:49:00 GMT donotreply@osnews.com (whartung) Comments great http://www.osnews.com/thread?349854 http://www.osnews.com/thread?349854 Great contribution, thank you! Thu, 19 Feb 2009 18:08:00 GMT donotreply@osnews.com (jwwf) Comments Awk and Sed One-Liners Explained? http://www.osnews.com/thread?349855 http://www.osnews.com/thread?349855 If you have to explain one-liners, then they aren't funny. Thu, 19 Feb 2009 18:14:00 GMT donotreply@osnews.com (tupp) Comments RE[2]: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349856 http://www.osnews.com/thread?349856 http://xkcd.com/208/

and of course:

http://xkcd.com/353/

and for what it's worth; Python > Perl

:) Thu, 19 Feb 2009 18:17:00 GMT donotreply@osnews.com (poundsmack) Comments RE[2]: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349861 http://www.osnews.com/thread?349861
How many lines do you have to write in python, just to open a file and read it?


One. cont = open("file.txt").read().

Or

for line open("file.txt"): do_stuff_with(line)

If that means significant extra work for you, you need a harder problem domain ;-) Thu, 19 Feb 2009 19:06:00 GMT donotreply@osnews.com (vivainio) Comments RE: great http://www.osnews.com/thread?349880 http://www.osnews.com/thread?349880 Just finished printing the two text files. Allthough sed and awk do already belong to my usual daily tools, I like to learn something new every day. Many thanks! Thu, 19 Feb 2009 20:23:00 GMT donotreply@osnews.com (Doc Pain) Comments RE: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349887 http://www.osnews.com/thread?349887
They use regular expressions, something I consider should be avoided at all costs on account of their impenetrable syntax.


You obviously subscribe to the school of thought that says someone who uses regular expressions to solve a problem now has two problems. ;)

True to a degree - they tend to be overused, or made overly complicated by people who think that because they can read it, so will the next person who comes along. But they're also an extremely powerful tool, for which there really isn't any practical alternative - e.g if you want to validate that a string matches a pattern, you can use a simple regex to do it, or you can write your own parser. And writing your own parser is almost never the right answer. Thu, 19 Feb 2009 20:55:00 GMT donotreply@osnews.com (Delgarde) Comments RE[3]: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349919 http://www.osnews.com/thread?349919 This manual has 360 pages (pdf-version). I would like to learn the basics of awk and has already bought a German cheap reference by O'Reilly.
Do you know a shorter tutorial? Thu, 19 Feb 2009 23:11:00 GMT donotreply@osnews.com (AtariFan) Comments RE[4]: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349929 http://www.osnews.com/thread?349929
Do you know a shorter tutorial?


Yep: http://www.grymoire.com/Unix/Awk.html Thu, 19 Feb 2009 23:39:00 GMT donotreply@osnews.com (pkrumins) Comments RE[2]: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349936 http://www.osnews.com/thread?349936 Regular expressions are very powerful but they cover a limited range of problems between the the trivial (the title is the example given to trim whitespace) and the very complex.

These old UNIX tools were terific in their day when they were the only way of doing things. But I think the fact that they continue to get so much air time has more to do with the fact that it's fun play with them. They're like crossword clues.

Don't underestimate what you can do with Python. It's trivial to read in a text file and split it into an array using whatever seperator you want. Modern scripting languages have tremendously powerful string handling functions and they do all this using real words. RE's are there if you need them but you very rarely do.

My work rate isn't limited by the speed at which I type, it's limited by the speed at which I think (that is, severely limited). I think better if I'm not having to translate everything via these arcane hieroglyphics.

I wish people were better at distinguishing between a genuinely held (and valid) opinion and a troll. It is only my opinion. Fri, 20 Feb 2009 00:14:00 GMT donotreply@osnews.com (Michael) Comments RE[2]: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349939 http://www.osnews.com/thread?349939 no, there is not a billion people who can read Chinese, unlike popular believe, and there is a reason for that. Guess why they had to 'invent' simplified on top of traditional? The same applies to all other fancy writing systems.

However, the point is that different tools serve different purposes and you may get along with it without knowing it inside out. I wished I knew any which one of these to some extent, but absent an actual need, I cannot justify putting the required time into it. You will know better... Fri, 20 Feb 2009 00:25:00 GMT donotreply@osnews.com (Googol) Comments Benefits of Awk and Sed http://www.osnews.com/thread?349947 http://www.osnews.com/thread?349947 It's been said before but you don't always have Python or Perl available and knowing Awk and Sed can be a life saver. They are also great tools for formatting output for input into other programs. An entire scripting language is overkill for that kind of work. Fri, 20 Feb 2009 02:25:00 GMT donotreply@osnews.com (abraxas) Comments RE[3]: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?349949 http://www.osnews.com/thread?349949
These old UNIX tools were terific in their day when they were the only way of doing things. But I think the fact that they continue to get so much air time has more to do with the fact that it's fun play with them. They're like crossword clues.


I think it's because most people still think that bringing Python into the equation for simple formatting and extracting information from output is overkill. If I'm just writing a simple filter to be used between the output of one program and the input of another program on my own machine then I'm just going to use the shell, sed, and awk. If I want to distribute some kind of application that performs all the functions of my script I'm probably going to want to develop it in a language like Python or Perl. Fri, 20 Feb 2009 02:48:00 GMT donotreply@osnews.com (abraxas) Comments RE[3]: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?350024 http://www.osnews.com/thread?350024
Modern scripting languages have tremendously powerful string handling functions and they do all this using real words. RE's are there if you need them but you very rarely do.

Depends on how you think (hammer & nails, anyone?). I always think of a regexp solution to any given problem (that I solve in Python) first - typically, I slurp in a string, run re.findall on it, then do a for loop over the resulting tuples.

Regular expressions have the advantage of being insanely fast, and very easy to work with. I agree that for the problems that are trivial enough to solve with awk/sed, regexs may be overkill - you can just do s.replace(), s.split() and s.join(). Fri, 20 Feb 2009 17:56:00 GMT donotreply@osnews.com (vivainio) Comments RE[3]: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?350033 http://www.osnews.com/thread?350033

for line open("file.txt"): do_stuff_with(line)

That should be:
for line in open("file.txt"): do_stuff_with(line)

If that means significant extra work for you, you need a harder problem domain

Agreed. Fri, 20 Feb 2009 18:50:00 GMT donotreply@osnews.com (RandomGuy) Comments RE[4]: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?350066 http://www.osnews.com/thread?350066

for line open("file.txt"): do_stuff_with(line)

That should be:

for line in open("file.txt"): do_stuff_with(line)


It's neither. It should be:


with file("file.txt") as fd:
for line in fd:
do_stuff_with(line)
Edited 2009-02-20 23:27 UTC Fri, 20 Feb 2009 23:26:00 GMT donotreply@osnews.com (pkrumins) Comments RE[5]: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?350087 http://www.osnews.com/thread?350087
It's neither. It should be: with file("file.txt") as fd:

Such pedantic diligence ('with' statement) is probably not appropriate in a thread that mentions awk/sed. Using with statement is needed when you are worrying about closing file handles in some-future-version-of-python that may not do reference counting (which could imply that the file handle would remain open until the next garbage collection cycle).

Not really something you need to worry about if you are mainly targeting your normal python installation. Furthermore, a linear script can just close() the filehandle without caring about exceptions (because it would just exit the process and close everything anyway). Sat, 21 Feb 2009 08:52:00 GMT donotreply@osnews.com (vivainio) Comments Thank you for this http://www.osnews.com/thread?350120 http://www.osnews.com/thread?350120 Many thanks for this - particularly for the cheat sheet which is very convenient, but the one liners are excellent and well commented too. We can always learn.

The nice thing about awk is, its lightweight, reasonably fast, terse, intuitive once you know and are used to it. What its good for, its great for. I think of it a bit like an old pruning knife with a well sharpened slightly worn blade and a handle polished with use, an Opinel for instance. Its safe, cheap, its sharp, it just fits in the hand, and you hardly think about it any more. And would be very sorry indeed if you ever lost it. Sat, 21 Feb 2009 17:47:00 GMT donotreply@osnews.com (alcibiades) Comments RE: PDF http://www.osnews.com/thread?350122 http://www.osnews.com/thread?350122 If you use GNOME just File->Print->Print to file. Sat, 21 Feb 2009 18:35:00 GMT donotreply@osnews.com (abraxas) Comments RE[3]: awk '{ gsub(/^[ \t]+|[ \t]+$/, ""); print }' http://www.osnews.com/thread?350219 http://www.osnews.com/thread?350219 If you want to convince me python is better than sed/awk for this kind of tasks you should write all the one-liners in python. Then I will listen to you. Sun, 22 Feb 2009 21:21:00 GMT donotreply@osnews.com (yiyus) Comments