Linked by Thom Holwerda on Mon 3rd Dec 2012 22:51 UTC
"Few tools are more indispensable to my work than Unix. Manipulating data into different formats, performing transformations, and conducting exploratory data analysis (EDA) is the lingua franca of data science.1 The coffers of Unix hold many simple tools, which by themselves are powerful, but when chained together facilitate complex data manipulations. Unix's use of functional composition eliminates much of the tedious boilerplate of I/0 and text parsing found in scripting languages. This design creates a simple and succinct interface for manipulating data and a foundation upon which custom tools can be built. Although languages like R and Python are invaluable for data analysis, I find Unix to be superior in many scenarios for quick and simple data cleaning, idea prototyping, and understanding data. This post is about how I use Unix for EDA."
Thread beginning with comment 544241
To view parent comment, click here.
To read all comments associated with this story, please click here.
To view parent comment, click here.
To read all comments associated with this story, please click here.
Member since:2011-01-28
Hypnos,
"It's better software design for tools to be simpler by default, not more complex by default."
I'm on the fence. Text is more open. There's no need to reverse engineer anything or be dependent upon binary parsers, which makes using standard command line tools very practical.
On the other hand, I almost wish all data were stored in a database where we can easily build complex indexes and queries to our hearts content, all while maintaining relational & transactional integrity as needed. So, if they replace the text files with a real database such as sqlite or mysql, I wouldn't object much to that.
SQL is so second nature to me, it's much easier than trying to build scripts using bash/sed/grep/etc for anything complex. I hate escaping command line arguments for bash scripts. Searching indexed tables is so much faster than searching text files too.
This is not a defence for syslogd's proprietary tools, but I can see lots of benefits in using structured databases over text files.
Member since:2005-08-18
On the other hand, I almost wish all data were stored in a database where we can easily build complex indexes and queries to our hearts content, all while maintaining relational & transactional integrity as needed.
Might want to look into Logstash. It's awesome.
Speaking of logstash, how does systemd's logging integrate with other tools? Can we easily transport the data to, say, logstash?
SQL is so second nature to me
SQL isn't that well-suited for logs though. Not much relational data to speak of in a log record.
( mysql? real database? surely you jest
) Edited 2012-12-05 07:30 UTC
Member since:2008-11-19
Features
Linked by Thom Holwerda on 05/21/13 21:38 UTC
At an event earlier today, Microsoft unveiled the next Xbox - the third model, but confusingly named Xbox One. The big focus was TV, integrated Kinect, and all the other stuff we all expected to be forced down our throats. I think it took them 25 minutes to actually come to what should be the core of the story: gaming. Nothing groundbreaking in the gaming department, except for how Microsoft intends to handle the used games market and borrowing games from friends: pay up, buddy!
Linked by Thom Holwerda on 05/20/13 11:29 UTC
The day has finally come! Jolla has finally announced the launch device for its Sailfish operating system - and by god this is a looker. It's decidedly different from other phones out there, but it has good specifications and carries a relatively reasonably price tag - EUR 399, and it's up for pre-order today, shipping in the fourth quarter of this year.
Linked by Thom Holwerda on 05/18/13 21:33 UTC
Why does Google get so much credit in the technology industry? Why, despite the company's many obvious failings, do many geeks and enthusiasts still hold a somewhat positive view on the all-knowing technology giant? A specific talk at Google I/O this week provides the answer.
Linked by David Adams on 05/16/13 4:23 UTC
OSNews is a sponsor of this year's O'Reilly OSCON in Portland, Oregon, USA. A lucky OSNews reader will win a free three-day pass, including two tutorials days. To win the pass, post a comment on this story saying something about Open Source Software or OSCON. We'll pick a winner at random next week. If you don't have an OSNews account, you may email us your entry. Part of the conference is the 9th annual Open Source Awards, and today the 16th is the deadline for nominations. If you'd like to nominate an outstanding open source contributor, do it here. Read on for more information about OSCON. Update: The 20% discount code for OSNews readers is "OSN."
Linked by Thom Holwerda on 05/11/13 21:41 UTC
"Windows is indeed slower than other operating systems in many scenarios, and the gap is worsening." That's one way to start an insider explanation of why Windows' performance isn't up to snuff. Written by someone who actually contributes code to the Windows NT kernel, the comment on Hacker News, later deleted but reposted with permission on Marc Bevand's blog, paints a very dreary picture of the state of Windows development. The root issue? Think of how Linux is developed, and you'll know the answer.
Linked by Thom Holwerda on 05/08/13 14:22 UTC
This is one of those news items that's fun to write, fun to read, fun to comment on, and where no one will be able to say anything unkind. It's all just one big ball of awesome fluffiness. TuneTracker, the BeOS radio automation software, has just released something very special: TuneTracker System 5, the first version designed entirely and specifically for Haiku. In fact, it actually includes Haiku in the software package. Better yet, TuneTracker also unveiled several system-in-a-box products - which have Haiku and TuneTracker pre-installed.
Linked by Thom Holwerda on 05/02/13 15:28 UTC
Exactly twenty years ago, a document was published that played a huge role in establishing the web as we know it today. Twenty years later, and this simple and straightforward document is proof of an irrefutable fact: while closed technologies can change markets, open technologies can change the world.
Linked by Thom Holwerda on 04/29/13 21:06 UTC
Oh multitasking. That staple of computing that got thrown out the window with many modern smartphones. We got some rudimentary thing in its place - but even as multitasking on phone and tablets improves, its user-visible side remains cumbersome. Windows 8 has a neat implementation, and now it's time Android follows in it footsteps.
Linked by Thom Holwerda on 04/24/13 22:24 UTC
They're here! Whether that excites you or not remains to be seen, but the Galaxy S4, which will most likely become the best selling Android smartphone of the year by a huge margin, has been reviewed by all the major sites, and there's lots of interesting conclusions in there - although I think most of you will get the gist.
Linked by Thom Holwerda on 04/18/13 11:21 UTC
Nokia has posted its quarterly results for the first quarter of 2013, and just like the quarters that came before, there's not a whole lot of good news in there. The rise in Lumia sales still can't even dream of making up for the sales drop in Symbian phones, and when broken down in versions, the sales figures for Windows Phone 8 Lumias in particular are very disappointing. In North America, Nokia is getting slaughtered.More Features »
Sponsored Links
Member since:
2008-11-19
It's better software design for tools to be simpler by default, not more complex by default. This gives better maintainability and fork-ability, as well as fewer bugs and security problems.