OSNews

Vim comes packaged with a default encryption mechanism referred to as VimCrypt in the documentation. I typically use this functionality as a temporary solution in a situation where my GPG is not immediately available, like a remote system that is not mine.

Using Vim's default VimCrypt feature

Creating a new encrypted file or open a plain text file you wish to encrypt:

vim -x 

This will create a new file if it does not exist or open an existing file and then prompt you for a password. This password is then used as the key to encrypt and decrypt the specified file. Upon saving and exiting this file, it will be saved in this encrypted format using your crypt key.

You can also save encrypt an open file you are currently working on like so:

:X 

Reasons I usually don't use this option:

• Weak encryption
  
• Typically uses swap files that can compromise the security of the encrypted file
  

Integrating with GPG

In order to seamlessly integrate with GPG encrypted files you will need to add the following to your .vimrc file

" Transparent editing of gpg encrypted files.
" By Wouter Hanegraaff
augroup encrypted
  au!

  " First make sure nothing is written to ~/.viminfo while editing
  " an encrypted file.
  autocmd BufReadPre,FileReadPre *.gpg set viminfo=
  " We don't want a swap file, as it writes unencrypted data to disk
  autocmd BufReadPre,FileReadPre *.gpg set noswapfile

  " Switch to binary mode to read the encrypted file
  autocmd BufReadPre,FileReadPre *.gpg set bin
  autocmd BufReadPre,FileReadPre *.gpg let ch_save = &ch|set ch=2
  " (If you use tcsh, you may need to alter this line.)
  autocmd BufReadPost,FileReadPost *.gpg '[,']!gpg --decrypt 2> /dev/null

  " Switch to normal mode for editing
  autocmd BufReadPost,FileReadPost *.gpg set nobin
  autocmd BufReadPost,FileReadPost *.gpg let &ch = ch_save|unlet ch_save
  autocmd BufReadPost,FileReadPost *.gpg execute ":doautocmd BufReadPost " . expand("%:r")

  " Convert all text to encrypted text before writing
  " (If you use tcsh, you may need to alter this line.)
  autocmd BufWritePre,FileWritePre *.gpg '[,']!gpg --default-recipient-self -ae 2>/dev/null
  " Undo the encryption so we are back in the normal text, directly
  " after the file has been written.
  autocmd BufWritePost,FileWritePost *.gpg u
augroup END

This works by detecting the extension on the files you are opening with Vim. This allows you to open, edit, and save files as if they were plain text in a seamless fashion.

Now you can create a new GPG encrypted file or edit an existing GPG encrypted doing this:

vim <filename>.gpg

This should prompt you for GPG password either with a GUI window or command line depending on your environment's configuration. If the file did not exist Vim creates one and when you save it will encrypt when you write and quit.

Another thing I like to do is save a file that I have already decrypted, but what to save in plain text. This can be done by simply opening the encrypted GPG file as seen above and change the extension when saving. Simply save like so:

:w <newfilename>.txt

Any extension other than .gpg will save your file as plain text.

Reason to use GPG

• Much safer encryption as it uses GPG
• No swap files thanks to this line in .vimrc autocmd BufReadPre,FileReadPre *.gpg set noswapfile

Of course when using Vim there are many features and many different ways of doing this. This is simply how I use Vim to easily work with encrypted files in my daily life.

For more information on Vim and external encryption programs please see and making use of GPG:

• Vim Encryption
• Working with GPG - Linux

It's been a fairly straight forward week at work, but I have stumbled a few interesting finds along the way this week.

Vim Adventures

Finally! A game based approach to learning Vim keyboard commands. I was hoping someone would do this. It's just getting started (only two levels) and sadly, it looks like it'll be charging money to unlock higher levels. However, some things are worth paying for. I've found just playing the first two levels a few times have helped retrain my brain to not take my fingers off the home row. It's still quite buggy and seems to only work in Chrome. I found several times I needed to close all my Chrome windows after playing. Also, incognito mode seems to help with the bugs, as it disables all extensions you may have installed.

MySQL query comments in Rails

Ever wanted to know where that slow query was being called from? Well, if you're using MySQL with your Rails 2.3.x or 3.x.x app, you can get debug information about what controller's action made the call. Check out 37Signals new marginalia gem.

How to use EC2 as a web proxy

Kevin Burke provides a very detailed HOWTO article for working around restrictions you may experience in the course of an Internet based life. Pretty amazing what Amazon's free usage tier puts out there; of course it's only free for 12 months.

Include PIDs in your Logs

For many Rails developers we get comfortable looking at development log files. Sometimes when I have to investigate a customer issue on a production server using logs, I wished I had the level of detail the development logger had. While that's a wish, I'm finding it mandatory to include PID numbers in my production logs. In production systems with multiple requests being handled simultaneously, Rails logs start to become unusable. It's not clear which log lines are from which requests. Adding a PID in front of the time stamp can help untangle the mess. Here are some example approaches to this for Rails 3.x.x and Rails 2.3.x. Also, if you're really a log-lover and manage a lot of servers, check out Papertrail, it looks very impressive for $7/mo.

Spectrum Shortages - Why it's happening and what can be done

Telecom is not an area I have much familiarity with, but I found this article to be an interesting read. For example did you know that that largest owner of spectrum licenses are "under-capitalized or unwilling to build out networks" to use the spectrum? So while AT&T and Verizon struggle to meet the iPhone 4S's data demands (twice as much as iPhone 4!), "there are some companies that have spectrum, but they're struggling financially. Or they aren't quite sure what to do with the spectrum. And others that have the money and business model, but need the spectrum." It seems the way out of the mess is 4G, offering to improve the efficiency of spectrum use by 700 percent.

If you don't want to use SELinux, fair enough. But I find many system administrators would like to use it but get flustered at the first problem it causes, and disable it. That's unfortunate, because often it's simple to customize SELinux policy by creating what's known as a local policy module. That way you allow the actions you need while retaining the added security SELinux brings to the system as a whole.

A few years ago my co-worker Adam Vollrath wrote an article on this same subject for Red Hat Enterprise Linux (RHEL) 5, and went into more detail on SELinux file contexts, booleans, etc. I recently went through the process of building an SELinux local policy module on a RHEL 6 mail server and found a few differences and want to document some of the details here. This applies to RHEL 5 and RHEL 6, and near relatives CentOS, Scientific Linux, et al.

When under pressure ?

If you're tempted to disable SELinux, consider leaving it on, but in "permissive" mode. That will leave it running but stop it from blocking disallowed actions until you have time to deal with them properly. It's as simple as:

setenforce 0

That will last until you reboot, unless otherwise changed manually. You can edit /etc/sysconfig/selinux and set:

SELINUX=permissive

To keep permissive mode even after a reboot. To see what mode SELinux is in, you can do either of:

getenforce
# or
cat /selinux/enforce

Prerequisites

First make sure you have installed:

yum install policycoreutils
yum install policycoreutils-python   # also needed on RHEL 6

You must have SELinux enabled, though enforcing isn't required; permissive mode is fine. If it's not enabled, edit /etc/sysconfig/selinux for permissive mode and reboot.

You'll need an up-to-date file /var/lib/sepolgen/interface_info, which is created by /usr/sbin/sepolgen-ifgen for the specific machine you're running it on. That should be done automatically, but be aware of it in case it somehow got stale. If you run into any unexpected problems, make sure the timestamp on interface_info is recent, or just regenerate it, which is harmless.

Making the policy module

Choose a unique name for your local policy module. It's better to use something specific to your organization, or the hostname, rather than just "postfix" or "dovecot" or something similar which may conflict with existing vendor policy modules.

Run semodule -l to list the existing modules. For this example I'll use "epmail".

Create a directory for your new policy module:

mkdir -p /root/local-policy-modules/epmail
cd /root/local-policy-modules/epmail

Copy relevant error messages verbatim from /var/log/audit/audit.log to a new file. Here for example are two denials of a script called by Postfix as a transport agent, which needed to connect to PostgreSQL locally:

type=AVC msg=audit(1335581974.308:69047): avc:  denied  { write } for  pid=14649 comm=F9616121202873696E676C65206D65 name=".s.PGSQL.5432" dev=sda2 ino=79924 scontext=system_u:system_r:postfix_pipe_t:s0 tcontext=system_u:object_r:postgresql_tmp_t:s0 tclass=sock_file
type=AVC msg=audit(1335581974.308:69047): avc:  denied  { connectto } for  pid=14649 comm=F9616121202873696E676C65206D65 path="/tmp/.s.PGSQL.5432" scontext=system_u:system_r:postfix_pipe_t:s0 tcontext=system_u:system_r:postgresql_t:s0 tclass=unix_stream_socket

In the logs you want to look for "AVC", which stands for Access Vector Cache and is how SELinux logs denials. You can grab all the recent denials with:

grep ^type=AVC /var/log/audit/audit.log > epmail.log

and then filter it manually to contain just what you need.

You can see a usually more informative explanation of each error by piping it into audit2why:

audit2why < epmail.log

Now you're ready to create your policy module:

audit2allow -m epmail < epmail.log > epmail.te
checkmodule -M -m -o epmail.mod epmail.te
semodule_package -o epmail.pp -m epmail.mod
semodule -i epmail.pp

That's a somewhat longwinded way to do things, but that's how I learned it from my co-worker Kiel, and it's easy once put into a script. See the man page of each program for more details on what that step is doing, and various options.

A more streamlined way that has audit2allow performing the functions of checkmodule and semodule_package is:

audit2allow -M $module_name -R -i epmail.log
semodule -i epmail.pp

Wrap-up

You will of course need to keep an eye on the audit log to look for any more AVC denials, as you exercise all the functions of the system. For a production system it may be best to leave SELinux permissive for a few weeks, and once you're confident you've allowed all the actions needed, you can switch it to enforcing mode.

Finally, I have not normally had to do this, but if you need to force reload the SELinux policy on the server, you can do it with:

semodule -R

Have fun with the extra security SELinux offers!

Here's another entry in my Three Things series, where I share a few small tips I've picked up lately.

1. Rails and Dramas

Sometimes I think that since Rails allows you write code efficiently, [a few] members of the Rails community have time to overdramatize incidents that otherwise would go relatively unnoticed :) Someone with a good sense of humor created this website to track these dramas. While it's probably a waste of time to get caught up on the personal aspects of the drama, some of the dramas have interesting technical aspects which are fiercely defended.

2. JOIN with concat

Recently I needed to perform a JOIN on a partial string match in MySQL. After some investigation, I found that I had use the CONCAT method in a conditional (in an implicit inner JOIN), which looked like this:

SELECT * FROM products p, related_items ri WHERE concat(p.sku, '%') = ri.id

In modern MVC frameworks with ORMs, databases are typically not designed to include data associations in this manner. However, in this situation, data returned from a third party service in a non-MVC, ORM-less application was only a substring of the original data. There may be alternative ways to perform this type of JOIN, and perhaps my fellow database experts will comment on the post with additional techniques ;)

3. Responsiveness

Responsive web design is all the rage lately from the increase in mobile web browsing and tablets. Here is a fun tool that that the ecommerce director at Paper Source pointed out to me recently. The website allows you to render a single URL at various widths for a quick review of the UI.

I spent three days last week attending the Utah Open Source Conference, in company with Josh Ausborne and Jon Jensen. Since End Point is a "distributed company", I'd never met Josh Ausborne before, and was glad to spend a few days helping and learning from him as we demonstrated the Liquid Galaxy he has already written about.

This time around, the conference schedule struck me as being particularly oriented toward front-end web development. The talks were chosen based on a vote taken on the conference website, so apparently that's what everyone wanted, but front end stuff is not generally my cup of tea. That fact notwithstanding, I found plenty to appeal to my particular interests, and a number of talks I didn't make it to but wished I had.

I delivered two talks during the conference, the first on database constraints, and the second on Google Earth and the Liquid Galaxy as they apply to geospatial visualization (slides here and here, respectively). Though I couldn't get past the feeling that my constraints talk dragged quite a bit, it was well received. Where possible I kept it as database-agnostic as possible, but no talk on the subject would be complete without mentioning PostgreSQL's innovative exclusion constraints. Their applicability to scheduling applications, by easily preventing things like overlapping time ranges, seemed particularly interesting to one attendee with recent experience writing such an application. Should I have opportunity to deliver the talk again, it will definitely include more examples of some of the more overlooked constraint types, as well as a more detailed description of the surrogate vs. natural keys, which generated quite a bit of discussion after I mentioned it in passing.

My mapping talk was less enthusiastically attended, which may well be due to the topic or the speaker, but it was also scheduled at 6:00 PM, in the last slot of the day, and I expect many attendees had gone home. UTOSC features an unusually high number of attendees with young families, compared to most conferences I've attended, and clears out relatively rapidly toward evening. The last day's tracks tend to be family-focused specifically because of all the parents who want to bring their children, and included hands-on labs, board game sessions, and child-friendly demonstrations.

Sparse attendance notwithstanding, I enjoyed introducing my audience to Google Earth's KML language, the Kamelopard library I've been working on to facilitate making KML, and some of the applications of Google Earth for visualization. We moved the Liquid Galaxy from our display booth to the classroom for my presentation; I expect it was one of the more involved demonstrations in any talk, and certainly deserves honorable mention for being a live demo that actually worked.

This post discusses version 1.25.0 of tail_n_mail, which can be downloaded at http://bucardo.org/wiki/Tail_n_mail

One of our clients recently had one of their Postgres servers crash. In technical terms, it issued a PANIC because it tried to commit a transaction that had already been committed. We are using tail_n_mail for this client, and while we got notified six ways to Sunday about the server being down (from Nagios, tail_n_mail, and other systems), I was curious as to why the actual PANIC had not gotten picked up by tail_n_mail and mailed out to us.

The tail_n_mail program at its simplest is a Perl script that greps through log files, finds items of interest, and mails them out. It does quite a bit more than that, of course, including normalizing SQL, figuring out which log files to scan, and analyzing the data on the fly. This particular client of ours consolidates all of their logs to some central logging boxes via rsyslog. For the host in question that issued the PANIC, we had two tail_n_mail config files that looked like this:

## Config file for the tail_n_mail program
## This file is automatically updated
## Last updated: Fri Apr 27 18:00:01 2012
MAILSUBJECT: Groucho fatals: NUMBER

INHERIT: tail_n_mail.fatals.global.txt

FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-err.log
LASTFILE: /var/log/2012/groucho/04/27/18/pgsql-err.log
OFFSET: 10199

## Config file for the tail_n_mail program
## This file is automatically updated
## Last updated: Fri Apr 27 18:00:01 2012
MAILSUBJECT: Groucho fatals: NUMBER

INHERIT: tail_n_mail.fatals.global.txt

FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-warning.log
LASTFILE: /var/log/2012/groucho/04/27/18/pgsql-warning.log
OFFSET: 7145

The reason for two files was that rsyslog was splitting the incoming Postgres logs into multiple files. Which is normally a very handy thing, because the main file, pgsql-info.log, is quite large, and it's nice to have the mundane things filtered out for us already. Because rsyslog also splits things based on the timestamp, we don't give it an exact file name, but use a POSIX template instead, e.g. /var/log/apps/%Y/groucho/%m/%d/%H/pgsql-warning.log. By doing this, tail_n_mail knows where to find the latest file. It also uses the LASTFILE and OFFSET to know exactly where it stopped last time, and then walks through all files from LASTFILE until the current one.

So why did we miss the PANIC? Because it was in a heretofore unseen and untracked log file known as pgsql-crit.log. (Which goes to show how rarely Postgres crashes: this was the first time in well over 700,000 log files generated that a PANIC had occurred!) At this point, the solution was to either create yet another set of config files for each host to watch for and parse any pgsql-crit.log files, or to give tail_n_mail some more brains and allow it to handle multiple FILE entries in a single config file. Obviously, I chose the latter.

After some period of coding, testing, debugging, and caffeine consumption, a new tail_n_mail was ready. This one (version 1.25.0) allows multiple values of the FILE parameter inside of a single config. Thus, for the above, I was able to combine everything into a single tail_n_mail config file like so:

MAILSUBJECT: Groucho fatals: NUMBER

INHERIT: tail_n_mail.fatals.global.txt

FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-warning.log
FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-err.log
FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-crit.log

The INHERIT file is a way of keeping common config items in a single file: in this case, groucho and a bunch of other similar hosts all use it. It contains the rules on what tail_n_mail should care about, and looks similar to this:

## Global behavior for all "fatals" configs
EMAIL: acme-alerts@endpoint.com
FROM: postgres@endpoint.com
FIND_LINE_NUMBER: 0
STATEMENT_SIZE: 3000
INCLUDE: FATAL:
INCLUDE: PANIC:
INCLUDE: ERROR:

## Client specific exceptions:
EXCLUDE: ERROR:  Anvils cannot be delivered via USPS
EXCLUDE: ERROR:  Jetpack fuel quantity missing
EXCLUDE: ERROR:  Iron Carrots and Giant Magnets must go to different addresses
EXCLUDE: ERROR:  Rocket Powered Rollerskates no longer available

## Postgres excceptions:
EXCLUDE: ERROR:  aggregates not allowed in WHERE clause
EXCLUDE: ERROR:  negative substring length not allowed
EXCLUDE: ERROR:  there is no escaped character
EXCLUDE: ERROR:  operator is not unique
EXCLUDE: ERROR:  cannot insert multiple commands into a prepared statement
EXCLUDE: ERROR:  value "d+" is out of range for type integer
EXCLUDE: ERROR:  could not serialize access due to concurrent update

Thus, we only have one file per host to worry about, in addition to a common shared file across all hosts. So now tail_n_mail can handle multiple files over a time dimension (by walking forward from LASTFILE to the present), as well as over a vertical dimension (by forcing together the files split by rsyslog). However, there is no reason we cannot handle multiple files over a horizontal dimension as well. In other words, putting multiple hosts into a single file. In this client's case, there were other hosts very similar to "groucho" that had files we wanted to monitor. Thus, the config file was changed to look like this:

MAILSUBJECT: Acme fatals: NUMBER

INHERIT: tail_n_mail.fatals.global.txt

FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-warning.log
FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-err.log
FILE: /var/log/%Y/groucho/%m/%d/%H/pgsql-crit.log

FILE: /var/log/%Y/dawson/%m/%d/%H/pgsql-warning.log
FILE: /var/log/%Y/dawson/%m/%d/%H/pgsql-err.log
FILE: /var/log/%Y/dawson/%m/%d/%H/pgsql-crit.log

FILE: /var/log/%Y/cosby/%m/%d/%H/pgsql-warning.log
FILE: /var/log/%Y/cosby/%m/%d/%H/pgsql-err.log
FILE: /var/log/%Y/cosby/%m/%d/%H/pgsql-crit.log

We've just whittled nine config files down to a single one. Of course, the config file cannot stay like that, as the LASTFILE and OFFSET entries need to be applied to specific files. Thus, when tail_n_mail does its first rewrite of the config file, it will assign numbers to each FILE, and the file will then look something like this:

FILE1: /var/log/%Y/groucho/%m/%d/%H/pgsql-warning.log
LASTFILE1: /var/log/2012/groucho/04/27/18/pgsql-warning.log
OFFSET1: 100


FILE2: /var/log/%Y/groucho/%m/%d/%H/pgsql-err.log
LASTFILE2: /var/log/2012/groucho/04/27/18/pgsql-err.log
OFFSET2: 2531

FILE3: /var/log/%Y/groucho/%m/%d/%H/pgsql-crit.log

FILE4: /var/log/%Y/dawson/%m/%d/%H/pgsql-warning.log
LASTFILE4: /var/log/2012/dawson/04/27/18/pgsql-warning.log
OFFSET4: 42

# etc.

By using this technique, we were able to reduce a slew of config files (the actual number was around 60), and their crontab entries, into a single config file and a single cron call. We also have a daily "error" report that mails a summary of all ERROR/FATAL calls in the last 24 hours. These were consolidated into a single email, rather than the half dozen that appeared before.

While tail_n_mail has a lot of built-in intelligence to handle Postgres logs, it is ultimately regex-based and can be used on any files which you want to track and receive alerts when certain items appear inside of them, so feel free to use it for more than just Postgres!

One of my first tasks at End Point was to inherit a production application from the lead developer who was no longer going to be involved. It was a fairly complex domain model and had passed through many developers' hands on a tight client budget. Adding to the challenge was the absence of any active development; it's difficult to "own" an application which you're not able to make changes to or work with users directly. Moreover, we had a short amount of time; the current developer was leaving in just 30 days. I needed to choose an effective strategy to understand and document the system on a budget.

Taking Responsibility

At the time I was reading Robert C. Martin's The Clean Coder, which makes a case for the importance of taking responsibility as a "Professional Software Developer". He defines responsibility for code in the broadest of terms.

From there he continues to expound in his declarative style about how to do no harm to the function and structure of the code. What struck me most about this was his conclusions about the necessity of testing. The only way to do no harm to function is know your code works as expected. The only way to know your code works is with automated tests. The only way to do no harm to structure is by "flexing it" regularly.

Test Suite for the Triple Win

Fortunately, there was a fairly large test suite in place for the application, but as common with budget-constrained projects, the tests didn't track the code. There were hundreds of unit tests, but they weren't even executable at first. After just a few hours of cleaning out tests for classes which no longer existed, I found about half of the 500 unit tests passed. As I worked through repairing the tests, I was learning the business rules, classes, and domain of the application, all without touching "production" code (win). These tests were the documentation that future developers could use to understand the expected behavior of the system (double win). While rebuilding the tests, I got to document bugs, deprecation warnings, performance issues and general code quality issues (triple win).

By the end of my 30 day transition, I had 500+ passing unit tests that were more complete and flexible than before. Additionally, I added 100+ integration tests which allowed me to exercise the application at a higher level. Not only was I taking responsibility for the code, I was documenting important issues for the client and myself. This helps the client feel I had done my job transitioning responsibilities. This trust leaves the door open to further development, which means a better system over the long haul.

A couple of days ago I had to change my notebook. I installed Ubuntu 12.04 on the new one, while on the previous one there was Ubuntu 11.10. There were no problems with copying all the files from the old to the new machine, including GPG and SSH keys. Everything went smoothly and I could connect to all the machines I needed.

The only problem was with VPN. While working for one of our clients, I need to connect to their VPN. On the old machine I did that through the Network Manager. Nothing easier, I went to the Network Manager, chose the Export option and saved all the settings to a file. I copied the file to the new computer and loaded it into the Network Manager.

The file loaded correctly. I could switch the VPN on. It said everything works. But in fact it didn't. The message was "VPN is connected", I could switch it on and off, but I couldn't access any of the client's resources available from my previous notebook.

The first thing I checked was the content of /etc/resolv.conf on both computers. The file without connecting to VPN looked like this on both computers:

$ cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 127.0.0.1

When I connected to the VPN the files on both computers were quite different. For example on my new computer (and Ubuntu 12.04) the content of the file looked like this:

$ cat /etc/resolv.conf 
# Generated by NetworkManager
domain something.net
search something.net
nameserver 127.0.0.1

I changed the data a little bit of course, so the domain names and IP addresses (except for 127.0.0.1) are not real.

On my old computer the resolv.conf file had a lot more entries, however I thought the above file should work as well. The problem was still the same: I couldn't connect to the client's resources.

The client is using the CISCO VPN, so I had to install network-manager-vpnc. This is just a plugin for network-manager which uses the vpnc program internally. I thought that maybe the plugin was doing something wrong.

I checked the plugin versions. Yes, they really differ. I started thinking about using the program without the Network Manager.

It turned out to be very simple to use. I need just a config file. The file is really simple:

IPSec gateway   something.net
IPSec ID        something.id
IPSec secret    somethingpass
Xauth username  mylogin
Xauth password  mypass

I keep all my local scripts in ~/bin (which can also be accessed as /home/szymon/bin). The directory ~/bin is added to the PATH environment variable. This way I can access all the scripts placed there in the console without providing the whole path. I did it by adding the following line at the end of my local ~/.bashrc file.

PATH=$PATH:$HOME/bin

To keep the things together I saved the config file at the same location ~/bin/vpn.conf.

Now I can connect to the VPN using:

$ sudo vpnc-connect /home/szymon/bin/vpn.conf

I can also stop the VPN using:

$ sudo vpnc-disconnect

To automate it a little bit I created a simple script stored at ~/bin/vpn:

#!/usr/bin/env bash


case "$1" in

start)
  sudo vpnc-connect /home/szymon/bin/vpn.conf
  ;;
stop)
  sudo vpnc-disconnect
  ;;
status)
  ps uaxf | grep vpnc-connect | grep -v grep
  ;;
restart)
  sudo vpnc-disconnect
  sudo vpnc-connect /home/szymon/bin/vpn.conf
  ;;
*)
  echo "Usage: vpn (start|stop|status|restart)"
  exit 1
  ;;

esac

This way I can simply write:

$ vpn start
[sudo] password for szymon: 
VPNC started in background (pid: 13771)...

I noticed that now the /etc/resolv.conf file contains different entries than when using the Network Manager plugin:

$ cat /etc/resolv.conf 
#@VPNC_GENERATED@ -- this file is generated by vpnc
# and will be overwritten by vpnc
# as long as the above mark is intact
# Generated by NetworkManager
nameserver 1.2.3.4
nameserver 1.2.3.4
search something.net

I can also disconnect from the VPN with simple command:

$ vpn stop
Terminating vpnc daemon (pid: 13771)

I'm using this script for a couple of days and I don't have any problems with the CISCO VPN. It seems like the vpnc program in Ubuntu 12.04 is OK, however there is something wrong with the Network Manager plugin for vpnc.

End Point had a table at the Utah Open Source Conference at Utah Valley University this week. We implemented a "mini Liquid Galaxy" system for the event, and it was a big hit. Most of the other sponsors were offering services or recruiting, but End Point offered a physical item which people could touch and engage with. This allowed us to present our product and services to people, as well as make contact with people who may be interested in joining our team.

Numerous people were really excited about the Liquid Galaxy. The most common thing that people did first when they started using it was to look for their own homes, but they quickly moved onto other areas that contained more 3D building content. They asked questions about the practical application of the system, as well how well it would play games. Multiple other people asked about the ability to build video walls with the system. One of the biggest things that people found interesting about the hardware was the 3D mouse from 3Dconnexion. It took some visitors a while to get the hang of it, but others picked it up quickly and found themselves really liking the way it interacts.

The mini LG consisted of a headless head node, three display nodes powering five 24" inch HP monitors, and one HP touch screen being used as a control panel. The head node was a Dell Inspiron 531s with a 2.8GHz AMD Athlon, 4GB of RAM, and a 160GB hard drive. The display nodes are Puget Systems-built with 2.6GHz i5 processors, 8GB of RAM, 80GB SSDs, and GeForce 210 video cards.

One of the things that we had to do differently with this system, as opposed to our normal installs, was to flatten out the semicircle of displays. The HP displays' viewing angle wasn't very wide, and was preventing people from being able to view the system unless they were right up near the displays. By flattening it out, more people were able to view the system and have their interest captured.

One thing that we also noticed during the event is that a Liquid Galaxy system performs very well when it has 60 Mbps of bandwidth available to it.

There was even a kid who found a way to keep himself entertained with the Liquid Galaxy. He found his house and started using the LG to "jump" on the trampoline in his backyard.

As developers, a lot of what we do is essentially problem solving. Sometimes it's a problem of how to implement a specific feature. Sometimes it's a problem *with* a specific feature. Last week, I ran into a case of the latter in some relatively mature code in the Rails app I was working on.

I was getting a sporadic exception while trying to save an instance of my StoredFile model. I encountered the problem while implementing a pretty trivial boolean field in my model, while I was playing around with it in the rails console. This is where it gets a little weird.

The exception message:

#<NoMethodError: undefined method 'uniq' for "":String>
#Backtrace:
... acts-as-taggable-on-2.2.2/lib/acts_as_taggable_on/acts_as_taggable_on/core.rb:264:in 'block in save_tags'
...rest of backtrace...

Note that none of my work was related to my model's use of acts_as_taggable_on. I looked briefly at line 264 and its cohorts in core.rb, but nothing jumped out as "a giant, obvious bug in acts-as-taggable-on" (which I wouldn't expect.) Also, the actual error is a bit suspicious. I love duck typing (and ducks) as much as anyone, but it's pretty rare to see well traveled code try to call uniq() on a String. An empty array, sure, no problem. I'll call uniq() on an empty array all day - but a String? Madness.

In the absence of any obvious answers, I switched to serious debug mode. I remembered that I ran into this same issue briefly about two weeks ago, but it had fixed itself. Then I remembered that this app is not Skynet and is therefore incapable of "fixing itself." OK, so there's been a bug lurking for a while. I was going to need a code debugging montage to get to the bottom of this. I fired up some techno music and the blue mood lighting in my office, and got down to debugging.

By the end of the debugging montage, I had determined that my StoredFile model was overriding the "tag_list" method supplied by ActsAsTaggableOn, in order to return the tags for this StoredFile instance regardless of which User owned the tags. Here's our entire method:

def tag_list
    @tag_list ||= self.anonymous_tag_list(:tags)
end

Can you guess where the issue might be with this code? Hint: It's the @tag_list instance variable. We're actually only using it here as a lazy way to cache the return value of self.anonymous_tag_list(:tags) during the lifespan of this single instance. I came to discover that ActsAsTaggableOn defines and uses an instance variable of the same exact name within the scope of my model. So, my tag_list method was assigning "" to the @tag_list instance variable for a StoredFile instance with zero tags, and that was trampling the [] that ActsAsTaggableOn would expect in that scenario. Hence, the ill-fated attempt to call "".uniq().

As a bonus, because it's an instance variable assigned inside a method, it would only get populated/trampled by my StoredFile model if an instance's tag_list was examined in any way. A sort of reverse Heisenbug, if you will. Renaming @tag_list to something else fixed the bug.

It was a problem and I solved it, which is cool. But, is this injection of instance variables considered poor behavior on ActsAsTaggableOn's part? What do you think? I'm still not sure how I feel about it, but I do know how I felt when I figured it out and fixed it:

Git is great, but it's not always easy to use. For example, reverting a commit is a very nice feature. There are git commands for reverting a commit which has not been pushed to the main repository. However after pushing it, things are not so easy.

While I was working for one of our clients, I made about 20 commits and then I pushed them to the main repository. After that I realised that I was working on a wrong branch. The new branch I should have used wasn't created yet. I had to revert all my commits, create the new branch, and load all my changes into it.

Creating the branch named NEW_BRANCH is as easy as:

$ git branch NEW_BRANCH

Now the harder part... how to delete the commits pushed to the main repo. After reading through tons of documentation it turned out that it is not possible. You cannot just delete a pushed commit. However you can do something else.

As an example of this, I created a simple file, added a couple of lines there, and made four commits. The git log looks like this:

$ git log
commit dc47a884f7b303fc8b207550104f5a1de192c91c
Author: Szymon Guz 
Date:   Mon Apr 30 12:14:21 2012 +0200

    replaced b with d

commit 68f56d3321324bd14cd1e73d003b1e151c4d43b4
Author: Szymon Guz 
Date:   Mon Apr 30 12:14:05 2012 +0200

    added c

commit a77427d8151f143cacb85f00eb6c8170079dc290
Author: Szymon Guz 
Date:   Mon Apr 30 12:13:58 2012 +0200

    added b

commit 73e586bb6d401f4049cf977703f25bf47c93b227
Author: Szymon Guz 
Date:   Mon Apr 30 12:13:49 2012 +0200

    added a

Now let's move the last 3 commits to another branch. I will create one diff for reverting the changes and one for replaying them on the new branch. Let's call these the 'down' and 'up' diff files: 'down' for reverting, and 'up' for recreating the changes.

The up diff can be created with:

$ git diff 73e586bb6d401f4049cf977703f25bf47c93b227 dc47a884f7b303fc8b207550104f5a1de192c91c
diff --git a/test b/test
index 7898192..3171744 100644
--- a/test
+++ b/test
@@ -1 +1,3 @@
 a
+d
+c

The down diff can be created using exactly the same command, but with switched parameters:

$ git diff dc47a884f7b303fc8b207550104f5a1de192c91c 73e586bb6d401f4049cf977703f25bf47c93b227
diff --git a/test b/test
index 3171744..7898192 100644
--- a/test
+++ b/test
@@ -1,3 +1 @@
 a
-d
-c

I saved the diffs into files called 'up.diff' and 'down.diff'.

On the old branch I want to revert the changes, after doing this I will just commit the changes and the branch will look like it was before all the commits. However all the commits stay in the branch. This something like a revert commit.

I reverted the changes on current branch with:

$ patch -p1 < down.diff 
patching file test
$ git commit -a -m "reverted the changes, moved to another branch"

Now let's move the changes into the new branch. I need to create the new branch from the repo after the first commit:

$ git branch NEW_BRANCH 73e586bb6d401f4049cf977703f25bf47c93b227

Switch to the new branch:

$ git checkout NEW_BRANCH

Apply the up.diff patch to the new branch:

patch -p1 < up.diff

And commit the changes:

$ git commit -a -m "Applied changes from the other branch"

I know that all the steps can be replaced with different ones, however this solution worked for me pretty well and without any problem.

This week I was asked to isolate some serious performance problems in a Rails application. I went down quite a few paths to determine how to best isolate the issue. In this post I want to document what tools worked most quickly to help find offending code.

Benchmarks

Before any work begins finding how to speed things up, we need to set a performance baseline so we can know if we are improving things, and by how much. This is done with Ruby's Benchmark class and some of Rail's Benchmark class.

The Rails guides would have you setup performance tests, but I found this cumbersome on this Rails 2.3.5 application I was dealing with. Initial attempts to set it up were unfruitful, taking time away from the task at hand. In my case, the process of setting up the test environment to reflect the production environment was prohibitively expensive, but if you can automate the benchmarks, do it. If not, use the logs to measure your performance, and keep track in a spreadsheet. Regardless of benchmarking manually or automatically, you'll want to keep some kind of log of the results keeping notes about what changed in each iteration.

Isolating the Problem

As always, start with your logs. In Rails, you get some basic performance information for free. Profiling code slows down runtime a lot. By reviewing the logs you can hopefully make a first cut at what needs to be profiled, reducing already long profile runs. For example, instead of having to profile an entire controller method, by reading the logs you might notice that it's just a particular partial which is rendering slowly.

Taking a baseline benchmark

Once you've got a sense of where the pain is, it's easy to get a benchmark for that slow code as a baseline.

module SlowModule
  def slow_method
    benchmark "SlowModule#slow_method" do
      #my slow code
    end
  end
end

Look to your log files to see results. If for some reason, you're outside your Rails enviornment, you can use Ruby's Benchmark class directly.

require 'benchmark'
result = Benchmark.ms do
  #slow code
end
puts result

This will tell you the process time in milliseconds and give you a precise measurement to compare against.

Profiling with ruby-prof

First, setup ruby-prof. Once installed, you can add these kinds of blocks around your code.

require 'ruby-prof'

module SlowModule
  def slow_method
    benchmark "SlowModule#slow_method" do
      RubyProf.start
      # your slow code here
      results = RubyProf.end
    end
    File.open "#{RAILS_ROOT}/tmp/SlowModule#slow_method_#{Time.now}", 'w' do |file|
      RubyProf::CallTreePrinter.new(results).print(file)
    end
  end
end

Keep in mind that profiling code will really slow things down. Make sure to collect your baseline both with profiling and without, to make sure you're doing apples-to-apples comparison.

By default ruby-prof measures process time, which is the time used by a process between any two moments. It is unaffected by other processes concurrently running on the system. You can review the ruby-prof README for other types of measurements including memory usage, object allocations and garbage collection time.

If you choose to measure any of these options, make sure your Ruby installation has the tools a profiler needs to collect data. Please see the Rails guides for guidance on compiling and patching Ruby.

Interpreting the Data with KCachegrind

At this point you should have a general sense of what code is slow having reviewed the logs. You've got a benchmark log setup with baseline measurements to compare to. If you're going to Benchmark while your profiling, make sure your baseline includes the profile code; it will be much slower! Remember we want an apples-to-apples comparison! You're ready to start profiling and identifying the root source of the performance problems.

After manually or automatically running your troubled code with the profiling block above, you can open up the output from ruby-prof and quickly find it not to be human friendly. Fortunately, KCachegrind turns that mess into something very useful. I found that my Ubuntu installation had a package for it already built, so installation was a breeze. Hopefully things are as easy for you. Next simply open your result files and start reviewing there results.

The image above shows what's called a "call graph" with the percentages representing the relative amount of time that method uses for the duration of the profile run. The CacheableTree#children method calls Array#collect and takes up more then 90% of the runtime. The subsequent child calls are relatively modest in proportion. It's clear we can't modify Array#collect so let's look at CacheableTree#children.

module CacheableTree
  def children(element = @root_element)
    full_set.collect { |node| if (node.parent_id == element.id)
      node
    }.compact
  end
end

Defined elsewhere, full_set is an array of Ruby objects. This is common performance optimization in Rails; collecting data looping through arrays works well with a small data set, but quickly becomes painful with a large one. It turned out in this case that full_set had 4200+ elements. Worse yet the children method was being called recusrively on each of them. Yikes!

At this point I had to decide how to optimize. I could go for broke and completely break the API and try and clean up the mess, or I could see if I could collect the data more quickly, some other way. I looked at how the full_set was defined and found I could modify that query to return a subset of elements rather easily.

module CacheableTree
  def children(element = @root_element)
    FormElement.find_by_sql(...) #the details aren't important
  end
end

By collecting the data directly via a SQL call, I was able to cut my benchmark by about 20%. Not bad for a single line change! Let's see what the next profile told us.

The above is another view of the profile KCachegrind provides. It's essentially the same information, but in table format. There were a few indicators that my optimization was helpful:

• The total process_time cost had dropped
• The amount of time spent in each function seemed to better distributed - I didn't have a single method soaking up all the process time
• Most of the time was spent in code wasn't mine!

module CacheableTree
  def children(element = @root_element)
    @children ||= {}
    @children[element] ||= FormElement.find_by_sql(...) #the details aren't important
  end
end

This is called memoization and let's us reuse this expensive method's results within the page load. This method took another 10% off the clock against the baseline. Yay!

Knowing When to Stop

Performance optimization can be really fun, especially once all the infrastructure is in place. However, unless you have unlimited budget and time, you have to know when to stop. For a few lines of code changed, the client would see ~30% performance improvement. It was up to them to decide how much further to take it.

If allowed, my next step would be to make use of the applications existing dependence on redis, and add the Redis-Cacheable gem. It allows you to marshal Ruby objects in and out of a redis server. The application already makes extensive use of caching, and this page was no exception, but when the user modified the page in a way that expired the cache, we would hit this expensive method again, unnecessarily. Based on the call graph above, we could eliminate another ~66% of the call time, and perhaps, by pre-warming this cache, could help the user to minimize the chances of experiencing the pain of slow browsing!

Are you excited about Google+ Hangouts? Would you be even more excited to implement live streaming yourself?
Today we will take a look under the hood of broadcasting and even start implementing our own mini hangout.

A bit of theory

Live streaming is based on RTMP protocol. It is able to transfer video, audio and generally any data in real-time over Internet between server and client.
The most popular combination for video/audio streaming is Adobe Flash Player as a client software and a proprietary Adobe Flash Media Server as a server software. Another option is Wowza Streaming solutions.
Luckily for us there is an open-source Red5 Media Server ? the most popular if not the only one stable of all open-source media streaming servers. We will be leveraging Red5 to dive into RTMP world and the cutting edge streaming technologies.

Installing Red5 Media Server

The official installation tutorial suggests building Red5 from scratch. However, the official compiled binaries for Mac OS X, Linux and Windows are also available.
We will choose an easy path and install Red5 from the corresponding binary. I am using Mac OS X Snow Leopard.
Sure enough, when we first click on the Red5 icon in the "Applications" folder...nothing really happens. A peek into /Applications/Red5.app/Contents/Resources/Java/log/red5.log reveals the following:

2012-04-24 02:01:03,748 [main] WARN  org.red5.server.ContextLoader - Context destroy failed for:
default.context org.springframework.beans.factory.NoSuchBeanDefinitionException:
No bean named 'default.context' is defined at 
org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition
(DefaultListableBeanFactory.java:527) ~[org.springframework.beans-3.0.5.RELEASE.jar:3.0.5.RELEASE]

It only means that Red5 cannot find the necessary plugins for the Tomcat server. Tomcat used to be part of the Red5 installation, but was recently taken out of it.
Let's create a folder /Applications/Red5.app/Contents/Resources/Java/plugins and drop the following files inside:

catalina-6.0.26.jar
jasper-6.0.26.jar
jasper-jdt-6.0.26.jar
jasper-el-6.0.26.jar
tomcat-coyote-6.0.26.jar
tomcat-juli-slf4j-1.5.0.jar

I also put these libraries on GitHub for convenience. And if you plan to run Red5 with the application container other than Tomcat, Red5 Wiki lists other JEE Container plugins as well.
This time Red5 should launch successfully, and in a second we should be able to enjoy the greeting screen at http://localhost:5080/. Red5 uses 5080 as the default port for its web applications.

Exploring Red5 Media Server

Red5 Media Server is successfully installed. Time to implement a sweeping live stream between our two laptops.
Red5 comes with the set of demo applications that are very handy to base the development on. Demo applications can be installed via the Installer. For the purpose of this article we need to install the "oflaDemo" application.
After installation is complete, we can launch the demo for Simple Broadcaster application. In Simple Broadcaster we click "Connect" button, grant Flash Player permission to use camera and microphone, and  ? voila! ? we should now be broadcasting live via Simple Broadcaster application to oflaDemo application.
If we use another browser  to open Simple Subscriber application, or even another laptop with the IP  of the broadcasting computer, we will be able to stream the video live. It is quite noticeable that quality drops in Simple Subscriber.

In this example we also learned that rtmp://localhost/oflaDemo is the server communication point to where we publish and stream videos.

Customizing SimpleBroadcaster and SimpleSubscriber

Next comes the most interesting part - embedding SimpleBroadcaster and SimpleSubscriber into the web page.
We can't really use their default versions "as is". We want to remove input box, "Connect" button and Red5 connection indicator. We also want the streaming to start automatically.
It's more convenient to implement "Start" button in HTML and trigger the loading of SWF on "click", because then we can easily change it, while changing SWF requires a lot of effort.  You will be able to understand how exactly much effort it takes in a minute.
My plan is to use Flash CS5 to modify SimpleBroadcaster.fla and SimpleSubscriber.fla. Flash Builder 4.6 to modify ActionScript classes for SimpleBroadcaster and SimpleSubscriber. Finally I will use Flash compiler to generate SimpleBroadcaster.swf and SimpleSubscriber.swf from .as and .fla files.
Finally, some coding versus talking!
Red5 sources are conveniently available as a Google Code project. It comes in server-side and the most interesting to us flash-side

 > svn checkout http://red5.googlecode.com/svn/flash/trunk/ red5

Step 1.
Let's open simpleBroadcaster.fla located in the very root of the download in Flash CS5.

and take a pick at the Library panel.

Let's double-click on the Embedded Video 1, open Window -> Properties panel and modify the size of the video to occupy the full documents area. Alternatively, we may change the document size to fit the video: right-click on the grey area, pick Document settings from the context menu and adjust dimensions. Either way, now we should drag the video rectangle to fit the document. We will save the document and repeat the same sequence of actions for simpleSubscriber.fla, because they are really no different.

Step 2.
Let's now open the "red5" directory that we checked out in FlashBuilder.

We immediately find the main class for simpleBroadcaster.fla org.red5.samples.livestream.broadcaster.Main in "classes" directory.
The key function here is "configUI".

private function configUI():Void   
{  
    // setup the tooltip defaults  
    Tooltip.options = {size:10, font:"_sans", corner:0};  
    // setup cam  
    cam = Camera.get();  
    cam.setMode(480, 320, 15);  
    cam.setQuality(0,80);  
    // setup mic
    mic = Microphone.get();  
    // get notified of connection changes  
    connector.addEventListener("connectionChange", this);  
    // set the uri  
    Connector.red5URI = "rtmp://localhost/oflaDemo";  
    // initialize the connector  
    connector.configUI();   
} 

Let's look into Connector class org.red5.utils.Connector.

 // UI Elements:
 // ** AUTO-UI ELEMENTS **
 private var alert:SimpleDialog;
 private var connect:IconButton;
 private var disconnect:IconButton;
 private var light:ConnectionLight;
 private var uri:TextInput;
 // ** END AUTO-UI ELEMENTS **
 
 public function configUI():Void 
 {
  // instantiate the con  qnection
  connection  = new Connection();

  // register the connection with the light so it can add a listener
  light.registerNC(connection);

  // hide disconnect button
  disconnect._visible = false;

  // set the URI
  uri.text = red5URI;

  // setup the buttons
  connect.addEventListener("click", Delegate.create(this, makeConnection));
  disconnect.addEventListener("click", Delegate.create(this, closeConnection));
  connect.tooltip = "Connect to Red5";
  disconnect.tooltip = "Disconnect from Red5";

  // add listener for connection changes
  connection.addEventListener("success", Delegate.create(this, manageButtons));
  connection.addEventListener("close", Delegate.create(this, manageButtons));
 }

 public function makeConnection(evtObj:Object):Void
 {
  if(uri.length > 0) 
  {
    var goodURI = connection.connect(uri.text, getTimer());
    if(!goodURI) alert.show("Please check connection URI String and try again.");
  }
 }

We will now make quick and dirty changes to get the proof of concept working. We will hide all the unnecessary UI elements, then convert the listener to the regular function, and call it from the configUI function.

public function configUI():Void 
 {
  disconnect._visible = false;
  connect._visible = false;
  uri._visible = false;
  light._visible = false;

  connection  = new Connection();
  light.registerNC(connection);
  if (makeConnection(Connector.red5URI)) {
    dispatchEvent({type:"connectionChange", connected: true, connection:connection});
  }
 }

 public function makeConnection(uri:String):Boolean
 {
  var result:Boolean = false;
  if(uri.length > 0) 
  {
    result = connection.connect(uri, getTimer());
    if(!result) {
      alert.show("Failed to connect.");
    }
  }
  return result;
 }

Done! After making these changes we are good with SimpleSubscriber as well, because both classes use the same Connector class.

Step3
Unfortunately Red5 is written in ActionScript 2. We can't really use any of the newer mxml compilers installed on our systems, because they compile ActionScript 3. Flex SDK is also only compatible with ActionScript 3. I spent quite a lot of time searching for a simple and working AS2 compiler on the Internet, and luckily, found one called MTASC.
In order to compile the file with MX components in it, the compiler needs to be supplied with the correct Component classes. Flex SDK will not help here, obviously, but old Macromedia Flash MX 2004 will. In case you don't have Macromedia Flash MX 2004 installed on your system, I put all the necessary files, including Macromedia Flash MX 2004 SDK and MTASC compiler for OS X on GitHub.

> export PATH="/path/to/Red5Kit/bin:/path/to/Red5Kit/mx2004/First Run/Classes:$PATH"
> cd "/path/to/red5/classes" 
> mtasc -cp "." -swf simpleBroadcaster.swf -mx org/red5/samples/livestream/broadcaster/Main.as -v
> mtasc -cp "." -swf simpleSubscriber.swf -mx org/red5/samples/livestream/broadcaster/Main.as -v

Are you ready to broadcast?

In order to enable broadcasting in our web application, we will need to drop the compiled swf files into its public directory.
We will then download swfobject.js JavaScript library and drop it into the public directory of our web application as well.
We will create broadcast.html file.

<html>
 <head>
 <title>Red5 Demo - Simple Broadcaster</title>
 <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
 <script type="text/javascript" src="swfobject.js"></script>
 <script type="text/javascript">
 swfobject.embedSWF("simpleBroadcaster.swf","myContent","360","240","8.0.0","jwplayer/expressInstall.swf");
 </script>
<style media="screen" type="text/css">#myContent {visibility:hidden}</style>
 </head>
 <body>
 <div id="myContent">
 <h1>You need the Adobe Flash Player for this demo, download it by clicking the image below.
</h1>
 <p><a href="http://www.adobe.com/go/getflashplayer">
<img src="http://www.adobe.com/images/shared/download_buttons/get_flash_player.gif" alt="Get Adobe Flash player"/>
</a></p>
 </div>
 </body>
</html>

Then we will create watch.html file.

<html>
<head>
  <title>Red5 Demo - Simple Subscriber</title>
 <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
 <script type="text/javascript" src="swfobject.js"></script>
 <script type="text/javascript">
 swfobject.embedSWF("simpleSubscriber.swf","myContent","360","240","8.0.0","assets/expressInstall.swf");
 </script>
<style media="screen" type="text/css">#myContent {visibility:hidden}</style>
 </head>
 <body>
 <div id="myContent">
 <h1>You need the Adobe Flash Player for this demo, download it by clicking the image below.
</h1>
 <p><a href="http://www.adobe.com/go/getflashplayer">
<img src="http://www.adobe.com/images/shared/download_buttons/get_flash_player.gif" alt="Get Adobe Flash player"/>
</a>
</p>
 </div>
</body>
</html>

At this point we are able to start a broadcast at http://localhost/broadcast.html and watch it at http://localhost/watch.html! In this case I am broadcasting from Macbook and streaming to Windows laptop.

Sweet! Next step is to implement the actual two-way hangout with Red5 Media Server, so stay tuned for the upcoming posts about it.

I tend to walk away from every RailsConf feeling that there was a trend for each conference. In 2009, I thought the trend was Passenger, in 2010, NoSQL and Rails 3 was all the rage, and in 2011, many of the talks were related to CoffeeScript and SASS. While there was some of these technical specifics at this conference, I felt that the overarching trend this year is that several folks that are extremely involved in Rails (DHH, Yehuda Katz, Aaron Patterson, etc.) are very passionately involved in the course correction needed to keep Rails relevent.

As I mentioned in my blog article on Day 1, DHH talked about progress. DHH focused on the limitations of progress (loss aversion) and how and why people move from curious to suspicious when working with technology. He didn't delve into any of the Rails 4 details, but simply ended by sharing that Rails 4 will break and challenge things.

On Day 2, Aaron Patterson covered a discussion about what it means to incur technical debt, and how developers have different thresholds of how much technical debt is tolerated.

He talked about the types of refactoring needed to reduce technical debt. He ended his talk with a slide that said, "be prepared", which I understood to mean that there's a lot of movement, deep consideration, and no doubt passion. Aaron also participated in a Rails core panel on Day 2. While it was thoroughly entertaining (coloring book contest included!), there was some discussion over where Rails is going and the various perspectives. Aaron talked about how he is critical of Rails because he cares a lot, and the critical analysis of Rails is happening to keep Rails relevent in the future.

On Day 3, Yehuda Katz presented his talk on Rails: The Next Five Years. Katz noted that his perspective is not representative of the core team nor of his employer, that he comes from the perspective of developing very rich client-side web applications (that may also translate to mobile applications), and that part of the justification of his talk was to evangelize some of the technical features that he believes should be included in the core to keep Rails relevant.

For him, every time his company starts a new project and reevaluates whether Rails meets their needs means that something about Rails isn't working right now. Yehuda covered his examination of why Rails was great in the first place. And the underlying answer to that is convention over configuration. Rails eliminates trivial choices like code organization, assets, naming, routing, etc. and there is a shared benefit in this solution with convention. Gaining deep knowledge about hard problems is hard (e.g. CSRF prevention) and Rails makes these decisions for us with convention. Yehuda's perspective is [now] that any convention is better than no convention, because adding any convention upfront will allow something to be built out and improved incrementally over time rather than introducing divergent paths for trivial decisions.

<sidebar>

Yehuda took a short time in his talk to discuss justification for getting things in the core, even if they need follow-up iterations:

1. Getting new features into core gets a lot of eyeballs on it.
2. More importantly, getting new features into core gets a lot of eyeballs attached to people who have a lot of domain expertise on the problem that the feature is solving.
3. Getting new features into core helps reveal conceptual blindspots and additional required functionality.

</sidebar>

After talking about Rails and his desire to get new relevent features into Rails from at a high level, Yehuda dug into a few specific wants:

• serialization: This took up a large proportion of the talk. I can't give justice to everything covered here, so check out the slides and I'll post a link to the video later.
• bulk editing
• better convention for application structure (e.g. assets)
• API between JS framework and Rails, as the popularity of JS frameworks grows

Coding with convention (middle) allows developers* to avoid making diverging trivial decisions (left). Conventions provide the ability to share resources and domain expertise and add new features quickly, even if course correction (right) is needed on those conventions.
* not to be confused with sperm :)

What Now?

I think what Yehuda said during QnA resonated with me most: Decoupling the data from the representation is really important because the data then becomes represented in a variety of ways. Because of the extremely quick growth in technology and tools of various media consumption devices (browser, mobile web, native client apps), data is represented in an increasing number of ways and with varied degrees of interactivity, so what I took away from the conference is the view-level is where there may be some serious surgery in Rails. Obviously, there are a lot of core members looking at a variety of things in the stack, but I perceive that Rails needs to solve the problem of offering a strong convention for data representation to keep it relevant to new and growing technologies, while allowing Rails to continue to flourish.

It's Day 3 of RailsConf 2012 here in Austin Texas. Check out my posts from Day 1 and Day 2.

RailsConf 5k

To kick off the day, I ran the low-key RailsConf 5k along the lake here in Austin. I'm glad that this event was organized — I think it's good to take a mental break from all the conference activities.

RubyRogues

This morning kicked off with a RubyRogues live session. The topic for today's podcast was a discussion about what Rails developers care about and/or should care about. I thought that the answers apply to developers in general. Below are some of the topics covered in the talk.

• James Edward Gray II
  • James focused on the point that experimentation is important and to perceive all code as experimental.
• Avdi Grimm
  • Avdi's main argument was a suggestion to practice deliberate and mindful coding. While he believes that there is a place for the "shut up and code" mentality in some projects depending on risk, it's not acceptable to have this perspective when it comes to community sharing of knowledge.
  • Avdi recommends that exercising introspection by reviewing your code and explaining it to someone else (how and why) will make you a better programmer and communicator.
• David Brady
  • David shared that Ruby is not a block oriented lexically scoped language.
  • He suggests that you should a) learn Ruby as if it's unlike the other languages you know b) learn CoffeeScript because it writes better JavaScript than you will c) join the community and d) don't be a douchebag (translation: be humble, not arrogant) because d-bags make a bad stereotype for Rails programmers
• Josh Susser
  • Josh suggests to exercise prudence, or acting with or showing care and thought for the future by building a strong foundation.
  • He also discussed how while he agrees with comments of previous keynotes (loss aversion is pillar of conservatism, experienced developers have a lower tolerance for technical debt), that he also has the perspective that he has a limited budget for risk in projects. From his experience, bundler was a huge win in minimizing risk, but asset pipeline (in its current state) spent all of his risk budget.
• Charles Max Wood
  • Charles focused on how the Rails community should try to encourage and bring other developers to the community by evangelizing it. We should reach out to other communities and groups to encourage them to try Rails.

Lightning Talks

The last session I attended at RailsConf was the Lightning Talk session. I know I've abused lists in my RailsConf posts a bit, but Lightning Talks are meant to be presented in list form! Here's a quick rundown of some of the topics presented in the Lightning Talks, which were a mix of 1-minute and 5-minute talks. I'll update with more links as they become available.

• NSRails is a gem that bridges objective-C and rails, and will give Objective-C a bunch of methods similar to Rails methods (.all,.find(x), CRUD behavior)
• WindTunnel is a JavaScript testing framework without hassle.
• config.threadsafe is a setting in configuration that allows threading. Tony Arcieri provided a lot more details — I'll update this post to include a link to the slides later.
• Worker is a threading JavaScript tool.
• iwanttolearnruby.com is a resource for Ruby/Rails learning resources
• @marksim talked about how private teams should work like open source by a) doing more code reviews and b) using distributed communication tools like HipChat, Campfire, IRC, message boards, mailing lists, internal blogs, Google+ Hangout, and Skype to have success like open source does.
• sidekiq is a distributed message queue (like resque)
• erd is a gem that was written during RailsConf to allow visualization and changes of the database. I'll post a link to it when it becomes available.
• Rowe is a development technique that means trading results for money (as opposed to time).
• The job_interview gem is a silly (and extremely entertaining) gem created during RailsConf for helping you answer those tough interview questions.
• In Rails Engines, routes should a) always be drawn on the engine, and b) the engine should call isolate_namespace.
• Tokaido is a kickstarter project that Yehuda's been working on.
• railcar is an isolated, compiler-optional Rails environment (arcturo/Railcar).
• ninjascript is a JavaScript tool for unobtrusively adding event listeners, and more.
• tddium is a continuous testing integration tool.
• wicked is a gem for rendering wizards.
• flash_s3_rails is a gem for embedding a multi-file upload widget into your application.
• javascript-and-friends is a Google Group for discussing and acting on embedding JavaScript in Ruby.
• graphite is a scalable real-time monitoring tool.

A Little Bit More...

I also attended Yehuda Katz's talk on Rails: The Next Five Years. I'm saving a write-up on this for another post, because I'd like to wrap it in with some of the other keynotes discussions.