OSNews

Lately, Ron, Ethan, and I have been blogging about several of our CakePHP learning experiences, such as incrementally migrating to CakePHP, using the CakePHP Security component, and creating CakePHP fixtures for HABTM relationships. This week, I came across another blog-worthy topic while troubleshooting for JackThreads that involved auto login, requests that were forced to be secure, and infinite redirects.

Ack! Users were experience infinite redirects!

Some users were seeing infinite redirects. The following use cases identified the problem:

• Auto login true, click on link to secure or non-secure homepage => Whammy: Infinite redirect!
• Auto login false, click on link to secure or non-secure homepage => No Whammy!
• Auto login true, type in secure or non-secure homepage in new tab => No Whammy!
• Auto login false, type in secure or non-secure homepage in new tab => No Whammy!

So, the problem boiled down to an infinite redirect when auto login customers clicked to the site through a referer, such as a promotional email or a link to the site.

After I applied initial surface-level debugging without success, I decided to add excessive debugging to the code. I added debug statements throughout:

• the CakePHP Auth object
• the CakePHP Session object
• the app's app_controller beforeFilter that completed the auto login
• the app's component that forced a secure redirect on several pages (login, checkout, home)

I output the session id and request location with the following debug statement:

$this->log($this->Session->id().':'.$this->here.':'.'/*relevant message about whatsup*/', LOG_DEBUG);

With the debug statement shown above, I was able to compare the normal and infinite redirect output and identify a problem immediately:

2009-12-09 11:44:55 Debug: d3c2297ddea9b76605cb7a459f45965b:/:     User does not exist!
2009-12-09 11:44:55 Debug: d3c2297ddea9b76605cb7a459f45965b:/:     Success in auto login!
2009-12-09 11:44:55 Debug: d3c2297ddea9b76605cb7a459f45965b:/:     redirecting to /sale
2009-12-09 11:44:55 Debug: d3c2297ddea9b76605cb7a459f45965b:/sale: User exists!
2009-12-09 11:44:55 Debug: d3c2297ddea9b76605cb7a459f45965b:/sale: calling action!

2009-12-09 11:43:30 Debug: 65cb23e4ca358b7270513cca4a52e9b7:/:      User does not exist!
2009-12-09 11:43:30 Debug: 65cb23e4ca358b7270513cca4a52e9b7:/:      Success in auto login!
2009-12-09 11:43:30 Debug: 65cb23e4ca358b7270513cca4a52e9b7:/:      redirecting to /sale
2009-12-09 11:43:30 Debug: 397f099790347716e0bc58c73f23358d:/sale:  User does not exist!
2009-12-09 11:43:30 Debug: 397f099790347716e0bc58c73f23358d:/sale:  redirecting to /login
2009-12-09 11:43:30 Debug: 0dfee15a4295b26aad115ae37d470d30:/login: User does not exist!
2009-12-09 11:43:30 Debug: 0dfee15a4295b26aad115ae37d470d30:/login: Success in auto login!
2009-12-09 11:43:30 Debug: 0dfee15a4295b26aad115ae37d470d30 /login: redirecting to /sale
2009-12-09 11:43:31 Debug: 3f23b7f7bead5d23fd006b6d91b1d195:/sale:  User does not exist!
2009-12-09 11:43:31 Debug: 3f23b7f7bead5d23fd006b6d91b1d195:/sale:  redirecting to /login
...

What I immediately noticed was that sessions were dropped at every redirect on the infinite redirect path. So I researched a bit and found the following resources:

• http://groups.google.com/group/cake-php/browse_thread/thread/4d7807465be56b03: A CakePHP google group message about lost sessions.
• http://book.cakephp.org/view/42/The-Configuration-Class: CakePHP documentation on the Security.level setting.
• http://www.php.net/manual/en/session.configuration.php#ini.session.referer-check: PHP documentation on referer_check.

As it turns out, the Security.level configuration affected the referer check for redirects. The CakePHP Session object set the referer_check to HTTP_HOST if Security.level was equal to 'high' or 'medium'. A couple of the resources mentioned above recommend to adjust the Security.level to 'low', which sounded like a potential solution. But I wasn't certain that this was the cause of the redirect, so I tested several changes to verify the problem.

First, I tested the Security.levels to 'high', 'medium', and 'low'. With the Security.level set to 'low', the infinite redirect would not happen and the debug log would show a consistent session id. Next, I commented out the code in the CakePHP Session object that set the referer_check and set the Security.level to 'high'. This also seemed to fix the infinite redirect, although, it wasn't ideal to make changes to the the core CakePHP code. Finally, I changed this->host to HTTPS_HOST instead of HTTP_HOST in the CakePHP Session object, so that the referer would be checked against the secure host rather than the non-secure host. This also fixed the infinite redirect, but again, it wasn't ideal to change the core CakePHP code.

I concluded that the secure redirect to the homepage or login page coupled with the auto login caused this infinite redirect. As pages were redirected between /login and /sale, the session (that stored the auto logged in user) was dropped since the referer check against HTTP_HOST failed.

In an ideal world, I would like to see HTTP_HOST and HTTPS_HOST included in the CakePHP referer check. But because we didn't want to edit the CakePHP core, I investigated the affect of changing the Security.level on the app:

I provided this information to the client and let them decide which scenario met their business needs. For this situation, I recommended commenting out the Security.level configuration so that the session timeout would stay the same, but the cookie lifetime and inactiveMins values would increase.

This was an interesting learning experience that helped me understand a bit more about how CakePHP handles sessions. It also gave me exposure to referer checks in PHP, which I haven't dealt with much in the past.

I am a person who lives in his emotions. There are many things that are trivial to some, but are deeply important to me. I have an emotional attachment to various digital objects. These can be in the form of a level or character in a computer game, a piece of music, a piece of art, even simple doodles an artist is quick to dismiss. These things are as important and meaningful to me as real friends and people because art is personality, and when you are struck by art, it is a connection to a person and an emotional state. Nostalgia is a lot like this, however I also feel nostalgia for things I am seeing for the first time.

So, getting to the point, I want to share with you with something I find very special to me, a comic produced by a young amateur artist as an outlet for experimentation, growth and just the sheer fun of it.

Zenith Cover © Kameira, 2009

Stunningly beautiful in moments and soberingly emotional in others, the least I could do is to try bring this comic to the attention of a wider audience. But know ye this, it is highly flawed, and that is key to why it so important to me. I consume tons of digital media all the time, and I don?t feel for all of it. That which is professional and well packaged and well delivered is easy to forget once you?re done with it. It fits neatly into your busy life.

Zenith is a record of a young artist?s life for the last four years. A complete journey into maturity as she slogs it through page after page of improving art, writing, characterisation and complete depth in the story. Zenith begins with naïve enthusiasm, improvising furiously with messy writing and odd pacing.

It is a patchwork of beauty and ugliness as her skill finds itself flung from one corner to another, going from rather unsure and rushed artwork to absolute gorgeousness, and from wacky writing to solid, captivating story telling?and back again.

Zenith Pg.4 © Kameira, 2007

Nobody has asked her to do this, nobody is paying her to work on this project, it was never started as a commercial venture?where there is some kind of financial goal to work towards. It?s just to scratch an itch, to get an idea out in the open. It is the same solitary effervescence that drives me and my website (it took me five years of repeat attempts to eventually succeed in releasing this site to the public).

Like many good projects, Zenith encapsulates an idea that is greater than the whole. Like the literal meaning of the world ?meme?, the comic has gotten to the stage where it defines a world beyond the confines of the comic alone. The comic is like an imperfect ?slice? through a world; just one of many possible angles by which to view it. For this, I can easily forgive Zenith?s flawed-by-it?s-nature approach, because I care about the characters, and I care what happens, sod the art; the fact the art is sometimes so captivating is icing to the cake.

Zenith Pg.63 © Kameira, 2009

You must forgive the awful prologue, in Kameira?s own words originally they were going to indicate Zenith?s massive love for stories, but as I went along (13 years old and attention span of a fly) I sort of forgot about that aspect. Now they lie here, dusty and useless. There?s always that gut feeling of trepidation when you start a new, large scale, project. Like you are stepping into the unknown and you have no idea how far you will have to travel, or if you will even make it. You start, fuelled almost entirely by enthusiasm and caffeine. But as things drag on?as weeks become months?it?s hard to keep at something that you never planned out properly to begin with. I?ve done this hundreds of times alone. But when months become years, that?s iron will. That?s sheer spirit pushing you along on the strength of the idea, even if the implementation is never as good as you want it.

There are many elegant aspects to the story, chapter 2 is really where it takes on its own form. The connection that?s drawn between Vethes and Aviel?these abusive personalities, one evil in intent the other intent on the greater good but the end result sounding much the same to Zenith?is so well played out. A father and mother figure drawn across time and space with Zenith caught in between. We have not seen Vethes in some while but my heart races to know what will happen when the two cross paths. This is better than The Lion King meets Finding Nemo.

Zenith Pg.46 © Kameira, 2008

So thank you Kameira. Thank you. Thank you for unfurling this story and keeping at it for these years, it has brought me great joy and I am hooked. Please, my visitors, give Zenith your time and consideration and please provide Kameira the much needed feedback and encouragement she deserves.

Kind Regards,
Kroc Camen.

Experimental music is not meant to be very accessible or well-understood, but there are some bands that do manage to be both fresh, and accessible. And there are others, who I just don’t get at all. Here are my own favorites and least favorites such bands. In parenthesis, find free, legal, promo mp3 downloads.

Best
1. Cloud Cult (”The Tornado Lessons“)
2. Blitzen Trapper (”Crushing The Wheat“)
3. Yeasayer (”2080“)
4. HEALTH (”Crimewave“)
5. Sin Fang Bous (”Catch The Light“)

Worst
1. Radian (”Git Cut Noise“)
2. Múm (”Illuminated“)
3. Dirty Projectors (”Knotty Pine“)
4. Deerhoof (”Offend Maggie“)
5. Beirut (”My Night With the Prostitute from Marseille“)

Ok, Beirut are listenable, but just barely.

After just 4 hours of overall shooting time, but 2 months of various setbacks, the music video I shot for local Bay Area artist Andy Kong, is finally up. Andy is a great singer/songwriter, so if you like the song, go ahead and check his music on iTunes!

Video was shot with a bare HV20 (just an ND filter was attached to it) in 24p, and in 60i for the slow-mo scenes. There are a few shots that I’d like them to be different, but overall I’m reasonably happy with the result. HD version here.

I had a 3-hour nap yesterday, and during that time I had the weirdest dream ever (although I’m known to have adventurous dreams). I saw some gangs that some of its members were hideous monsters, I saw my mom telling me that the little girl that’s part of one of these gangs was my twin sister that I never knew I had. While trying to free her (with… Adam Lambert’s help), I got chased and I had to swim away and fight the bad guys like a ninja.

By the time I got out of the water, the gang boss, none other than Samuel L. Jackson, took away my mother and my sister and he wouldn’t tell me where they were. Myself and some ex-gang members… tortured him, to no avail. Then, another monster comes in, and told us that Jackson has a secret place in his basement, a labyrinth. To get in and out of there without getting lost, you need to be accompanied by a kid that was a twin (and that was the reason he had kidnapped my supposed twin sister as a baby). I decided to go in.

It was an amazing place, and for the first time in a long time I did not realize that I was dreaming. It felt real. Some of it had places where you fall “up”, some of it had floors that would break apart and re-arrange itself, some of it had corridors with doors that monsters would come out and bite you, and the rest had a lot of stolen art, technology and what not. Even Adam Lambert was stashed there, and couldn’t find the exit. I asked him if he saw my mom and sister, he led me to them, and with my excitement for finding them, I woke up. I guess we’ll never know if I was able to lead these trapped people out.

John Gruber originally made available his script to Title Case text, working around the fringe-cases.

From this, a number of ports were made of the script of which particularly noteworthy David Gouch?s Javascript port that was smaller, simpler and handled more fringe cases.

I?ve ported this to PHP and put it to use on this site. My version is based on David Gouch?s Javascript port, unlike the WordPress port which is, frankly, crap. Ironically, now there?s a WordPress port that uses my port. The circle is complete! :P

Code below.

//original Title Case script © John Gruber <daringfireball.net>
//javascript port © David Gouch <individed.com>
//PHP port of the above by Kroc Camen <camendesign.com>

function titleCase ($title) {
	//remove HTML, storing it for later
	//       HTML elements to ignore    | tags  | entities
	$regx = '/<(code|var)[^>]*>.*?</1>|<[^>]+>|&S+;/';
	preg_match_all ($regx, $title, $html, PREG_OFFSET_CAPTURE);
	$title = preg_replace ($regx, '', $title);
	
	//find each word (including punctuation attached)
	preg_match_all ('/[wp{L}&`'??"?.@:/{([<>_]+-? */u', $title, $m1, PREG_OFFSET_CAPTURE);
	foreach ($m1[0] as &$m2) {
		//shorthand these- "match" and "index"
		list ($m, $i) = $m2;
		
		//correct offsets for multi-byte characters (`PREG_OFFSET_CAPTURE` returns *byte*-offset)
		//we fix this by recounting the text before the offset using multi-byte aware `strlen`
		$i = mb_strlen (substr ($title, 0, $i), 'UTF-8');
		
		//find words that should always be lowercase?
		//(never on the first word, and never if preceded by a colon)
		$m = $i>0 && mb_substr ($title, max (0, $i-2), 1, 'UTF-8') !== ':' && preg_match (
			'/^(a(nd?|s|t)?|b(ut|y)|en|for|i[fn]|o[fnr]|t(he|o)|vs?.?|via)[ -]/i', $m
		) ?	//?and convert them to lowercase
			mb_strtolower ($m, 'UTF-8')
			
		//else:	brackets and other wrappers
		: (	preg_match ('/['"_{([??]/u', mb_substr ($title, max (0, $i-1), 3, 'UTF-8'))
		?	//convert first letter within wrapper to uppercase
			mb_substr ($m, 0, 1, 'UTF-8').
			mb_strtoupper (mb_substr ($m, 1, 1, 'UTF-8'), 'UTF-8').
			mb_substr ($m, 2, mb_strlen ($m, 'UTF-8')-2, 'UTF-8')
			
		//else:	do not uppercase these cases
		: (	preg_match ('/[])}]/', mb_substr ($title, max (0, $i-1), 3, 'UTF-8')) ||
			preg_match ('/[A-Z]+|&|w+[._]w+/u', mb_substr ($m, 1, mb_strlen ($m, 'UTF-8')-1, 'UTF-8'))
		?	$m
			//if all else fails, then no more fringe-cases; uppercase the word
		:	mb_strtoupper (mb_substr ($m, 0, 1, 'UTF-8'), 'UTF-8').
			mb_substr ($m, 1, mb_strlen ($m, 'UTF-8'), 'UTF-8')
		));
		
		//resplice the title with the change (`substr_replace` is not multi-byte aware)
		$title = mb_substr ($title, 0, $i, 'UTF-8').$m.
			 mb_substr ($title, $i+mb_strlen ($m, 'UTF-8'), mb_strlen ($title, 'UTF-8'), 'UTF-8')
		;
	}
	
	//restore the HTML
	foreach ($html[0] as &$tag) $title = substr_replace ($title, $tag[0], $tag[1], 0);
	return $title;
}

Anything broken, please let me know.
Kind regards,

In the vein of the Phish Wishlist, here’s a list of my least favorite Phish songs.

There aren’t many Phish songs I don’t like… in fact, I don’t even dislike all of these, but these are the songs that I like least/want to hear least. Character Zero, for example, is a song I used to like, but now I’m just tired of it and hearing the opening riff at a show is always a bit of a downer until the song part is over and the jam part begins. Got any least favorite songs?

And don’t say Time Turns Elastic!

Kiel and I had a fun time tracking down a client's networking problem the other day. Their scp transfers from their application servers behind a Cisco PIX firewall failed after a few seconds, consistently, with a connection reset.

The problem was easily reproducible with packet sizes of 993 bytes or more, not just with TCP but also ICMP (bloated ping packets, generated with ping -s 993 $host). That raised the question of how this problem could go undetected for their heavy web traffic. We determined that their HTTP load balancer avoided the problem as it rewrote the packets for HTTP traffic on each side.

Kiel narrowed the connect resets down to iptables' state-tracking considering packets INVALID, not ESTABLISHED or RELATED as they should be.

Then he found via tcpdump that the problem was easily visible in scp connections when TCP window scaling adjustments were made by either side of the connection. We tried disabling window scaling but that didn't help.

We tried having iptables allow packets in state INVALID when they were also ESTABLISHED or RELATED, and that reduced the frequency of terminated connections, but still didn't eliminate them entirely. (And it was a kludge we weren't eager to keep in place anyway.)

We wanted to avoid some unpleasant possibilities: (1) turn off stateful firewalling or (2) perform risky updates or configuration changes on the Cisco PIX, which may or may not fix the problem, in the middle of the busy holiday ecommerce season.

Finally, Kiel found this netfilter mailing list post which describes how to enable a Linux kernel workaround for the mangled packets the Cisco generates:

echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal

Of course saving that in /etc/sysctl.conf so it persists after a reboot.

So we have reliable long-running scp connections with TCP window scaling working and iptables doing its job. I love it when a plan comes together.

Wow, just wow! A mixture of real footage and motion design, inspired by the look of Dutch ’70s postcards and class photos. Shot with a Canon HV20. HD version here.

As Steph noted, we recently embarked on an adventure with a client who had a legacy PHP app. The app was initially developed in rapid fashion, with changing business goals along the way. Some effort was made at the outset with this vanilla PHP app to put key business logic in classes, but as often happens over time the cleanliness of those classes degraded. While much of the business rules and state management (i.e. database manipulation, session wrangling, authentication/access-control, etc.) were kept separate from the "views" (the PHP entry pages), the classes themselves became tightly coupled, overburdened with myriad responsibilities, etc.

This was a far cry from the stereotypical spaghetti PHP app, but nevertheless it needed some reorganization; all but the smallest changes inevitably required touching a wide range of classes and pages, and the code would only grow more brittle unless some serious refactoring took place.

We determined at the outset that getting the application moved into an established MVC framework would be of great benefit, and further determined that CakePHP would be a good choice. (This is the point where anybody reading will inevitably ask in comments "Why CakePHP instead of My Preferred Awesome Framework?" Sigh.) The client agreed. The question became: how do we get there from here?

I spent some time investigating and inevitably came across the well-regarded three-part blog series:

1. Converting Legacy Apps to CakePHP Part I
2. Part II
3. Part III

For somebody new to MVC application design, especially in the PHP space, the series (and presumably the related book) probably makes for pretty good reading. They present a decent approach to how the refactoring of legacy code can be accomplished. However, the series also appears to operate under the assumption that you're in a scrap-and-rebuild situation: the legacy app can essentially go nowhere for a few weeks while it gets gutted into CakePHP.

As noted in a review of the related book, the rebuild-it-all assumption doesn't apply to many real world situations. The more money your application makes, the more users it affects, the larger the feature set, the more likely it is that the business cannot afford to have an application sit in a code freeze while an entire rewrite takes place.

We ultimately opted for a different approach: iteratively migrate to CakePHP. The simplicity of the basic PHP paradigm makes this remarkably easy.

The basic steps:

1. Rearrange the legacy application so it runs "within" CakePHP, with the CakePHP dispatcher handling the request but ultimately invoking the original legacy view
2. Make adjustments to the legacy code such that it gets its database handle(s) from CakePHP rather than internally, it uses CakePHP's session, etc.
3. New development can proceed within CakePHP; legacy logic can be refactored into CakePHP over time as the opportunity presents itself (or the situation demands)

Getting the application to run within CakePHP in this manner does not require that much effort. Of course, this would depend on your situation, but in the traditional model of presentation-oriented code relying on some business objects and a database, it works out. For the initial step:

1. Prepare a basic CakePHP application
2. Pull the legacy code into the CakePHP webroot, with the legacy pages moved under a new legacy/ subdirectory
3. Prepare a "legacy" action in the default PageController that maps the requested URI path to a path relative to the legacy/ directory, then invokes the file living at that path
4. Set up a new catch-all route that invokes this legacy action

    function includeLegacyPage($path = null) { 
        // map the path passed in or from the request to the legacy/ subdirectory
        $cakeRequestPath = $path ? $path : $this->controller->params['url']['url'];
        $path = WWW_ROOT . 'legacy/' . $cakeRequestPath;

        // This just maps input arguments to globals
        $this->prepareGlobals(array('cakeRequestPath' => $cakeRequestPath));

        // Resolve directories to an index.php page as necessary
        if(is_dir($path))
        {
            if(substr($path, -1) != '/') 
                $path .= '/'; 
            $path .= 'index.php';
        }

        if (!file_exists($path)) {
            $this->controller->render('error');
        }

        try {
            // buffer PHP output
            ob_start();

            // this "invokes" the legacy page and gathers its content
            include $path;

            // pull in the buffered content
            $this->controller->output = ob_get_contents();

            // stop output buffering
            ob_end_clean();
        } catch (JackExceptionRedirect $e) {
            // We adjusted the legacy app's redirect functions to throw a custom exception
            // class that we catch here, so we can use CakePHP's native redirection
            $this->controller->redirect($e->location, $e->getCode(), false); 
        } catch (Exception $e) {
            // All other errors propagate up
            throw $e;
        }

        $this->controller->autoRender = false;
        $this->controller->autoLayout = false;
    }
  

The second step, of getting CakePHP to control the session, the database handle, etc., involves some minor hacks. They don't feel elegant. They go outside the MVC pattern. But they provide the crucial glue necessary to put CakePHP in charge.

• Make the controller's session available from a global; adjust legacy code to use it instead of direct use of the PHP session. This means that CakePHP controls the session configuration.
• Make the CakePHP database handle available from a global as well; adjust your legacy database initialization code so it simply uses the global handle from CakePHP. Now CakePHP controls your database configuration, and CakePHP and the legacy code will use the same handle in a given request.
• And so on and so forth.

        App::import('ConnectionManager');
        $standard_globals = array(
            'cakeDbh'       => ConnectionManager::getDataSource('default')->connection,
            'cakeSession'   => $this->Session
        );

        $this->prepareGlobals($standard_globals);
  

Up until now, CakePHP's introduction into the mix hasn't added value. Having reached this point, however, you're ready to start taking advantage of CakePHP. From here, we refactored our special "legacy" action logic into a new "LegacyPage" component so any controller/action could use the mechanism. Then we were able to:

• Refactor legacy user authentication logic to use CakePHP's Auth core component
• Refactor various legacy pages to be fronted by CakePHP controller actions, moving the high-level flow control (input validation, user validation, and associated redirects) out of the legacy page and into the controller. This simplifies the legacy page (making it more strictly limited to presentation) and puts flow control where it belongs.
• For a new feature involving new data structures, developed a new CakePHP component to implement the business operations, new controllers/actions for aspects of the new functionality, and adjusted some legacy code to get data from the new component rather than original direct database calls or legacy class calls

So, what are the advantages of this approach, versus a slash-and-burn rewrite-it-all approach?

• We get to a point in which we're tangibly benefitting from CakePHP with minimal investment of time/money; contrast that with the potential expense of rewriting the entire application before the business sees any benefit
• While we proceeded in this work, the client was actively developing their legacy system; there was no need for a code freeze, and reconciling their changes with our work was fairly trivial; one Git rebase took care of it (though I admittedly missed a couple things during the rebase, which we caught and fixed with some spot-checking).
• No repeating of oneself: by making the entire legacy application available within the context of the target framework, we don't need to spend cycles rewriting existing functionality; the do-it-from-scratch approach would, by contrast, require reimplementation of everything
• We can refactor the legacy code in a prioritized, iterative fashion: refactor the most important stuff first, and the less important stuff later.
• We can partially refactor specific pieces of legacy code, such as removing business/data logic from pages such that legacy pages become more like views in the MVC triad; we're not forced to redo an entire legacy subsystem to improve the code organization
• The legacy work that is solid and doesn't need much refactoring stays put, and is usable from the rest of the CakePHP application

We may well get to the point (in late 2010, perhaps) when all legacy code has been refactored into CakePHP's MVC architecture. Or perhaps not: the business has to balance competing priorities, and it may ultimately be that some aspects of the legacy code just don't get refactored because they aren't especially broken and the business need simply doesn't come up. That's part of the beauty here: we don't have to make that decision right now; we can let the real-world priorities make that decision for us over time.

It's easy to imagine an engineer finding this less attractive than a redo-it-in-my-favorite-framework-du-jour approach. It reeks of compromise. Yet, from a business standpoint the advantages are hard to dispute. From a technical standpoint, they're hard to dispute as well: faster, shorter cycles of development bring a higher likelihood of success, particularly for small teams (or lone individuals); the management of change is much simpler with iterative design; the iterative approach is arguably less prone to second-system effect than is a rewrite; etc.

This asks more of the engineer than does a ground-up rewrite in Framework X. So many modern frameworks positively shine with possibility; the engineer lusts for the opportunity to Do It Right, and falls prey to the fallacy that the framework will solve all their problems given that Done Right investment. But, whatever the features and community offerings may be, modern frameworks ultimately help us organize our code better; better organization of code is amongst the most obvious benefits one gets in moving into a modern framework. The iterative approach gets us there with far less risk and, in many cases, far more naturally than does the rewrite-it-all approach, but it asks us to have the patience to move in small steps. It asks that we have the mental room and rigor to envision what the Done Right system might look like, as well as a long chain of interrim steps taking us from here to there. But it delivers value much faster, at lower risk, at lower cost, and crucially, reduces redundant work and gives us the opportunity to change direction as we go. Consequently, for many -- even most -- business situations the iterative transformation is the system Done Right.

A couple of months ago, I worked on an project for Survival International that required two dimensional product optioning for products. The shopping component of the site used Spree, an open source rails ecommerce project that End Point previously sponsored and continues to support. Because this open source project is quickly evolving, we wanted to implement a custom solution that would "stand the test of time" and work with new Spree releases. I worked with the existing data structures and functionality as much as possible. The product optioning implementation discussed in this article should translate to other ecommerce platforms as well.

Here's what I mean when I say "two dimensional product optioning".

The first step to extending the core ecommerce functionality was to understand the data model. A single product "has many" option types (size, color). An option type "has many" option values (size: small, medium, large). Each product also "has many" variants. Each variant was tied to an option value for each product option type. For example, each variant would requires a corresponding size and color option value in the example above. Ideally, each variant represents a unique size and color combination.

An *awesome* database dependency diagram.

Using the Spree demo data, I set up the Apache Baseball Jersey to have option types "PO_Size" and "PO_Color". PO_Size contains option values Red, Blue, and Green. PO_Color contains option values Small, Medium, and Large.

Variants assigned to the Apache Baseball Jersey

The second step to producing a two dimensional product option table was to generate the required data in a before_filter method in the controller. Below are the contents of the module that generates the hash in the before_filter method with color and size information. The module retrieves active variants first, then verifies that the required option types are tied to the product. Then, size, color, and variant ids are collected from the active variants producing the data structure described above.

  def self.included(target)
    target.class_eval do
      before_filter :define_2d_option_matrix, :only => :show
    end
  end
  def define_2d_option_matrix
    variants = Spree::Config[:show_zero_stock_products] ?
      object.variants.active.select { |a| !a.option_values.empty? } :
      object.variants.active.select { |a| !a.option_values.empty? && a.in_stock }
    return if variants.empty? ||
      object.option_types.select { |a| a.presentation == 'PO_Size' }.empty? ||
      object.option_types.select { |a| a.presentation == 'PO_Color' }.empty?
    variant_ids = Hash.new
    sizes = []
    colors = []
    variants.each do |variant|
      active_size = variant.option_values.select { |a| a.option_type.presentation == 'PO_Size' }.first
      active_color = variant.option_values.select { |a| a.option_type.presentation == 'PO_Color' }.first
      variant_ids[active_size.id.to_s + '_' + active_color.id.to_s] = variant.id
      sizes << active_size
      colors << active_color
    end
    size_sort = Hash['S', 0, 'M', 1, 'L', 2]
    @sc_matrix = { 'sizes' => sizes.sort_by { |s| size_sort[s.presentation] }.uniq,
 'colors' => colors.uniq,
 'variant_ids' => variant_ids }
  end

The code above produces a hash with three components:

• @sc_matrix['variant_ids']: a hash that maps size and color combinations to variant id
• @sc_matrix['sizes']: an array of sorted unique sizes of product variants
• @sc_matrix['colors']: an array of unique colors of product variants

In the view, the output of size and color arrays is used to generate a table. In this hardcoded view, sizes are displayed as the horizontal option across the top of the table, and colors as the vertical option along the left side of the table.

...
<% if @sc_matrix -%>
<p>Choose your colour, size and quantity below.</p>
<table id="option-matrix">
    <tr>
        <th></th>
        <% @sc_matrix['sizes'].each do |s| %>
        <th class="size"><%= s.presentation %></th>
        <td class="spacer"></td>
        <% end -%>
    </tr>
    <% @sc_matrix['colors'].each do |c| -%>
    <tr>
        <th class="color"><%= c.presentation %></th>
        <% @sc_matrix['sizes'].each do |s| -%>
        <td>  
            <% if @sc_matrix['variant_ids'][s.id.to_s + '_' + c.id.to_s] -%>
            <input type="radio" value="<%= @sc_matrix['variant_ids'][s.id.to_s + '_' + c.id.to_s] %>" name="products[<%= @product.id %>]" />
            <% else -%>
            <img src="/images/radio-notavailable.png" alt="X" width="20" height="20" />
            <% end -%>
        </td>
        <td class="spacer"></td>
        <% end -%>
    </tr>
    <% end -%>
</table>
<% elsif #check for other stuff
...

Here is a comparison of the current variant display method versus two dimensional variant display of the same product:

Current variant display method.

Two dimensional variant display method. Two variants shown here are out of stock.

And here is another example of two dimensional optioning in use at Survival International (more glamorous styling):

Spree extensions are similar to WordPress plugins or Drupal modules that do not typically require you to edit core code. The primary components of the extension are a module with the before_filter functionality and a custom view that overrides the core product view. An extension was created for this functionality and it lives at http://github.com/stephp/spree-product-options.

Possibilities for future work include editing the extension to be more robust by eliminating the use of the hard-coded option types of "PO_Size" and "PO_Color" and removing the hard-coded size ordering hash. It would be ideal to be able to assign the two dimension option types (horizontal axis and vertical axis) in the Spree admin for each product or a set of products. Another option for future work with this extension includes extending the functionality to multi-dimensional product optioning that would allow you to select more than two option types per product (for example: size, color, and material), but this functionality is more complex and may be dependent on Javascript to hide and show option types and values.

Learn more about End Point's rails development or rails shopping cart development.

As a consultant, I'm often called to make changes on production systems - sometimes in a hurry. One of my rules is to document all changes I make, no matter how small or unimportant they may seem. In addition to local notes, I always check in any files I change, or might change in the future, into version control. In the past, I would always use RCS. However, Jon Jensen challenged me to rethink my automatic use of RCS and give git a try for this.

This makes sense on some levels. We use git for everything here at End Point, and it is our preferred version control system. I still use other systems: there are some clients and projects that require the use of subversion, mercurial, and even cvs. The advantage of git for quick one off checkins is that, similar to RCS, there is no central repository, and setup is extremely easy.

As an example, one of the files I often check into version control is postgresql.conf, the main configuration file for the Postgres database. Before I even edit the file, I'll check it in, so the sequence of events looks like this:

mkdir RCS
ci -l postgresql.conf
edit postgresql.conf

The creation of the RCS directory is optional but recommended. RCS (which stands for Revision Control System) uses a very simple tracking mechanism. A new file that tracks all changes is created for each file. This new file takes the original name of the file and adds a ",v" to the end of it. However, it's annoying to have all those "comma vee" files laying around, so RCS has a nice trick that when a directory named RCS exists, all the comma vee files will be placed into that directory. The "ci -l postgresql.conf" checks in (ci) the file, and the "-l" file instructs RCS to immediately check it back out again and lock it (as the current user). This is an RCS specific advisory lock, and only gets in the way if you try to check in the file as a different user. The final command above, "edit postgresql.conf" calls up my editor of choice so I can start modifying the file.

Once the file has been modified, checking in the changes made is as simple as once again doing:

ci -l postgresql.conf

Now that it has been checked in, I can perform other common version control tasks against it. To see the complete log of changes:

rlog postgresql.conf

To see the differences between the current version and the last checkin, or against a specific version:

rcsdiff postgresql.conf
rcsdiff -r1.3 postgresql.conf

To find a string in a specific previous version:

co -p -r1.3 postgresql.conf | grep foobar

Using git for this purpose is fairly similar. The first steps now become:

git init
git add postgresql.conf
git commit postgresql.conf
edit postgresql.conf

Technically, one more step than before, but not really a big deal. Note that we don't need to create a special directory to hold the versioning information: by default, git puts everything in a ".git" directory. Once we've made changes to the file, we can commit out changes with:

git commit postgresql.conf

to see the log of changes:

git log postgresql.conf

To see the differences between the current version and the last checkin, or against a specific version:

git diff postgresql.conf
git diff 11a049bc80fe4a2f4584465fe13d8bb4ee479f23 postgresql.conf

To find a string in a specific previous version:

git show 11a049bc80fe4a2f4584465fe13d8bb4ee479f23:postgresql.conf | grep foobar

With git, there is also quite a bit more than an be done now - easy branching, grepping, generating diffs, etc. However, most of it is overkill for the simple purpose of tracking local changes. On the downside, git does not have the simple version numbering that RCS has, and the syntax can be a bit trickier and non-intuitive.

So, did I make the switch? Well, yes and no. I've been trying to use git for simple checkins the last few weeks, and have had mixed results. Here's my breakdown of areas in which they differ:

Ease of use

RCS wins this one. All you really need to remember to use RCS is "ci -l filename". The only other commands you might possibly need is "rlog filename" and "rcsdiff filename". On the other hand, git requires a deeper understanding of objects, trees, add vs. commit, and the use of long, hard to type hexadecimal numbers. It's also not very intuitive, and the command arguments can be complex. To be fair, for this *particular* use case git is not really that much more complex, but the advantage still goes to RCS.

Availability

RCS wins this one as well. On many systems, RCS is already installed by default. Even when it is not, a "yum install rcs" or the equivalent works just fine 100% of the time. RCS has been around a long, long time, and it's solid, tested, and very available on any system you run into. In contrast, git is fairly new, does *not* come pre-installed on most systems, and is not even available via all packaging systems. This is one factor that would definitely prevent me from using it everywhere. Maybe years from now when it is a standard tool, this will change, but for now, RCS wins this one.

Diffs

The rcsdiff command is handy, but very limited. If all you want is the simplest of bare-bones diffs, all is good. However, git allows you to view diffs in different formats, add color, generate patches, and many other features that can be nice to have.

Fancy tricks

RCS is designed to be dirt simple and good at what it does: track single files. The design of git was for a large, distributed project with complex needs. This means that git has many tricks and features that the designers of RCS did not even dream of. While most of them are not needed when you are simply doing versioning of local files, there are definitely times when the full power of git is nice to have.

Grouping

RCS has no concept of projects or trees: everything is simply a file. This means that you cannot track relationships between files. The only possible way to do so is to compare the timestamps that two files were checked in. In contrast, git does not consider files at all, but simply treats everything as objects in a tree. This allows easy grouping of files together in a single logical commit. It also allows for things such as branching and merging.

Versioning

While git uses SHA1 checksums to name each object with a unique identity, RCS simply uses a "single dot" version number, and increments it for you. Thus, the first time you check in a file, it is set as version 1.1. The second version is 1.2, and so on. This is very useful when you are simply tracking a lone file - you know that version 1.20 is the 20th recorded change, and that comparing or viewing an earlier version is as simple as using the "-r x.y" option. Calling what git does "versioning" is somewhat of a misnomer - it has a completely different philosophy about how objects are tracked, which lends itself great to distributed and collaborative projects, but not so well to single files.

Blame

Here's one area where git wins hands down. For RCS, you do a checkin, and the file is locked as the current local user. There is no indication of the actual *person* doing the checkin (as opposed to the account name), unless you add it to the checkin comment each time, and that gets laborious and annoying. With git, you can set some standard environment variables (even on a shared account), and git will record who made the change. Not only can you see who made each commit and when, but you can use the awesome "git blame" command to view who made the last change to each line in a file.

As an aside, how do we do the assignment mentioned above in a shared account? Setting the author for git commits is as simple as setting environment variables like so:

$ export GIT_AUTHOR_NAME="Greg Sabino Mullane"
$ export GIT_AUTHOR_EMAIL="greg@endpoint.com"

On a shared account, just create an alias. For example:

cat > .gregs_stuff
export GIT_AUTHOR_NAME="Greg Sabino Mullane"
export GIT_AUTHOR_EMAIL="greg@endpoint.com"

cat >> .bashrc
alias greg='source ~/.gregs_stuff'

Editor support

One of the nice things about RCS is that it has been around for so long that many editors have integrated support for it. For example, calling up a file in emacs that has been checked in via RCS shows a display in the status line at the bottom of the screen showing that the file is controlled by RCS, what the current version number is, whether it is locked or not. While there is git support as well, it's only available in very new versions of emacs (and other editors). Advantage, RCS

Bloat

Because git is a real version control system, and a complicated one at that, it carries a lot of setup baggage. Just creating a repository and checking in a single file creates about 37 files underneath the .git directory. This number grows sharply with every commit you do. By contrast, RCS creates a single file (and one additional for each file you track). This means you can easily ship around the "dot vee" files to other systems.

Final analysis

When looking at all the factors, RCS still wins. It's simple, gets the job done, and most important of all, is available on all systems. I may revisit this in a few years when git is more widespread.

Recently, Ron, Ethan, and I worked on a JackThreads project. We are in the process of moving JackThreads' legacy PHP application to the CakePHP framework in addition to introducing new functionality for this project.

Several of the pages require secure requests:

• the home page (where users log in or create accounts)
• the login page
• the "invite" page (where users create an account)
• the checkout page

We referred to this article that discusses using the security component in CakePHP. Although this article covered the basics, we extended the concepts of the article by creating a CakePHP component with the custom security functionality to force a secure request and includes query string parameters. Below are the contents of the component that was created:

class StephsSecurityComponent extends Object {
    var $components = array('Security');
    function forceSecure($args) {
        $this->Security->blackHoleCallback = 'forceSSL';
        $this->Security->requireSecure($args);
    }
    function forceSSL($controller) {
        $redirect_location = 'https://'.HTTPS_HOST.$controller->here;
        $params = $controller->params['url'];
        unset($params['url']);
        if(count($params) > 0)
        {
            $param_string = '';
            foreach($params as $key => $value) 
                $param_string .= '&'.$key.'='.$value;
            $param_string = preg_replace('/^&/', '?', $param_string);
            $redirect_location .= $param_string;
        }
        $controller->redirect($redirect_location);
    }
}

This design required the following definition in the application's app_controller:

function forceSSL() {
    $this->StephsSecurity->forceSSL($this);
}

And any controller that required an action to be secure would call the forceSecure function in the beforeFilter:

function beforeFilter() {
    $this->StephsSecurity->forceSecure('my_action');
}

For the most part, the security redirect worked as expected. The before filter in each controller correctly registered the action that required a secure request, and logging statements in the CakePHP core security component verified that the secure component would call the blackHoleCallback if the request was not secure. But then, we came across a bug!

One of the controllers that included this new functionality was not working as expected. The controller had two actions; both actions accepted inputs from forms and did stuff with those forms, only one of the actions required the force secure, one of the actions received form inputs from the CakePHP form helper and the other action received inputs from a legacy PHP page. The action that received inputs from a legacy PHP page didn't do stuff correctly. Below is a simplified version of this controller:

class ThisController extends AppController {
   ...
    var $uses = array('Security', 'StephsSecurity');
    function beforeFilter() {
        $this->StephsSecurity->forceSecure('action_one');
    }
    function action_one() {
        //receives inputs from a cakephp form helper
        //do stuff with $this->params
    }
    function action_two() {
        //receives inputs from a legacy php page
        //do stuff with $this->params -- FAIL
    }
}

We added debugging and found that $this->params (or the form parameters) to action_two was empty. We added logging to the beforeFilter to determine if the parameters were deleted during the beforeFilter process. We found that the parameters were present at the conclusion of the beforeFilter. So, at some point in between the beforeFilter and before the action, our form parameters were deleted.

function beforeFilter() {
    $this->log($this->params, LOG_DEBUG);
    //some other unrelated before filtering
    $this->log($this->params, LOG_DEBUG);
    $this->StephsSecurity->forceSecure('action_one');
    $this->log($this->params, LOG_DEBUG);   //parameters looked ok here!
}

After more troubleshooting, we determined that if the CakePHP core Security component wasn't included in the controller, the parameters were not deleted and the action did it's stuff. A review of the CakePHP core Security component revealed that the component performs a validation on posts, which includes a check for a Token input. Because the post to this action originated from a legacy PHP page, it did not include any special hidden form variables included with the use of the CakePHP form helper (much like the Token inputs included via the Rails form helper):

<input type="hidden" value="POST" name="_method"/>
<input type="hidden" id="Token123123123 value="123123123131231231223" name="data[_Token][key]"/>

As a result, the black hole security redirect was called before action_two was reached, then action_two was called with missing parameters. Ethan realized there was a simple fix to this post validation failure. The Security->validatePost variable was set to false inside the controller's beforeFilter to bypass the _validatePost check in the security component. No more post validation produced expected action_two behavior.

function beforeFilter() {
    $this->Security->validatePost = false;
    $this->StephsSecurity->forceSecure('index');
}

Unfortunately, there isn't a lot of documentation on the CakePHP Security component that would have helped us identify this issue quickly. Configuration of the CakePHP Security component, discussed here, fails to mention the validatePost value, but it is included in the CakePHP API documentation.

Fortunately, it wasn't too difficult to troubleshoot once we observed the undesired behavior originated from the inclusion of the Security component in the controller. We are now aware of this Security post validation as we continue to transition legacy PHP to CakePHP. I'm sure we'll come across situations where data is passed from legacy pages or 3rd party services that do not contain the required Token variables and will require bypassing the _validatePost check.

Given the recent setlist madness, I decided to compile my Phish Wishlist.  Here are the 13 songs I most want to hear played live, in no particular order:

The number following each song is the average show gap between performances since the debut.  As you can see, given the number of shows I currently attend each year and the number I expect to attend in the next few years, it’s increasingly unlikely that I will see… well… ANY of these songs live, ever.  With each passing show, many of these number are increasing just a touch to the right of the decimal point, and the odds I actually see them go down inversely.  Even Lizards, which is still really low, is deceivingly so, given that it was so overplayed in the “old days” and underplayed these days.  In fact, ZZYZX’s stats say the odds of me not seeing Lizards in 35 shows is 0.0%.

I realize that with NYE being my only remaining show this year, any of these showing up is unlikely, because Phish has a history, for several years now, of not  going too crazy on NYE, but rather, doing that in the nights leading up to NYE.  Expect a Harpua on 12/30, and another set of standards for NYE.   The most likely candidates to show up on NYE? I’d have to bet on Lizards, Scents and Subtle Sounds, or Dinner and a Movie before any of the others.  But I’m expecting none of them.  Sigh.

* Edit: Add this song after the fact
Anyway, know that if I catch any of my wishlist on NYE, I will go nuts.

There’s this idea on the internets the last few years that if you’re a musician you must give your music for free, and then try to make money off of live performances and special packages for fans who are collectors. I personally don’t share this idea. I don’t believe that anyone can be a Radiohead or NiN. These bands are already established with a known number of fanatic fans who would buy anything. But the reality is that for the 99.99% of the rest of the artists, this won’t work. I, and anyone I know, would never buy collector’s items. It’s not our style, I guess. We follow a musician for his music only (ok, and for those dreamy eyes).

Live performances only bring so much money too, and all we know how bad music sales are these days. So, you ask, how can the music industry survive? And my answer is: it can’t, and it won’t. Why is it so hard to acknowledge that times changed, and no matter what, only a handful of artists will make it big, and the rest should keep their backup jobs? The age of rock’n'roll and Hollywood glam are over. Just like being a weldor was a cool job back in the Middle Ages, and it’s not anymore. Times changed.

Just the other day I was reading on BBC what journalists and music industry specialists suggested that should be done in order to save the music industry. They suggested from subscription streaming, to universal licensing, to anything else you can think of. And guess what: none of this will work. Nada. Reasons being: 1. Over-saturation of the market, 2. There’s already enough legally free music out there, 3. Streaming a problem in a non-100% internet-connected world, 4. Piracy.

In my opinion, trying to be a musician that can sustain a family financially, is a very difficult thing to do, and it’s only going to be more difficult. And if you happen to have a drug habit, well, good luck with your cheapo MacDonalds daily diet and still look young & sexy in your ’30s. However, if you still feel that you want to give it a try in that industry, here are my suggestions.

Production

1. Experiment with new sounds, new instruments. If you’re just another 1 guitar, 1 bass, 1 drums, 1 vocals band, well, good luck with that. Oh, and stop seeing rock with keyboards as “pop/disco” (and therefore “bad music”). Keyboards is just a tool, it can be very flexible sound-wise, and so use that to your advantage. On the other side of the spectrum, violins are still cool.

2. Chances are you’re not a new Elvis or Nirvana. Therefore, you’ll have to play within the constraints of the current market. This means that you will have to write music people WANT to hear — even if you don’t. Oh, shut up already with your “I won’t sacrifice my artistic integrity” bullshit. Do you think Leonardo daVinci only created the stuff he wanted to create, or did he also got side jobs for governors and the church, and had to abide within the rules and needs of these employers? Because, he totally did, and you’re no different. Today’s “employer” are the consumers. And today’s consumers are so overrun by their hectic life, that they simply have no time to decode your experimental/avant-garde/whatever-weird-shit you’re writing. They need HOOKS, melodies that their brain can hold on to after a SINGLE LISTEN. The rule of thumb here is this: if you can imagine your neighbor being able to sing your song in his shower, then you have a marketable song. If not, go back to drawing board and rewrite it. Now, I am not saying that you should only write “pop” music (by “pop” I mean “easily understood”, it could be any genre, including heavy metal or punk — more explanation on all this in the comment section below). But you need to write such music in order to make a BUCK, so you can then produce the music you REALLY want to make available (e.g. Blitzen Trapper took off when their last two albums became more accessible musically). You can do this via two ways: have 3 out of the 10 songs on your album being the music you really want to make, and poll your customers what they thought about it. The other way is to give away your not-very-commercial music via your web site, and then see the reactions of your fans. If that kind of music sells, consider it for your next album. If not, keep making “pop” music, write the music you want on the side, and give it away for free, and await for another 20 years until it gets appreciated. If you’re not willing to do that, then I wouldn’t consider you a wise professional. There’s no shame in making a living. The shame only comes when you are a musical fanatic, a “purist”. Nothing good ever came out of fanaticism.

3. Learn how to use recording software, e.g. Logic, Pro Tools etc. It is absolutely possible with today’s equipment to record in very high quality on your own home. You will only need professional mixing and mastering to be done by others.

4. Design your own artwork. If you’re a true artist, it doesn’t matter if you’re a musician and not a painter or a Photoshoper. Let it come out. Improvise. Or, ask your fans to do art for you.

Business aspect

1. Avoid label contracts (including with indie labels), unless you’re 100% you are getting a good deal. Majors never offer a good deal btw, avoid them like the plague.

2. Avoid managers. If you need to be told what to do then this is not the right profession for you. Being a professional musician today means more than just writing music. If you’re only interested in writing/play music, then keep a day job, and play music on the weekends. Do employ a manager/helper when you become too successful and can’t take care of the daily business all by yourself anymore.

3. Hire a PR company to do specifically the TV/magazines/radio promotion for you (there are 5-6 good ones in the US), a live-show booking company, a licensing company, a distribution company (e.g. CDBaby, who will also get you to Amazon, iTunes, Spotify etc). If you’re successful, also get legal counsel.

Your side of promotion, online

1. Send ONE mp3 to music blogs from your new album. IF you can give away more, give away up to three mp3s from your album, but there has to be a 1-2 months of space between the freebies. Otherwise, overloading listeners won’t work well, and you might give the wrong impression that your music doesn’t worth much to give it away so easily. It’s a bit of a mental game. You will have to keep listeners think “oh, I remember these guys, they had this other mp3 a couple of months back“. Very important: always tag your mp3s properly, including with album art.

2. You must spend time to find the top-100 music blogs out there to send your info, mp3s for reviews or for freebie promotional reasons. The top-5 such blogs will probably ignore you, but the rest 95 probably won’t. If the top-5 can get you 1000 new fans, and the rest 95 combined can get you 2000 new fans, it’s still a good plan despite the extra work. As for the top-5, that’s why you hired the PR company suggested above.

3. While commercial FM radio might be inaccessible to indie artists (even with a PR company is difficult), college & internet radio are not. Send your free CDs or mp3s to these radios. There are hundred of thousands of listeners in these radios these days. This should also include services like Pandora and Last.fm which are not internet radios with the normal sense of the word.

4. TV is also inaccessible for indie artists, but Youtube/Vimeo are not. And I am not talking here about just shooting a music video or capturing a live performance. Instead, contact top amateur videographers and ask them to make videos off of your songs. The band Barcelona became more known in the last few months just because a videographer used their song for his video (currently, the most “liked” HD video on Vimeo ever). The interesting thing here is that this is NOT the best song off of the album, and yet, AFTER that video became popular, that song became their No1 sale on iTunes. One big thing that ticks the music industry is the unauthorized usage of RIAA music by amateurs. This is a major point that the indie artist should use against his major artists’ competition! If they don’t allow people to use their music with their random non-commercial videos, then the indie artist should! All you need to ask is for attribution, so your new fans will know who plays that song.

5. Always maintain at least a Facebook, Myspace, and Twitter account. Twitter must be updated regularly. It’s very important to actually reply to your fans there (I personally unfollow bands that don’t reply to fans). Then there’s Imeem, purevolume, hypem, ilike etc. You can use a web site like ArtistData to manage them all at once. And of course, have a main band web site, that uses NO Flash. Flash takes ages to load and it doesn’t work with mobile phones. Keep it simple, and accessible. Also blog.

6. Do shoot 2-3 official music videos from your album. There are amateur videographers out there in your area that do want to shoot your official video, for free, or for very low cost. They get to brag that they shoot real music videos for artists, and you get a music video. And if the outcome is not stellar, it’s still better than nothing. Upload to both Youtube and Vimeo.

7. Sign up for sessions with Daytrotter or HearYa, then make these sessions known to your fans so they can download them for free. Just the other day I bought the EP of ‘Magic Wands‘ after I heard them for the first time at Daytrotter.

8. Tell your label or your PR company to give away for free 1-2 mp3s to mp3 manufacturers. If you can get 1 of your mp3s onto the Sandisk players (which usually come with a few mp3s for free), or an Android or Nokia phone, then you will get *millions* of new listeners, for free. Bay Area’s Loquat made a name for themselves by giving away their single via the Sandisk players.

9. For the top-5 or top-10 of the music blogs/mags, along the CD send an actual mp3 player with your top-3 songs in there, asking for a review (and mentioning which one of the 3 songs is to be given as a freebie for the blog/mag’s audience). See, if you send just a CD, no one will rip it, it will be ignored. If you send an attached mp3 on an email, it will get wiped out by their firewall. If you send a link to an mp3, most probably it will be ignored too. But by sending an actual mp3 player, no one will say “no” to a free gadget. They will feel compelled and *obligated* to listen to what’s in there. These days, you can buy a cheap 512 MB mp3 player (which is way bigger in storage than what you need anyway) for $10. Geeks.com had some a few weeks ago for $7+tax+shipping, they are out of them now. But they might still be available elsewhere.

10. Do shows. A lot of shows. And don’t shun the rural America, since most bands don’t go there, and so you can get new hungry fans there, while you least expected it to. Not everyone’s listening to country over there.

And finally: look good. The ladeez like seeing good looking men up on the stage. Sorry mate, part of the job too.