All opinions expressed are those of the authors and not necessarily those of, our sponsors, or our affiliates.
  Add to My Yahoo!  Subscribe with Bloglines  Subscribe in NewsGator Online

published by (Greg Sabino Mullane) on 2015-05-18 12:00:00 in the "database" category

Determining if your current database connection is using PgBouncer, or going directly to Postgres itself, can be challenging, as PgBouncer is a very low-level, transparent interface. It is possible, and here are some detection methods you can use.

This was inspired by someone asking on the Perl DBD IRC channel if it was possible to easily tell if your current database handle (usually "$dbh") is connected to PgBouncer or not. Since I've seen this question asked in other venues, I decided to take a crack at it.

There are actually two questions to be answered: (1) are we connected to PgBouncer, and if so, (2) what pool_mode is being run? The quickest and easiest way I found to answer the first question is to try and connect to a non-existent database. Normally, this is a FATAL message, as seen here:

$ psql testdb -p 5432
testdb=# c ghostdb
FATAL:  database "ghostdb" does not exist
Previous connection kept

However, a slightly different ERROR message is returned if the same thing is attempted while connected to PgBouncer:

$ psql testdb -p 6432
testdb=# c ghostdb
ERROR:  No such database: ghostdb
Previous connection kept

Thus, an ERROR will always indicate that you are connected to PgBouncer and not directly to Postgres, which will always issue a FATAL.

In the future, there will be an even simpler method. As of this writing, pgBouncer 1.6 has not been released, but it will have the ability to customize the application_name. This is a configurable session-level variable that is fairly new in Postgres. Andrew Dunstan wrote a patch which enables adding this to your pgbouncer.ini file:

application_name_add_host = 1

This will make PgBouncer modify the application_name to append some information to it such as the remote host, the remote port, and the local port. This is a feature many PgBouncer users will appreciate, as it offers an escape from the black hole of connection information that PgBouncer suffers from. Here is what it looks like on both a normal Postgres connection, and a PgBouncer connection. As you can see, this is an easier check than the "invalid database connection" check above:

## Postgres:
$ psql testdb -p 5432 -c 'show application_name'

## PgBouncer:
$ psql testdb -p 6432 -c 'show application_name'
 psql - unix(

## DBD::Pg connections to PgBouncer get a very similar change:
$ perl --port 6432
app - unix(

Now we have answered question of "are we connected to PgBouncer or not?". The next question is which pool mode we are in. There are three pool modes you can set for PgBouncer, which controls when your particular connection is returned to "the pool". For "session" mode, you keep the same Postgres backend the entire time you are connected. For "transaction", you keep the same Postgres backend until the end of a transaction. For "statement", you may get a new Postgres backend after each statement.

First, we can check if we are connected to PgBouncer in a statement level pool mode by taking advantage of the fact that multi-statement transactions are prohibited. PgBouncer enforces this by intercepting any attempts to enter a transaction (e.g. by issuing a BEGIN command). A very PgBouncer specific error about "Long transactions not allowed" is issued back to the client like so:

$ psql testdb -p 6432
testdb=# begin;
ERROR:  Long transactions not allowed

So, that takes care of detecting a pool_mode set to 'statement'. The other two modes, transaction and session, will *not* give the same error. Thus, seeing that error indicates you are using a statement-level PgBouncer connection.

The next pool mode is "transaction", which means that the server connection if released back to the pool at the end of a transaction. To figure out if we are in this mode, we take advantage of the fact that PgBouncer can be set to clean up the connection at the end of each transaction by issuing a specific command. By default, the command set by server_reset_query is DISCARD ALL, which invalidates any prepared statements, temporary tables, and other transation-spanning, session-level items. Thus, our test will see if these session-level artifacts get discarded or not:

## Direct Postgres:
$ psql testdb -p 5432
testdb=# prepare abc(int) as select $1::text;
testdb=# execute abc(1);

## PgBouncer:
$ psql testdb -p 6432
testdb=# prepare abc(int) as select $1::text;
testdb=# execute abc(1);
ERROR:  prepared statement "abc" does not exist

Keep in mind that there are no true "transactionless" commands in Postgres. Even though we did not use a BEGIN in the psql prompt above, each command is treated as its own mini-transaction. In the case of the PgBouncer connection, the prepare is immediately followed with a DISCARD ALL, which means that our prepared statement no longer exists. Hence, we have determined that we are using a transaction-level PgBouncer connection.

Unfortunately, not getting an error does not necessarily mean your PgBouncer is NOT in transaction mode! It could be that server_reset_query is empty, meaning that temporary artifacts are not discarded at the end of the transaction. In such a case, we can take advantage of the fact that PgBouncer will allow other clients to share in our current connection, and thus be able to see the temporary items. If we create a temporary table in one pgbouncer connection, then connect again as a new client, the temporary table will only show up if we are sharing sessions but not transactions. Easier shown than explained, I suspect:

## Regular Postgres gets a fresh session:
$ psql test1 -p 5432
test1=# create temp table abc(a int);
test1=# select * from abc;
(No rows)
test1=# ^Z ## (we suspend with CTRL-Z)
[2]+  Stopped                 psql test1 -p 5432

$ psql test1 -p 5432
test1=# select * from abc;
ERROR:  relation "abc" does not exist

## PgBouncer will re-use the same session:
$ psql test1 -p 6432
test1=# create temp table abc(a int);
test1=# select * from abc;
(No rows)
test1=# ^Z
[2]+  Stopped                 psql test1 -p 6432

$ psql test1 -p 6432
test1=# select * from abc;
(No rows)

The final PgBouncer pool mode is "session", and basically means the only advantage over a normal Postgres connection is the overhead to start up and connect to a new Postgres backend. Thus, the PgBouncer connections are only returned to the pool upon disconnection. The only way to tell if you are in this mode is by determining that you are *not* in the other two modes. :)

So, although PgBouncer is extremely transparent, there are some tricks to determine if you are connected to it, and at what pool_mode. If you can think of other (SQL-level!) ways to check, please let me know in the comments section.

published by (Jon Jensen) on 2015-05-13 05:05:00 in the "company" category

We are looking for another talented software developer to consult with our clients and develop their web applications in AngularJS, Node.js, Ruby on Rails, and other technologies. If you like to focus on solving business problems and can take responsibility for getting a job done well without intensive oversight, please read on!

What is in it for you?

  • Flexible full-time work hours
  • Health insurance benefit
  • Paid holidays and vacation
  • 401(k) retirement savings plan (U.S. employees)
  • Annual bonus opportunity
  • Ability to move without being tied to your job location

What you will be doing:

  • Work from your home office, or from our offices in New York City and the Tennessee Tri-Cities area
  • Consult with clients to determine their web application needs
  • Build, test, release, and maintain web applications for our clients
  • Work with open source tools and contribute back as opportunity arises
  • Use your desktop platform of choice: Linux, Mac OS X, Windows
  • Learn and put to use new technologies
  • Direct much of your own work

What you will need:

  • Professional experience building reliable server-side apps in Ruby on Rails, Node.js and Express, Django, CakePHP, etc.
  • Good front-end web skills with responsive design using HTML, CSS, and JavaScript, including jQuery, AngularJS, Backbone.js, Ember.js, etc.
  • Experience with databases such as PostgreSQL, MySQL, MongoDB, CouchDB, Redis, Elasticsearch, etc.
  • A focus on needs of our clients and their users
  • Strong verbal and written communication skills

About us

End Point is a 19-year-old web consulting company based in New York City, with 39 full-time employees working mostly remotely from home offices. We are experts in web development, database, and DevOps, collaborating using ssh, Screen/tmux, IRC, Google+ Hangouts, Skype, and good old phones.

We serve over 200 clients ranging from small family businesses to large corporations. We use free software frameworks in a variety of languages including JavaScript, Ruby, Python, Perl, PHP, and Java, tracked by Git, running on Linux.

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of gender, race, religion, color, national origin, sexual orientation, age, marital status, veteran status, or disability status.

How to apply

Please email us an introduction to to apply. Include a resume, your GitHub or LinkedIn URLs, or whatever else that would help us get to know you. We look forward to hearing from you! Full-time employment seekers only, please. No agencies or subcontractors.

published by (Josh Ausborne) on 2015-05-12 14:15:00 in the "Google Earth" category
The availability to activate layers within Google Earth is one of the things that makes Earth so powerful. In fact, there are many standard layers that are built into Earth, including weather, roads, place names, etc. There are also some additional layers that have some really interesting information, including one I noticed relatively recently called "Appalachian Mountaintop Removal" which is interesting to me now that I live in Tennessee.

As you can see, however, that while some of these available layers are interesting on a desktop, they're not necessarily very visually appealing on a Liquid Galaxy. We have identified a standard set of layers to enable and disable within Earth so that things don't appear too cluttered while running. Some things we've disabled by default are the weather and the roads, as well as many levels of place names and boundaries. For example, we have boundaries of countries and water bodies enabled, but don't want lines drawn for states, provinces, counties, or other areas such as those.

To disable these layers, we modify the GECommonSettings.conf file on the machines that running Earth. This file has everything pretty well spelled out in a readable manner, and it's fairly easy to determine what layers you're enabling or disabling.

Here's an example of some of the entries contained within the GECommonSettings.conf file:

See, that's pretty self-explanatory, isn't it?

Well, it is until you start introducing other languages to the mix. For example, we needed to run Earth in Korean language mode for one of our clients. Once we fired up Earth and specified that the language would be Korean, Earth came up with all of the default layers turned on. All of those layers that we'd disabled in English Earth stayed on and cluttered the displays with so many icons for each and every layer.

It took some trial and error, but I was eventually able to figure out what to do to resolve this. I loaded Google Earth in Korean mode on my Ubuntu VM, made my changes to the selected layers via the Layers selection area within Earth, then quit. When I looked at the GECommonSettings.conf file after quitting Earth, I found a bunch of new line items added to the file. It seems that each of the options had new lines, though I couldn't exactly decipher which lines controlled which options.

Here's an example of some of the new entries that are now contained within my GECommonSettings.conf file:

Now, I'll be honest and say that I don't have clue exactly what %UAD6D%UACBD and 1%UCC28%20%UD589%UC815%20%UACBD%UACC4%UC120%28%UC8FC_%UB3C4%29 mean, but I really don't have to know. All I know is that they got disabled when I disabled the undesired layers within Earth. I then copied these lines to the configuration on my Liquid Galaxy, and the next time I fired it up in Korean, the layers were no longer cluttering up the displays.

I was able to use this exact same method to determine which layers to enable or disable for one of our Spanish-language clients, as well.

published by (Kirk Harr) on 2015-04-29 11:30:00 in the "python" category


Scapy is a Python-based packet manipulation tool which has a number of useful features for those looking to perform raw TCP/IP requests and analysis. To get Scapy installed in your environment the best options are to either build from the distributed zip of the current version, or there are also some pre-built packages for Red Hat and Debian derived linux OS.

Using Scapy

When getting started with Scapy, it's useful to start to understand how all the aspects of the connection get encapsulated into the Python syntax. Here is an example of creating a simple IP request:

Welcome to Scapy (2.2.0)
>>> a=IP(ttl=10)
>>> a
<IP  ttl=10 |>
>>> a.dst=""
>>> a
<IP  ttl=10 dst= |>
>>> a.src
>>> a.ttl

In this case I created a single request which was point from one host on my network to the default gateway on the same network. Scapy will allow the capability to create any TCP/IP request in raw form. There are a huge number of possible options for Scapy that can be applied, as well as huge number of possible packet types defined. The documentation with these options and packet types is available on the main site for Scapy.

Creating custom scripts with Scapy

Using Scapy within Python rather than as a standalone application would allow for creating more complex packets, sending them, and then parsing the response that is given. Here is a simple tester script example in which I will initiate a HTTP 1.1 request:

#! /usr/bin/env python
import logging

from scapy.all import *

def make_test(x,y):
    request = "GET / HTTP/1.1rnHost: " + y  + "rn"
    p = IP(dst=x)/TCP()/request
    out = sr1(p)
    if out:
if __name__ == "__main__":
    interact(mydict=globals(), mybanner="Scapy HTTP Tester")

Within this script there is the make_test function which takes as parameters the destination address and host header string respectively. The script will attempt to send the HTTP GET request to that address with the proper Host header set. If the request is successful, it will print out the details of the response packet. It would also be possible to perform more complex analysis of this response packet using the built in psdump and pdfdump functions which will create a human readable analysis of the packet in PostScript and PDF respectively.

Welcome to Scapy (2.2.0)
Scapy HTTP Tester
>>> make_test("","")
Begin emission:
...Finished to send 1 packets.
Received 5 packets, got 1 answers, remaining 0 packets
###[ IP ]###
  version= 4L
  ihl= 5L
  tos= 0x20
  len= 56
  id= 64670
  frag= 0L
  ttl= 42
  proto= tcp
  chksum= 0x231b
###[ TCP ]###
     sport= http
     dport= ftp_data
     seq= 1130043850
     ack= 1
     dataofs= 9L
     reserved= 0L
     flags= SA
     window= 42900
     chksum= 0x8c7e
     urgptr= 0
     options= [('MSS', 1430), (254, 'xf9x89xcex04bmx13xd3)xc8')]


Scapy is a powerful tool, if a bit daunting in syntax initially. Creating raw TCP/IP packets systematically will probably challenge most people's understanding of the TCP/IP stack (it certainly did mine!) but exposing this level of configuration has serious advantages. Full control of the requests and responses as well as ability to add custom Python logic allows Scapy to become a packet foundry which you can use for things like unit testing of web applications, verification of state of an unknown network, etc. I will definitely be using Scapy in the future when performing raw HTTP testing of web applications.

published by (Steph Skardal) on 2015-04-28 12:39:00 in the "Conference" category

This blog post is really for myself. Because I had the unique experience of bringing a baby to a conference, I made an extra effort to talk to other attendees about what sessions shouldn't be missed. Here are the top takeaways from the conference that I recommend (in no particular order):

Right now, the videos are all unedited from the Confreaks live stream of the keynote/main room, and I'll update as the remaining videos become available.

A Message From the Sponsors

My husband: My conferences never have giveaways that cool.
Me: That's because you work in academia.

You can read the full list of sponsors here, but I wanted to comment further:

Hired was the diamond sponsor this year and they ran a ping pong tournament. The winner received $2000, and runners up received $1000, $500, $250. The final match was heavily attended and competitive! Practice up for next year?

Engine Yard also put on a really fun scavenger hunt using Scavify. Since I couldn't attend the multiple parties going on at night, this was a really fun way to participate and socialize.

published by (Mark Johnson) on 2015-04-28 12:00:00 in the "administration" category

Interchange default configurations have not done a good job of keeping up with the best available password security for its user accounts. Typically, there are two account profiles associated with a standard Interchange installation: one for admin users (access table) where the password is stored using Perl's crypt() command (bad); and one for customers (userdb) where the password isn't encrypted at all (even worse). Other hashing algorithms have long been available (MD5, salted MD5, SHA1) but are not used by default and have for some time not been useful protection. Part of this is convenience (tools for retrieving passwords and ability to distribute links into user assets) and part is inertia. And no small part was the absence of a strong cryptographic option for password storage until the addition of Bcrypt to the account management module.

The challenge we face in protecting passwords is that hardware continues to advance at a rapid rate, and with more computational power and storage capacity, brute-force attacks become increasingly effective and widely available. Jeff Jarmoc's Enough with the Salts provides some excellent discussion and background on the subject. To counter the changing landscape, the main line of defense moves toward ensuring that the work required to create and test a given stored password is too expensive, too time-consuming, for brute-force attacks to be profitable.

One of the best options for handling encrypted password storage with a configurable "hardware cost" is Bcrypt. We chose to integrate Bcrypt into Interchange over other options primarily because of its long history of operation with no known exploits, and its cost parameter that allows an administrator to advance the work required to process a password slowly over time as hardware continues to increase in efficiency. The cost feature introduces an exponential increase in calculation iterations (i.e., required processing power and time) as powers of 2, from 1 (2 iterations, essentially no cost) to 31 (2^31, or 2,147,483,648 iterations). Ideally an administrator would want to identify a cost that causes no perceptible penalty to end users, but would be such a burden to any brute-force attack as to have no worthwhile return on investment to crack.

Converting an existing user base from any of the existing encryption schemes to Bcrypt is trivial in Interchange. The existing UserDB profile is changed to the "bcrypt" option and the "promote" boolean set to true. Promote allows your users to continue to validate against their existing stored password, but after the next access will upgrade their storage to the Bcrypt password. In the mean time, a backend process could be developed using the construct_bcrypt() routine in Vend::UserDB to update all outstanding accounts prior to being updated organically.

If the switch on the front end involves going from no encryption to any encrypted storage, including Bcrypt, and your front end uses the default tools for retrieving lost passwords, you'll also need to construct some new code for resetting passwords instead. There is no such facility for the admin, and since the admin accounts are typically far more valuable than the front end accounts, making the change for the admin should be the first priority and have the least effort involved.

Switching accounts to Bcrypt password storage is a simple, effective means for increasing protection on your users' and business' information. Every bit as importantly, it also helps protect your business' reputation, that can be severely damaged by a data breech. Lastly, in particular for your admin accounts, Bcrypt password storage is useful in meeting PCI DSS requirements for strong password hashing.

published by (Steph Skardal) on 2015-04-27 13:02:00 in the "Conference" category

Last week, I brought my 4 month old to RailsConf. In a game-day decision, rather than drag a two year old and husband along on the ~5 hour drive and send the dogs to boarding, we decided it would ultimately be easier on everyone (except maybe me) if I attended the conference with the baby, especially since a good amount of the conference would be live-streamed.

Daily morning photos at the conference.

While I was there, I was asked often how it was bringing a baby to a conference, so I decided to write a blog post. As with all parenting advice, the circumstances are a strong factor in how the experience turned out. RailsConf is a casual three-day multi-track tech conference with many breaks and social events – it's as much about socialization as it is about technical know-how. This is not my first baby and not my first time at RailsConf, so I had some idea of what I might be getting into. Minus a few minor squeaks, baby Skardal was sleeping or sitting happily throughout the conference.

Here's what I [qualitatively] perceived to be the reaction of others attending the conference to baby Skardal:

In list form:

  • Didn't Notice: Probably about 50% didn't notice I had a baby, especially when she was sleeping soundly in the carrier.
  • Stares: Around 50% may have stared. See below for more.
  • Joke: A really small percentage of people made some variation of the joke "Starting her early, eh?"
  • Conversation: An equally small percentage of people started a conversation with me, which often led to more technical talk.

Here are some reasons I imagined behind the staring:

  • That baby is very cute (I'm not biased at all!)
  • Shock
  • Wonder if day care is provided (No, it wasn't. But with a 4 month old who hasn't been in day care yet, I probably wouldn't have used it.)
  • Too hungover to politely not stare

Pros & Cons

And here is how I felt after the conference, regarding pros and cons on bringing the baby:

  • A baby is a good conversation starter, which is beneficial in a generally introverted crowd.
  • I realized there are helpful & nice people in the world who offered to help plenty of times.
  • The baby was happiest staying with me.
  • Because children were the focus of many conversations, I missed out on talking shop a bit.
  • It's tiring, but in the same way that all parenting is.
  • I couldn't participate in all of the social/evening activities of the conference.
  • Staring generally makes me feel uncomfortable.


And finally, some tips:

  • Plan ahead:
    • Review the sessions in advance and pick out ones you want to attend because you may not have time to do that on the fly.
    • Walk (or travel) the route from your hotel to the conference so you know how long it will take and if there will be challenges.
  • Be agile and adapt. Most parents are already probably doing this with a 4 month old.
  • Manage your expectations:
    • Expect the conference with a baby to be tiring & challenging at times.
    • Expect stares.
    • Expect you won't make it to every session you want, so make a point of talking to others to find out their favorite sessions.
  • If not provided, ask conference organizers for access to a nursing or stashing room.
  • Bring baby gear options: carrier, stroller, bouncy seat, etc.
  • Research food delivery options ahead of time.
  • Order foods that are easy to eat with one hand. Again, another skill parents of a 4 month old may have developed.
  • Sit or stand in the back of talks.

While in these circumstances I think we made the right decision, I look forward to attending more conferences sans-children.

published by (Steph Skardal) on 2015-04-23 23:40:00 in the "Conference" category

Today, RailsConf concluded here in Atlanta. The day started with the reveal of this year's Ruby Heroes, followed by a Rails Core panel. Watch the video here.

On Trailblazer

One interesting talk I attended was See You on The Trail by Nick Sutterer, sponsored by Engine Yard, a talk where he introduced Trailblazer. Trailblazer is an abstraction layer on top of Rails that introduces a few additional layers that build on the MVC convention. I appreciated several of the arguments he made during the talk:

  • MVC is a simple level of abstraction that allows developers to get up and running efficiently. The problem is that everything goes into those three buckets, and as the application gets more complex, the simplified structure of MVC doesn't answer on how to organize logic like authorization and validation.
  • Nick made the argument that DHH is wrong when says that microservices are the answer to troublesome monolithic apps. Nick's answer is a more structured, organized OO application.
  • Rails devs often say "Rails is simple", but Nick made the argument that Rails is easy (subjective) but not simple (objective). While Rails follows convention with the idea that transitioning between developers on a project should be easy if conventions have been followed, in actuality, there is still so much interpretation into how and where to organize business logic for a complex Rails application that makes transition between developers less straightforward and not simple.
  • Complex Rails tends to include fat models (as opposed to fat controllers), and views with [not-so-helpful] helpers and excessive rendering logic.
  • Rails doesn't introduce convention on where dispatch, authorization, validation, business logic, and rendering logic should live.
  • Trailblazer, an open source framework, introduces a new abstraction layer to introduce conventions for some of these steps. It includes Cells to encapsulate the OO approach in views, and Operations to deserialize and validate without touching the model.

There was a Trailblazer demo during the talk, but as I mentioned above, the takeaway for me here is that rather than focus on the specific technical implementation at this point, this buzzworthy topic of microservices is more about good code organization and conventions for increasingly complex applications, that encourages readability and maintenance on the development side.

I went to a handful of other decent talks today and will include a summary of my RailsConf experience sharing links to popular talks on this blog.

published by Eugenia on 2015-04-23 01:35:38 in the "General" category
Eugenia Loli-Queru

Gwyneth Paltrow got a lot of heat lately, claiming that $29 a week (SNAP food stamps’ allowance) buys you enough food to eat. Her visual example of $29 in groceries was something of a disaster, showing… limes as a must-have food, and extremely low calorie density to the point of malnutrition.

So I put together a list of groceries that are both nutritious, and healthy. These items, calculated from prices found at the DollarStore, Walmart and other very cheap stores are on the bottom line of what’s possible for $29 a week, while staying somewhat Paleo (according to the Chris Kresser Paleo version that allows beans/rice/dairy). But in all reality, to really get proper nutrition, you’ll need about $50 a week. At $29, a lot of shortcuts had to be made.

So, in my list, you get 12,000 calories a week (1715 calories a day on average). According to this calorie calculator here, that’s (barely) enough food for a 6′ tall male, doing a job of medium activity (e.g. a cleaner). The average of “net” carbs are 157 per day, which is lower carb than the national average, but higher than a low-carb diet. Finally, olive oil was added to the calories consumed, even if olive oil was not calculated in the $29 per week limit. The reason for this is because neither Paltrow did calculate it (and it’s obvious that you can’t just plainly boil everything in her list and expect people to eat it as such). We should consider olive oil in this list, like we would also consider salt & pepper: “extras”, that are on everyone’s pantry by default.

So, here’s my $29 list:
$1.50, 1 lb lentils
$1.50, 1 lb garbanzo beans
$1, 1 lb white rice
$1.50 for 12 eggs
$1.50 for full fat milk

$3 for 4 frozen drumsticks
$2 for 2 cans of sardines
$1 for 1 can of tuna
$3 for 3 cans of shellfish (mussels, clams, or mix)

$3 for fruits (bananas, frozen fruits)
$8 for veggies (e.g. carrot, onion, greens, zucchini, potato)
$1 canned tomato sauce

Extra: Olive oil (20 tbpsoons a week)
Extra: salt & pepper

And here’s the actual nutrition breakdown (rice & beans are calculated as raw):

$29 a week

This is a very solid nutrition (MUCH better than the average Westerner’s nutrition), with only a couple of hiccups: more vitamin C is needed, and more vitamin E too (found only in nuts & seeds in ample quantities, which unfortunately are too expensive for this list). For these two vitamins, the person might have to supplement (if there’s no financial ability to buy nuts/seeds or more fruits). Also, supplementing in Magnesium and D3 are suggested for all people too, either poor or rich (most Westerners don’t get enough of it).

For maximum nutrition, here are some additional ideas:
– Boil the drumsticks for 3 hours on low, and keep the water (which is now bone broth). Using that broth, you can later add the tomato sauce, and some carrots, to eat with rice.
– Don’t drink the milk as-is: ferment it to kefir. Add the fruits and blend them with the kefir, as a smoothie for breakfast.
– Buy veggies in season for more variety and better prices. Get a bit of everything found in a plant: tubers, roots, bulbs, greens, fruits. Prefer greens though.
– Place the lentils and beans in water for 24 hours before cooking, to remove most of its lectins. Cook them in high heat for the same reason.
– While more expensive, always prefer olive oil to industrial seed oils. The money you would save buying seed oils, you’d pay later to doctors.

So, anyway, all this is kinda possible, but $50 a week per person is probably what’s needed for a more optimized nutrition.

published by (Steph Skardal) on 2015-04-22 23:52:00 in the "Conference" category

It's day 2 of RailsConf 2015 in Atlanta! I made it through day 1!

The day started with Aaron Patterson's keynote (watch it here). He covered features he's been working on including auto parallel testing, cache compiled views, integration test performance, and "soup to nuts" performance. Aaron is always good at starting his talk with self-deprecation and humor followed by sharing his extensive performance work supported by lots of numbers.

On Hiring

One talk I attended today was "Why We're Bad At Hiring (And How To Fix It)" by @kerrizor of Living Social (slides here, video here). I was originally planning on attending a different talk, but a fellow conference attendee suggested this one. A few gems (not Ruby gems) from this talk were:

  • Imagine your company as a small terrarium. If you are a very small team, hiring one person can drastically affect the environment, while hiring one person will be less influential for larger companies. I liked this analogy.
  • Stay away from monocultures (e.g. the banana monoculture) and avoid hiring employees just like you.
  • Understand how your hiring process may bias you to reject specific candidates. For example, requiring a GitHub account may bias reject applicants that are working with a company that can't share code (e.g. security clearance required). Another example: requiring open source contributions may bias reject candidates with very little free time outside of their current job.
  • The interview process should be well organized and well communicated. Organization and communication demonstrate confidence in the hiring process.
  • Hypothetical scenarios or questions are not a good idea. I've been a believer of this after reading some of Malcolm Gladwell's books where he discusses how circumstances are such a strong influence of behavior.
  • Actionable examples that are better than hypothetical scenarios include:
    1. ask an applicant to plan an app out (e.g. let's plan out strategy for an app that does x)
    2. ask an applicant to pair program with a senior developer
    3. ask the applicant to give a lightning talk or short presentation to demonstrate communication skills
  • After a rejected interview, think about what specifically might change your mind about the candidate.
  • Also communicate the reapplication process.
  • Improve your process by measuring with the goal to prevent false negatives. One actionable item here is to keep tabs on people – are there any developers you didn't hire that went on to become very successful & what did you miss?
  • Read this book.

Interview practices that Kerri doesn't recommend include looking at GPA/SAT/ACT scores, requiring a Pull request to apply, speed interviews, giving puzzle questions, whiteboard coding & fizzbuzz.

While I'm not extremely involved in the hiring processes for End Point, I am interested in the topic of growing talent within a company. The notes specifically related to identifying your own hiring biases was compelling.


I also attended a few talks on testing. My favorite little gem from one of these talks was the idea that when writing tests, one should try to balance between readability, maintainability, and performance, see:

Eduardo Gutierrez gave a talk on Capybara where he went through explicit examples of balancing maintainability, readability, and performance in Capybara. I'll update this post to include links to all these talks when they become available. Here are the videos & slides for these talks:

published by (Steph Skardal) on 2015-04-22 00:42:00 in the "Conference" category

I'm here in Atlanta for my sixth RailsConf! RailsConf has always been a conference I enjoy attending because it includes a wide spectrum of talks and people. The days are long, but rich with nitty gritty tech details, socializing, and broader topics such as the social aspects of coding. Today's keynote started with DHH discussing the shift towards microservices to support different types of integrated systems, and then transitioned to cover code snippets of what's to come in Rails 5, to be released this year. Watch the keynote here.

Open Source & Being a Hero

One of the talks I was really looking forward to attending was "Don't Be a Hero - Sustainable Open source Dev" by Lillie Chilen (slides here, video here), because of my involvement in open source (with Piggybak, RailsAdminImport, Annotator and Spree, another Ruby on Rails ecommerce framework). In the case of RailsAdminImport, I found a need for a plugin to RailsAdmin, developed it for a client, and then released it into the open source with no plans on maintaining a community. I've watched as it's been forked by a handful of users who were in need of the same functionality, but I most recently gave another developer commit & Rubygems access since I have historically been a horrible maintainer of the project. I can leverage some of Lillie's advice to help build a group of contributors & maintainers for this project since it's not something I plan to put a ton of time into.

With Piggybak, while I haven't developed any new features for it in a while, I released it into the open source world with the intention of spending time maintaining a community after being involved in the Spree community. Piggybak was most recently upgraded to Rails 4.2.

Lillie's talk covered actionable items you can do if you find yourself in a hero role in an open source project. She explained that while there are some cool things about being a hero, or a single maintainer on a project, ultimately you are also the single point of failure of the project and your users are in trouble if you get eaten by a dinosaur (or get hit by a bus).

Here are some of these actionable items to recovery from hero status:

  1. Start with the documentation on how to get the app running, how to run tests, and how to contribute. Include comments on your workflow, such as if you like squashed commits or how documentation should look.
  2. Write down everything you do as a project maintainer and try to delegate. You might not realize all the little things you do for the project until you write them down.
  3. Try to respond quickly to requests for code reviews (or pull requests). Lillie referenced a study that mentioned if a potential contributor receives a code review within 48 hours, they are much more likely to come back, but if they don't hear back within 7 days, there is little chance they will continue to be involved.
  4. Recruit collaborators by targeted outreach. There will be a different audience of collaborators if you open source tool is an app versus a library.
  5. Manage your own expectations for contributors. Understand the motivations of contributors and try to figure out ways to encourage specific deliverables.
  6. Have regular retrospectives to analyze what's working and what's not, and encourage introspection.

While Lillie also covered several things that you can do as a contributor, I liked the focus on actionable tasks here for owners of projects. The ultimate goal should be to find other collaborators, grow a team, and figure out what you can do to encourage people to progress in the funnel and transition from user to bug fixer to contributor to maintainer. I can certainly relate to being the single maintainer on an open source project (acting as a silo), with no clear plan as to how to grow the community.

Other Hot Topics

A couple of other hot topics that came up in a few talks were microservices and Docker. I find there are hot topics like this at every RailsConf, so if the trend continues, I'll dig deeper into these topics.

What Did I Miss?

I always like to ask what talks people found memorable throughout the day in case I want to look back at them later. Below are a few from today. I'd like to revisit these later & I'll update to include the slides when I find them.

published by (Greg Davidson) on 2015-04-21 18:26:00 in the "Chef" category

When Vagrant/Chef Provisioning Goes South

I recently ran into the following error when provisioning a new vagrant machine via the `vagrant up` command:

[2015-04-21T17:10:35+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2015-04-21T17:10:35+00:00] ERROR: Cookbook loaded at path(s) [/tmp/vagrant-chef/path/to/my-cookbook] has invalid metadata: The `name' attribute is required in cookbook metadata
[2015-04-21T17:10:35+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

After some googling and digging I learned version 12 of chef-client introduced a breaking change. From version 12 on, every cookbook requires a name attribute in their metadata.rb file . A quick grep through the metadata.rb files in the project revealed several did not include name attributes. You would be correct at this point to suggest I could have added name attributes to the cookbook metadata files and been done with this. However, in this case I was a consumer of the cookbooks and was not involved in maintaining them so an alternate solution was preferable.

Selecting a Specific Version of Chef in Vagrant

My idea for a solution was to install the most recent chef-client release prior to version 12. I was not sure how to do this initially but along the way I learned that by default, Vagrant will install the most recent release of chef-client. The Vagrant documentation for Chef provisioners described what I needed to do. The Chef version could be specified in config.vm.provision block in the Vagrantfile:

config.vm.provision :chef_solo do |chef|
      chef.version = "11.18"
      chef.cookbooks_path = "cookbooks"
      chef.data_bags_path = "data_bags"

      # List of recipes to run
      chef.add_recipe "vagrant_main::my_project"

With this configuration change, chef-client 11.18 completed the provisioning step successfully.

published by (Spencer Christensen) on 2015-04-21 15:38:00 in the "big-data" category

One of the biggest problems for web development environments is copying large amounts of data. Every time a new environment is needed, all that data needs to be copied. Source code should be tracked in version control software, and so copying it should be a simple matter of checking it out from the code repository. So that is usually not the problem. The main problem area is database data. This can be very large, take a long time to copy, and can impact the computers involved in the copy (usually the destination computer gets hammered with IO which makes load go high).

Often databases for development environments are created by copying a database dump from production and then importing that database dump. And since database dumps are text, they can be highly compressed, which can result in a relatively small file to copy over. But the import of the dump can still take lots of time and cause high load on the dev computer as it rebuilds tables and indexes. As long as your data is relatively small, this process may be perfectly acceptable.

Your database WILL get bigger

At some point though your database will get so big that this process will take too long and cause too much load to be acceptable.

To address the problem you can try to reduce the amount of data involved by only dumping a portion of the database data instead of all of it, or possibly using some "dummy sample data" instead. These techniques may work if you don't care that development environments no longer have the same data as production. However, one serious problem with this is that a bug or behavior found in production can't be replicated in a development environment because the data involved isn't the same. For example, say a customer can't checkout on the live site but you can't replicate the bug in your development environment to fix the bug. In this example, the root cause of the problem could be a bug in the code handling certain products that are out of stock, and since the dev database didn't have the same data it could make finding and fixing these types of problems a lot harder.


Another option is to use file system snapshots, like LVM snapshots, to quickly make clones of the database without needing to import the database dump each time. This works great if development environments live on the same server, or at least the development databases live on the same server. You would need to create a volume to hold a single copy of the database; this copy would be the origin for all snapshots. Then for each development environment, you could snapshot the origin volume, mount it read-write in a place accessible by the developer, customize the database configuration (like setting a unique port number to listen on), and then start up the database. This then provides a clone of the entire database in a tiny fraction of the time and uses less disk space and other system resources too.

In using snapshots there are some things you'll need to be careful about. Snapshots are usually created using copy-on-write tables. The more snapshots mounted read-write, the more IO overhead is involved for the volumes involved. For this reason it is important that writes to the origin volume be avoided as much as possible while the snapshots are open. Also, snapshots that get a lot of writes can fill up their copy-on-write table, and depending on the file system and database that you are using this can be a big problem. So it is important to monitor each open snapshot for how full it is and increase their size if needed so they don't fill up. Updating the origin database will require shutting down and removing all snapshots first, then update the origin database, then create and mount all the snapshots again. This is because all the copy-on-write tables would get full if you tried to update the origin while the snapshots are open.

Using snapshots like this may sound more complicated, and it is, but the processes involved can be scripted and automated and the benefits can be pretty significant if you have several developers and a lot of data to copy.

published by (Neil Elliott) on 2015-04-20 19:58:00 in the "linux" category

As a high-performance video rendering appliance, the Liquid Galaxy requires really good video cards -- better than your typical on-board integrated video cards. Despite ongoing attempts by competitors to displace them, Nvidia remains the best choice for high-end video, if you use the proprietary Nvidia driver for Linux.

In addition to providing regular security and system updates, End Point typically provides advanced remote monitoring of our customers' systems for issues such as unanticipated application behavior, driver issues, and hardware errors. One particularly persistent issue presents as an error with an Nvidia kernel module.  Unfortunately, relying on proprietary Nvidia drivers so as to maintain an acceptable performance level limits the available diagnostic information and options for resolution.

The issue presents when the system ceases all video output functions as Xorg crashes. The kernel log contains the following error message:

2015-04-14T19:59:00.000083+00:00 lg2 kernel: [  719.850677] NVRM: Xid (0000:01:00): 32, Channel ID 00000003 intr 02000000

The message is repeated approximately 11000 times every second until the disk fills and the ability to log in to the system is lost. The only known resolution at this time is to power-cycle the affected machine. In the error state, the module cannot be removed from the kernel, which also prevents Linux from shutting down properly. All affected systems were running some version of Ubuntu x86-64. The issue seems to be independent of driver version, but is at least present in 343.36 and 340.65, and affects all Geforce cards. Quadro cards seem unaffected.

The Xid message in the kernel log contains an error code that provides a little more information. The Nvidia docs list the error as "Invalid or corrupted push buffer stream". Possible causes listed include driver error, system memory corruption, bus error, thermal error, or frame buffer error. All affected systems were equipped with ECC RAM and were within normal operating temperature range when the issue presented.

Dealing with bugs like these can be arduous, but until they can be fixed, we cope by monitoring and responding to problems as quickly as possible.

published by (Brian Gadoury) on 2015-04-17 23:00:00 in the "culture" category

A conversation with a co-worker today about the value of improving one's professional skills reminded me of Joe Mastey's talk he gave at the 2015 Mountain West Ruby Conference. That then reminded me that I had never finished my write up on that conference. Blogger won't let me install harp music and an animated soft focus flashback overlay, so please just imagine it's the day after the conference when you're reading this. "That reminds me of the time..."

I've just finished my second MWRC and I have to give this one the same 5-star rating I gave last year's. There were a few small sound glitches here and there, but overall the conference is well-run, inclusive, and packed with great speakers and interesting topics. Rather than summarizing each talk, I want to dig into the one most relevant to my interests. "Building a Culture of Learning" by Joe Mastey

I was excited to catch Joe's talk because learning and teaching have always been very interesting to me, regardless of the particular discipline. I find it incredibly satisfying to improve upon my own learning skills, as well as improving my teaching skills by teasing out how different individuals learn best and then speak to that. There's magic in that one on one interaction when everything comes together just right. I just really dig that.

Joe's work as the Manager of Internal Learning at Enova has gone way beyond the subtleties of the one-on-one. He's taken the not-so-simple acts of learning and training, and scaled them up in an environment that does not sound, on paper, like it would support it. He's created a culture of learning ("oh hey they just said the title of the movie in the movie!") in a financial company that's federally regulated, saw huge growth due to an IPO, and had very real business-driven deadlines for shipping their software.

Joe broke his adventure down into three general phases after refreshingly admitting that "YMMV" and that you can't ignore the existing corporate culture when trying to build a culture of learning within.

Phase 1 - Building Credibility

I would hazard a guess that most software development shops are perpetually at Phase 1: Learning is mostly ad-hoc by way of picking things up from one's daily work; and has little to no people pushing for more formal training. People probably agree that training is important, but the mandate has not come down from the CTO, and there's "no time for training" because there's so much work to do.

How did Joe help his company evolve past Phase 1? Well, he did a lot of things that I think many devs would be happy to just get one or two of at the their company. My two favorites from his list probably appeal to polar opposite personality types, but that's part of why I like them.

My first favorite is that books are all-you-can-eat. If a developer asks Joe for a tech book, he'll say yes, and he'll buy a bunch of extra copies for the office. I like having a paper book to read through to get up to speed on a topic, ideally away from my desk and the computer screen. I've also found that for some technologies, the right book can be faster and less frustrating than potentially spotty documentation online.

My second favorite is how Joe implemented "new hire buddies." Each new hire is teamed up with an experienced dev from a different team. Having a specific person to talk to, and get their perspective on company culture, can really help people integrate into the culture much more quickly. When I joined End Point in 2010, I worked through our new hire "boot camp" training like all new hires. I then had the occasionally-maddening honor of working directly with one of the more senior evil super-geniuses at End Point on a large project that I spent 100% of my time on. He became my de facto new hire buddy and I could tell that despite the disparity in our experience levels, being relatively joined at the hip with someone like that improved my ramp-up and cultural integration time greatly.

Phase 2 - Expand Reach and Create Impact

If my initial guess about Phase 1 is correct, it follows that that dev shops in Phase 2 are more rare: More people are learning more, more people are driving that learning, but it's still mostly focused on new hires and the onboarding process.

Phase 2 is where Joe's successful efforts are a little more intimidating to me, especially given my slightly introverted nature. The efforts here scale up and get more people speaking publically, both internally and externally. It starts with a more formal onboarding process, and grows to things like weekly tech talks and half day internal workshops. Here is where I start to make my "yeah, but?" face. We all have it. It's the face you make when someone says something you don't think can work, and you start formulating your rebuttal immediately. E.g. "Yeah, but how do you get management and internal clients to be OK with ?shutting down development for half a day' for training?" Joe does mention the danger of being perceived as "wasting too much time." You'll want to be sure you get ahead of that and communicate the value of what you're spending "all that dev time on."

Phase 3 - Shift The Culture

It would be interesting to know how many shops are truly in Phase 3 because it sounds pretty intense: Learning is considered part of everyone's job, the successes from the first two phases help push the culture of learning to think and act bigger, the acts of learning and training others are part of job descriptions, and things like FOSS contributions and that golden unicorn of "20% personal project time" actually happen on company time. Joe describes the dangers or downsides to Phase 3 in a bit of a "with great power comes great responsibility" way. I've never personally worked somewhere that's in Phase 3, but it make sense that the increased upside has increased (potential) downside.

At End Point, we have some elements of all three phases, but we're always looking to improve. Joe's talk at MWRC 2015 has inspired me to work on expanding our own culture of learning. I think his talk is also going to serve as a pretty good road-map on how to get to the next phase.