All opinions expressed are those of the authors and not necessarily those of, our sponsors, or our affiliates.
  Add to My Yahoo!  Subscribe with Bloglines  Subscribe in NewsGator Online

published by (Jon Jensen) on 2015-07-31 16:50:00 in the "company" category

We are excited to announce an expansion of End Point?s ecommerce clientele and our developer ranks! The ecommerce consulting company Perusion has joined End Point. Perusion was founded in 2002 by Mike Heins and Greg Hanson. It quickly became a small powerhouse in the open source ecommerce space, focusing on Interchange, Perl, and MySQL on Linux. We were pleased to welcome Perusion in a merger with End Point at the beginning of July.

Mike Heins is the original creator of MiniVend in 1996. In 2000, Mike?s consultancy and MiniVend were acquired by the ecommerce startup Akopia. With numerous improvements, including the addition of a new full-featured administrative back-office, the new open source ecommerce platform Interchange was created. Akopia was acquired by Red Hat in 2001, and in 2002 the Interchange project became independent, led by its creators and a group of other open source developers who maintain it to this day.

Greg Hanson is a serial entrepreneur and business manager and has worked extensively with data systems of many types. In the mid-1990s he started a computer products ecommerce company, Valuemedia, and oversaw every aspect of its evolution. Greg joined Mike to launch Perusion in 2002, and is now a client consultant and developer. He has shepherded several businesses from small mom & pop operations into strong companies providing goods and services around the world.

Josh Lavin began creating websites professionally in 1998 at his consultancy Kingdom Design. He grew his work into the ecommerce space, helping many companies sell their products online for the first time. Josh joined Perusion in 2007, bringing skills in marketing and user experience that are just as important as his development abilities. In recent years he has enjoyed moving client sites over to responsive front-end designs that work well on desktop, tablet, and mobile phone.

Perusion?s development and hosting clients have joined End Point as well. Some of the noteworthy sites that bear mentioning here are American Welding Society, Bluestone Perennials, Vervanté, Bulk Herb Store, Northern Sun, Penn Herb Company, Air Delights, and Solar Pathfinder.

At End Point we got our start in the industry in 1995 by creating dynamic database-backed websites for our clients. Our tools of choice in those early days were Linux, Apache, msql and MySQL, and Perl. In the late 1990s we began focusing more on ecommerce websites specifically, and we added MiniVend and Interchange to the mix.

Later we branched out into PostgreSQL, Ruby on Rails and Spree, Python and Django, Perl Dancer, NodeJS, and other platforms, while we continued to support and enhance Interchange and the sites running on it. Today we still host and develop many successful ecommerce sites running Interchange, taking many thousands of orders worth millions of dollars every day. So we are delighted to have Perusion join us as we to continue to grow the Interchange-based part of our business.

We have already seen constructive collaboration in this merger, with longtime End Point employees able to add more helping hands to Perusion projects and further breadth of capabilities and depth of support, Perusion developers bringing their expertise to bear, and Perusion contacts leading to new projects and new business.

Perusion has always believed in going above and beyond the call of duty and has made their clients? success their own goal. Likewise, we at End Point feel this is more than a business, and we value the personal relationships we have developed with each other and our clients over the years. We look forward to the new possibilities that are now available through this change, and are glad to welcome Perusion on board at End Point!

published by (Kent K.) on 2015-07-31 13:00:00 in the "defaults" category

Recently, I needed to add some functionality to an older section of code. This code initialized and passed around a reasonably sized set of various hashes, all with similar keys. As those hashes were accessed and manipulated, there were quite a few lines of code devoted to addressing boundary conditions within those hashes. For example, an if/else statement setting a default value to a given key if it didn't already exist. With all the added safety checks, the main method dragged on for several screens worth of code. While puttering around amidst this section, I figured I'd be a good little boyscout and leave my campsite better than when I found it.

I figured a fairly easy way to do that would be to eliminate the need for all the extra if/else clauses laying around. All they were really doing was ensuring we ended up with a minimum set of hash keys. I decided to turn all the various hashes into instances of a very simple class inheriting from Ruby on Rails' HashWithIndifferentAccess along with some basic key management functionality.

My first draft came out looking something like:

class MyHash < HashWithIndifferentAccess
  def initialize(constuctor = {}) do
    self[:important_key_1] ||= "default 1"
    self[:important_key_2] ||= "default 2"
This seemed to work fine. I didn't need to worry about ensuring "important keys" were present anymore. And it was perfectly viable to pass in one of the important keys as part of the initialization.

I soon discovered in my test suite that my code did not do exactly what I intended it to do. In the tests, I wanted to ensure several of my hash keys came out with the right values. I made use of MyHash#slice to ensure I ended up with the right subset of values for my given test. However, no matter what I did, I could not weed out the important keys:

1.9.3 :003 >{foo: 'bar', bar: 'lemon'}).slice(:bar)
=> {"important_key_1"=>"default 1", "important_key_2"=>"default 2", "bar"=>"lemon"}
I admit I was quite perplexed by this. I tried several re-writes of the initialize method looking for some version that didn't exhibit this strange slice behavior. Finally, I took to the Ruby on Rails and Ruby docs.

Looking at the source for slice, I found the problem:

The method slice calls "new" (which includes the default values) to create another object to avoid clobbering the one the method is called on. Since I didn't feel like writing my own custom slice method, or trying to monkey patch Rails, I realized I was beaten. I needed to find a new solution.

After a little thought, I came up with this:

class MyHash < HashWithIndifferentAccess
  def = {}) do
    h = new(constructor)
    h[:important_key_1] ||= "default 1"
    h[:important_key_2] ||= "default 2"
This does not fix the problem, but manages to sidestep it. Most Ruby on Rails veterans will be familiar with methods called "build" so it shouldn't be too clumsy to work with. I replaced all the entries in my code that called with and went on my merry way.

published by (Emanuele 'Lele' Calo') on 2015-07-30 13:54:00 in the "image" category

After a fairly good experience with dnote installed on our own servers as an encrypted notes sharing service, my team decided that it would have been nice to have a similar service for images.

We found a nice project called that is based on NodeJS, Python, Redis and a lot of client-side JavaScript.

The system is divided into two components: the HTML/JS frontend and a Python FastCGI API.

Unfortunately the documentation is a still in its very early stage and it's lacking a meaningful structure and a lot of needed information.

Here's an overview of the steps we followed to setup on our own server behind nginx.

First of all we chose that we wanted to have as much as possible running and confined to a regular user, which is always a good idea with such young and potentially vulnerable tools. We chose to use the imgbi user.

Then since we wanted to keep as clean as possible the root user environment (and system status), we also decided to use pyenv. To be conservative we chose the latest Python 2.7 stable release, 2.7.10.

git clone ~/.pyenv
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
echo $SHELL -l
pyenv install -l  | grep 2\.7
pyenv install 2.7.10
pyenv global 2.7.10
pyenv version
which python
python --version

In order to use, we also needed NodeJS and following the same approach we chose to use nvm and install the latest NodeJS stable version:

curl -o- | bash
nvm install stable
nvm list
nvm use stable
nvm alias default stable
node --version

After that we updated pip and then installed all the needed Python modules:

pip install --upgrade pip
pip install redis m2crypto bcrypt pysha3 zbase62 pyutil flup

Then it's time to clone the actual code from the GitHub repo, install a few missing dependencies and then use the bower and npm .json files to add the desired packages:

git clone
npm install -g bower grunt grunt-cli grunt-multiresize
npm install -g grunt-webfont --save-dev
npm install
bower install

We also faced an issue which made Grunt fail to start correctly. Grunt was complaining about an "undefined property" called "prototype". If you happen to have the same problem just type

cd node_modules/grunt-connect-proxy/node_modules/http-proxy
npm install eventemitter3@0.1.6
cd -

That'll basically install the eventemitter3 NodeJS package module locally to the grunt-connect-proxy module so to overcome the compatibility issues which in turn causes the error mentioned above.

You should use your favourite editor to change the file config.json, which basically contains all your local needed configuration. In particular our host is not exposed on the I2P or Tor network, so we "visually" disabled those options.

# lines with "+" needs to be replace the ones starting with a "-"
-  "name": "",
+  "name": " - End Point image sharing service",

-  "maxSize": "3145728",
+  "maxSize": "32145728",

-  "clearnet": "",
+  "clearnet": "https://imgbi.example",

-  "i2p": "http://imgbi.i2p",
+  "i2p": "http://NOTAVAILABLE.i2p",

-  "tor": "http://imgbifwwqoixh7te.onion",
+  "tor": "http://NOTAVAILABLE.onion",

Save and close the file. At this point you should be able to run "grunt" to build the project but if it fails on the multiresize task, just run

grunt --force

to ignore the warnings.

That's about everything you need for the frontend part, so it's now time to take care of the API.

git clone
cd /home/imgbi/

You now need to edit the two Python files which are the core of the API.

# edit
-upload_dir = '/home/'
+upload_dir = '/home/imgbi/'

Verify that you're not having any Python import related error, due to missing modules or else, by running the Python file directly.


If that's working okay, just create a symlink in the build directory in order to have the API created files available to the frontend

ln -s /home/imgbi/ /home/imgbi/

And then it's time to spawn the actual Python daemon:

spawn-fcgi -f /home/imgbi/ -a -p 1234

The file is used by a cronjob which periodically checks if there's any image/content that should be removed because its time has expired. First of all let's call the script directly and if there's no error, let's create the crontab:

python /home/imgbi/

crontab -e

@reboot spawn-fcgi -f /home/imgbi/ -a -p 1234
30 4 * * * python /home/imgbi/

It's now time to install nginx and Redis (if you still haven't done so), and then configure them. For Redis you can just follow the usual simple, basic installation and that'll be just okay. Same is true for nginx but we'll add our configuration/vhost file content here as an example /etc/nginx/sites-enabled/imgbi.example.conf for everyone who may need it:

upstream imgbi-fastcgi {

server {
  listen 80;
  listen [::]:80;
  server_name imgbi.example;
  access_log /var/log/nginx/sites/imgbi.example/access.log;
  error_log /var/log/nginx/sites/imgbi.example/error.log;
  rewrite ^ https://imgbi.example/ permanent;

server {
  listen 443 ssl spdy;
  listen [::]:443 ssl spdy;
  server_name  imgbi.example;
  server_name  imgbi.example;
  access_log /var/log/nginx/sites/imgbi.example/access.log;
  error_log /var/log/nginx/sites/imgbi.example/error.log;

  client_max_body_size 4G;

  include include/;

  add_header Strict-Transport-Security max-age=31536000;
  add_header X-Frame-Options SAMEORIGIN;
  add_header X-Content-Type-Options nosniff;
  add_header X-XSS-Protection "1; mode=block";

  location / {
    root /home/imgbi/;

  location /api {
    fastcgi_param QUERY_STRING $query_string;
    fastcgi_param REQUEST_METHOD $request_method;
    fastcgi_param CONTENT_TYPE $content_type;
    fastcgi_param CONTENT_LENGTH $content_length;

    fastcgi_param SCRIPT_NAME "";
    fastcgi_param PATH_INFO $uri;
    fastcgi_param REQUEST_URI $request_uri;
    fastcgi_param DOCUMENT_URI $document_uri;
    fastcgi_param DOCUMENT_ROOT $document_root;
    fastcgi_param SERVER_PROTOCOL $server_protocol;

    fastcgi_param GATEWAY_INTERFACE CGI/1.1;
    fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;

    fastcgi_param REMOTE_ADDR $remote_addr;
    fastcgi_param REMOTE_PORT $remote_port;
    fastcgi_param SERVER_ADDR $server_addr;
    fastcgi_param SERVER_PORT $server_port;
    fastcgi_param SERVER_NAME $server_name;
    fastcgi_param HTTPS on;

    fastcgi_pass imgbi-fastcgi;
    fastcgi_keep_conn on;

Well, that should be enough to get you started and at least have all the components in place. Enjoy your secure image sharing now.

published by (Jon Jensen) on 2015-07-29 00:35:00 in the "philosophy" category

A brief thought:

You may have heard the saying that nothing is more permanent than a temporary fix. Or that prototypes are things we just haven't yet recognized will be permanent. Or some variation on the theme.

As an illustration of this, I recently came across the initial commit to the source code repository of our website when we ported it to Ruby on Rails back in April 2007. Our then co-worker PJ's comment is a perfect example of how long-lasting some of our planned temporary work can be:

commit 2ee55da6ed953c049b3ef6f9f132ed3c1e0d4de9
Author: PJ Cabreras <>
Date:   Wed Apr 18 13:07:46 2007 +0000

    Initial test setup of repository for mkcamp testing -- will probably throw away later
    git-svn-id: file:///home/camp/endpoint/svnrepo/trunk@1 7e1941c4-622e-0410-b359-a11864f70de7

It's wise to avoid big architecture up front for experimental things we don't know the needed shape and size of. But we should plan on iterating and being agile (in the real basic sense of the word), because we may never have the chance to start over from scratch. And starting over from scratch is often ill-advised in any case.

published by (Jon Jensen) on 2015-07-24 05:23:00 in the "ecommerce" category

The big picture

Computer security is a moving target, and during the past few years it's been moving faster than ever.

In the e-commerce world, the PCI Security Standards Council sets the rules for what merchants and vendors must do to have what they consider to be a sufficiently secure environment to handle cardholder data such as credit card numbers, expiration dates, and card security codes.

PCI DSS 3.1, released on 15 April 2015 puts us all on notice that TLS 1.0 is considered unfit to use for e-commerce website encryption (HTTPS), and will be disallowed soon. The new rules specify that new software implementations must not use TLS versions prior to 1.1. Existing implementations must require TLS 1.1 or 1.2 no later than 30 June 2016.

They provide some guidance and explain what is expected in more detail.

Long ago we were required to disable SSL 2, and last year we were expected to disable SSL 3, the predecessor to TLS 1.0. That turned out to not be particularly hard or cause too many problems, because almost all systems that supported SSL 3 also supported TLS 1.0.

This time we are not so lucky. Many clients (such as browsers) and servers did not support TLS beyond version 1.0 until fairly recently. That means much more work is involved in meeting these new requirements than just changing some settings and restarting servers.

Almost every client (browser) and server that supports TLS 1.1 also supports TLS 1.2, and almost everything that doesn't support TLS 1.2 doesn't support TLS 1.1 either. So to keep things simpler here I'll just talk about TLS 1.0 vs. TLS 1.2 below, and TLS 1.1 can be assumed to apply as well where TLS 1.2 is mentioned.

At End Point we deploy, support, and host e-commerce sites for many customers, so I'll talk about the server side of this first. Note that servers can act as both server and client in TLS connections, since servers often make outgoing HTTPS connections as well as accepting incoming requests. Let's review the situation with each of the major Linux server operating systems.


Debian 8 is the current version, and supports TLS 1.2. It is scheduled to be supported until April 2020.

Debian 7 has planned support until May 2018, but unfortunately it does not support TLS 1.2.

Debian's support lifetime has historically depended on how quickly future releases come, but recently the project began to offer long-term support (LTS) for Debian 6, which was supposed to be at end of life, so it will be supported until February 2016. It also only supports TLS 1.0.


Ubuntu's long-term support (LTS) server versions are supported for 5 years. Currently supported versions 12.04 and 14.04 both handle TLS 1.2.

Some sites are still using Ubuntu 10.04, which supports only TLS 1.0, but its support ended in April 2015, so it should not be used any longer in any case.

Red Hat Enterprise Linux (RHEL) and CentOS

Red Hat and CentOS are "enterprise" operating systems with a very long support lifetime of 10 years. Because that is so long, the oldest supported versions may become practically unusable due to changes in the world such as the deprecation of TLS 1.0.

RHEL/CentOS 7 is the current version, supported until June 2024. It supports TLS 1.2.

RHEL/CentOS 6 is supported until November 2020. It is mostly ok for TLS 1.2. One exception is that the bundled version of curl doesn't support TLS > 1.0 for some reason, so if you have applications making curl client calls to other systems, they may break without workarounds.

RHEL/CentOS 5 is the oldest version still supported, until March 2017, and it is very widely used, but it does not supprt TLS > 1.0.

Old server remediation

If you're on an older server that doesn't support TLS 1.2, the best thing to do is upgrade or migrate to a newer operating system, as soon as possible.

The common versions of OpenSSL that don't support TLS 1.2 also do not support Server Name Indication used for hosting multiple HTTPS sites on the same IP address. That is now becoming more commonly used since the thing holding back acceptance was old versions of Windows XP that didn't support it, and they are now are mostly dead. You can control whether you need SNI on the server side, avoiding it by continuing to get a separate IP address for each HTTPS site you host. But when you are a client of someone else's service that requires SNI, you'll wish you had it.

So migrate. That's easier said than done, of course. Moving to a new OS version involves a lot of new system library versions, language versions, web server version, etc. Some things aren't compatible. It takes work and time. That's life, so accept it and move ahead. Advocate it, schedule it, do it. But in the meantime, if you must cope with old servers, there are some not entirely terrible options.

You could use plain HTTP on a local private network to talk to a newer server running stunnel or an nginx proxy to do the TLS layer, or use a VPN if you have no private network.

You can use CDN in front of your site, which will certainly support TLS 1.2, and covers the path between the end user and the CDN, at least.

You can build your own versions of OpenSSL, any libraries that link to OpenSSL such as curl or wget, Apache or nginx, etc. This is tempting but is a terrible option, because you are almost certain to not update this hand-crafted stack often enough in the future to protect against new vulnerabilities in it. Sidestepping the operating system's native package management for core infrastructure software like this is usually a mistake.

You could avoid that problem by using someone else's backported parallel-install packages of all that, if you can find some, and if you think they're trustworthy, and if they're going to maintain them so you can get later updates. I'm not familiar with anyone doing this, but it may be out there and could be hired for the right ongoing price.

But the best bet is to start planning your upgrade or migration as soon as possible.

Browser support for TLS 1.2

Of course the other half of the connection is the client, primarily end-users' web browsers. On Wikipedia is a very detailed table showing various browsers' support of various features, broken down by version, here: TLS support history of web browsers. My summary follows:

Google Chrome and Mozilla Firefox have been automatically updating themselves for a long time, so unless you or your system administrator have disabled the auto-update, they will work with TLS 1.2.

Internet Explorer 8, 9, and 10 support TLS 1.2, but it is disabled by default. Not until IE 11 can TLS 1.2 be counted on to work.

Apple Safari 7, 8, and 9 for Mac OS X support TLS 1.2.

The built-in browser on Android < 4.4 doesn't support TLS > 1.0, and Android 4.4 has TLS > 1.0 disabled by default, which is the same thing for most users. So anyone with Android < 5.0 will not be able to connect to your site unless they're using a separate and newer mobile browser such as Chrome and Firefox.

Browser support for TLS 1.0

I have not heard of any timelines being announced for browsers to disable TLS 1.0 yet. I suspect that's because there are still so many servers that only support TLS 1.0. But before too long we may start to see servers catch up and then I expect browsers will eventually disable TLS 1.0.

Other clients

There are too many non-browser clients to list here. We've already mentioned curl; there is also wget, and the web client libraries in Perl, Python, Ruby, and Java. PHP uses libcurl. NodeJS and Go are likely not from the operating system and newer than it, so may be more current. At any rate, some of those clients will be old and won't support TLS 1.2, so when other sites stop allowing TLS 1.0 connections, whatever you were talking to them for will stop working.

PCI DSS will require client applications to stop using TLS 1.0 also, which may mean that applications need to be configured to require TLS 1.2 for outgoing HTTPS connections.


Your systems need to stop supporting TLS 1.0 by June 2016 at the latest. Start planning the migration now! We are available to help our current and new clients test, assess needs, and plan upgrades and migrations.


published by Eugenia on 2015-07-19 09:23:31 in the "Hardware" category
Eugenia Loli-Queru

As I wrote in the past, my first job out of college was to work in an artificial intelligence project. The idea back then among engineers in the field was that, given enough smart programming and a tree-like knowledge database, we would eventually manage to create true artificial intelligence (AI). When I was working on the project, I truly believed in it. It was the coolest thing in my life up to that point! As the years went by, I realized that AI via that model, is nothing but a fantasy. A fantasy circulating in the scientific circles since the late 1960s.

These days, there are a lot of people who have gone obsessed with the idea of the singularity (e.g. Ray Kurzweil), while others are sh1tting their pants about how “disastrous” that could be for the human race (Elon Musk being one of them).

Artificial intelligence has progressed since the 1990s, since now it’s not modeled around programming the knowledge directly into it, but it “learns” via the Internet. Meaning, by analyzing behaviors and actions of internet users, Google can “guess” what’s the most logical action for their AI to take for each given situation (crowd-sourcing the intelligence). Crowd-sourcing is a far smarter way to have your AI “learn”, than the “static” ways of teaching an AI system word-by-word as in the past.

However, even with crowd-sourcing, we will hit a wall in what AI can do for us via that method. It can surely become as “smart” as the Star Trek’s Enterprise computer, which is not very smart. What we really want when we talk about AI, is Mr Data, not the rather daft Enterprise computer. So we need a new model to base the design of our machines. Crowd-sourcing comes close, but it’s an exogenous way of evolution (because it takes decisions based on actions already taken by the people it crowd-sources from), and so it can never evolve into true consciousness. True consciousness means free will, so a crowd-sourcing algorithm can never be “free”, it will always be a slave to the actions it was modeled around for. In other words, that crowd-sourcing AI would always behave “too robotic”.

It took me 20 years to realize why AI hasn’t work so far, and why our current models will never work. The simple reason being: just like energy, you can’t create consciousness from nothing, but you can always divide and share the existing one.

The idea for this came during my first lucid dream almost exactly 2 years ago, when I first “met” my Higher Self (who names himself Heva or Shiva). In my encounter with him, which I described in detail here, I left one bit of information out from that blog post, because I promised him that I won’t tell anyone. I think it’s time to break that promise though, because it’s the right time for that information to be shared. Basically, when I met him, he had a laptop with him, and in its monitor you could see a whole load of live statistics about me: anger, vitality, love, fear etc etc etc. Please don’t laugh on why a “spiritual entity” had a… laptop. You must understand that dreams are symbolic and are modeled around the brain of the dreamer (in this case, a human’s dream, who is only familiar with human tech). So what looked like a laptop to me, from his point of view, was some 5D ultra-supa tech that would look like psychedelic colors, who knows? So anyway, he definitely tried to hide from me that screen. I wasn’t supposed to look at it. Oops.

For a few months, I was angry with that revelation: “Why am I being monitored?”, “why am I not truly free?”, “am I just a sack of nuts & bolts to him?”.

Eventually, during my quest to these mystical topics, I realized what I am: I’m an instance of consciousness, expressed in this life as a human. I’m just a part of a Higher Self (who has lend his consciousness to many life forms at once), and he himself is borrowing his consciousness from another, even higher (and more abstract) entity, ad infinitum, until you go so high up in the hierarchy, that you realize that everything is ONE. In other words, in some sense, I’m a biological robot, that Heva has lend me some consciousness so he can operate it (and Heva is the same too, for the higher entity that operates him, ad infinitum).

So, because “as above, so below”, I truly believe that the only way for us to create true AI in this world, is to simply lend our consciousness to our machines. The tech for that is nearly here, we already have brain implants that can operate robotic arms, for example. I don’t think we’re more than 20-30 years away from a true breakthrough on that front, that could let us operate machines with very little effort.

The big innovation in this idea is not that a human can sit on a couch and operate wirelessly a robot-janitor. The big idea is that one human, could operate 5 to 20 robots at the same time! This could create smart machines in our factories (or in asteroids, mining) by the billions, which could absolutely explode the amount of items produced, leading in an exploding growth (economic, scientific, social).

5 to 20 robots at the same time, you say? Well, look. Sure, it’s a myth that humans only use 10% of their brains. However, when a human, let’s say, sweeps the floor, he/she doesn’t use that much brain power. In fact, the most processing power is used to use the body, not to actually take logical decisions (e.g. sweep here, but not here). That part, is taken care by the robotic body and its firmware (aka instinct), which means that if, for example, we need 30% of brain power to sweep the floor with our own body, we might need only a 5% to do the same thing with a robotic body. In other words, we outsource the most laborious part to the machine, and we only lend the machine just as much smartness as it needs to operate.

I know that this article will be laughed at from the people who are purely scientific, after reading these “spiritual nonsense” above, however, you can’t argue that there’s some logic on these AI implementation suggestions that could work, and that could revolutionize our lives. Interestingly, I haven’t seen any robotic/AI projects using that idea yet. There are a few sci-fi books that have touched on the idea of operating a machine via brainwaves, but they haven’t gotten through to the significance of it (otherwise they would have described a world much different than the rather traditional one they try to sell us), and they haven’t even gotten the implementation right either. “Avatar” came close, but its implementation is too literal and limiting (one human becomes one alien, 1-to-1 exchange).

An FAQ on why I base my AI theory on “as above so below”:

Q: If all consciousness in all living things is borrowed, why aren’t we all smarter?

A: The amount of consciousness in a species is limited by the brain that species carries. Just like if you could lend your consciousness to a cleaning robot (that has a firmware specifically made for cleaning jobs, plus a fixed processing power), you would only be able to lend it as much as it can carry. Not very much, that is.

Also, these robots could make mistakes, since humans themselves make mistakes, as they now operate using our intelligence. But the mistakes will be limited, because that consciousness would only operate based on the brain’s (CPU/firmware) limitations. Additionally, having stupid machines with no errors, and much smarter ones with few errors, the choice is obvious for what the market would go for.

Q: If Higher Selves exist, why wait for billions of years until humans evolved to be smarter and to have enough capacity for consciousness? Sounds like a waste of time.

A: Anyone who has done a DMT trip or two will tell you that time is an illusion. Time exists only for us, who live in this simulation/holographic universe. Outside of this experience, there is no time. Everything happens at once.

Additionally, having a natural evolution program running is much more efficient, flexible and expanding than creating “fixed” robots. So it’s well worth the wait.

Here, we must also understand that our master reality (the one that we’re borrowing our consciousness from) is also creating our universe. The universe is simulated, and the computing power behind them is their minds. After eons, life evolves, and then their consciousness can take a more active role in that universe.

Q: Why can’t we remember who we truly are?

A: It’s a matter of efficiency, and also it’s about how much consciousness our brains can hold (e.g. humans are more self-aware than flies). To survive in this world, we need an “ego” (both with the Buddhism and Freudian/Jungian meaning). The ego believes that he or she is a human, she’s got a name, a job, a favorite movie, a life here. It’s that ego that keeps us grounded here to “do our job” (for whatever we meant to do). So we create this mask, this fake self, so we can survive here. The ego is also the one that keeps us in life with such perseverance. When under meditation or entheogens we can experience “ego death” (the mask falling off), then we can truly know that we’re everything that is, ever was and ever will be. In that state, we’re just consciousness, pure being. Unfortunately, that also would mean that we can’t operate in this world. So the ego is a safeguard for efficiency to stay grounded in this reality. It’s useful to experience ego death a few times in one’s life, in order to get perspective, but for our everyday life, we need it.

Q: Why our Higher Self hooked himself up and “re-incarnated” itself (aka lend consciousness) in many different life forms?

A: For the same reasons we will do so, when we have the technological capability: work, learning, research, fun, producing food, more fun… And of course, for no reason at all too. Sometimes, things just are. Natural evolution of things. For the same reason every species wants babies, maybe.

After eons of evolution, our own “robots” should be able to lend their consciousness to their own “robots”. It’s a dynamic system that happens automatically in the evolutionary process, until it reaches its most basic levels of consciousness and the simplest of universes (1D universes). Just as you can’t create energy, you can’t create grow consciousness. You can only divide from that. As such, every time a given realm creates their own versions of existence/ simulated universe, by definition these new universes are more rule-based and more rigid than their master’s universe. That’s why “higher dimensions” are seen as ever-changing while on DMT trips (entities and things constantly in 5D flux), while further down the line in the consciousness tree, the universes there have more laws that keep things solid and seemingly unchanging.

Q: Can we be hacked then?

A: YES. In a number of lucid dreams I was installed devices in me, or was “modified” in some way by entities that clearly weren’t human. Others on DMT will tell you the same thing too. Sometimes we’re upgraded by the “good” guys, sometimes by the “bad” guys. Just like in any digital system. What can ya do?

Equally possible is that other individual instances of consciousness can inject themselves into your body and take it over for a while. That should still be considered a hack.

Q: Lucid dreams? So why do we sleep or dream?

A: For two reasons: because both the body needs it (it must recharge in standby mode, while it runs a few daemons to fsck the file system, among others), and because the borrowed consciousness must take a break too.

Let’s assume that you were just driving a car. After a few hours, you would need to take a break — even if the car could still go on for many more hours without filling up with gas. If you are the consciousness that drives, and the car is the robot that is operated by that consciousness, you realize that the consciousness will need a break much earlier than the car itself would need one!

When we dream, our consciousness operates in levels of existence similar to that of our Higher Self, but because our consciousness is still “glued” to the human body, we interpret everything based on our human experiences. Most of these dreams are nothing but programs, designed to teach us things. In a recent dream, I was in a bus, among strangers. Then, out of nowhere, I became lucid. Suddenly, the people in the bus, looked at me not as strangers anymore, but as participants in a dream that they had programmed and played roles for me. But after the curtains fell, and I knew I was dreaming and so I wasn’t getting “no” for an answer, they were quickly answering for me questions like “why we dream”, and what is “the hierarchy of the cosmos”. Most of the time, they speak in symbolism that only I can interpret, but some times they’re very direct.

Some of these entities seen in dreams are “spirit guides”, which probably aren’t very different to what “IT personnel” are for our computers at the office. No wonder why spirit guides can change throughout our lives, why there is more than one of them (although there’s usually a main entity ordering the others around), and why Esther (my spirit guide) once told me: “what do you want again, calling me for second time in this session? I’m on lunch”. LOL. And yes, they can get pissed off too (I made them so a few months ago). But they can also give you some super-important info too (e.g. they showed me that my business Instagram account was under attack, and lo-and-behold, within a week, it happened).

People on DMT can de-couple from their bodies and shoot much further than dreams allow (dreams are controlled stand-by versions of experiencing the other realms, while DMT has no limits of where it can land you). While on DMT/LSD/Shrooms/etc you can become one with your higher self, or with everything, visit other realms etc. However, when you come back to your body/brain, you “forget” most of what you saw, or they suddenly become incomprehensible, because as I said above, your brain has a limited amount of function and capacity.

Q: If these other realms exist, why can’t we see them?

A: Our brain is a filtering mechanism. We live in a parallel existence, just like part of our consciousness inside a robot would experience its existence as separate than that of its human master. Inside the mind of the robot, it doesn’t perceive its external world the same way a human does (even if it has HD cameras, that “sees” the world as we do, the digital processing that ensues FILTERS out many things, because they’re not relevant to its function). Again, people who do low dose shrooms, LSD, or DMT, will start perceiving their own room as looking different (e.g. full of pixels, or full of colors that don’t exist etc), and often, on slightly higher dosage, they will see entities in that room that normally wouldn’t be able to see with their eyes (and yet, most dogs can, according to reports, since their brains have evolved differently). So basically, remove the (brain) filter for a while, and enjoy an upgraded version of reality.

Q: So if our brain can’t see these other realms, why can’t our machines/cameras see them either?

A: Some can. It’s just that seeing further in the light spectrum doesn’t mean that you can also humanly interpret it as a separate realm with its own intelligence. We’re still inside a box (a specific type of simulated universe that is computed by our own collective consciousness) and trying to see outside of it, it has its limitations. The only way to take a peek outside of it, is if you temporarily decouple your consciousness from your body (e.g. via DMT), so you’re not actively participating in the creation of this universe, but you’re free to explore other universes. The problem with DMT is that it can land you anywhere… including “hellish” worlds. It’s not predictable at all, it’s a crapshoot.

Q: Ok, so what’s the point of our life then?

A: The meaning of life is life itself. Or, no meaning at all. You give it one. Or not.

Q: So, is my life insignificant?

A: Yes and no. Yes, if you think yourself in the vast cosmos from the human, cynical point of view (everything is a point of view). And no, because you were “built” for a function, that you do perform, even if you don’t know it (even if you spend your life watching TV all day, you could still perform a specific function in another level of reality without knowing it — universes aren’t independent of each other).

You would only feel “small” after reading all this if you’re a selfish a$$hole. If you embrace that you’re everything instead, then all existential problems vanish.

Q: So why everything exists then?

A: In the high level of things (after you sum up all the hierarchical entities of consciousness), there’s only the ONE. One single consciousness, living inside the Void. Nothing else exists. It has decided that dividing itself in a top-down fashion, creating many parallel worlds and universes and lending its consciousness to its living things, for the sake of experience itself, was the best action to take. In reality, everything is just a dream in the mind of that ONE consciousness. And Everything is as it should be.

Each individual down the hierarchy of existence is differently-built and under fluctuating circumstances, and therefore each of these individuals provide a different point of view on the things it can grasp. It’s that different point of view that the system is after: novelty. Novelty == new experiences for the ONE.

Q: Why is there evil in the world?

A: There is no “good” and “evil”. Everything is just a perspective. For the human race for example, eating babies is a terrible thing. For the Komodo dragons (that routinely eat their babies), it’s just another day.

Also, in order for LOVE to exist (love == unity of all things), FEAR must also exist (fear == division of all things). One can’t exist without the other. And it’s being decided by this ONE consciousness that it’s better to experience SOMETHING, than to experience NOTHING for all eternity.

Q: Does this ONE consciousness answer to prayers?

A: No, it doesn’t give a sh1t, grow a pair. From its point of view, everything that can happen, must happen, so it can be experienced (including very “bad” things). The ONE is an abstract construct, not a person. It’s you, and me, and everything else as pure BEING.

published by Eugenia on 2015-07-15 18:36:25 in the "Politics" category
Eugenia Loli-Queru

An open letter to the Greek government:

It’s obvious to me that more preparations should have been done to switch currencies. It’s obvious that the European situation had no remedy, no matter the amount of negotiations that would take place. I believe that the way out of this tomb is to innovate. You can’t use old methods to fix old errors. You need new methods. And technology can provide those.

I know that printing a new currency would be difficult, and even more difficult would be to build trust. I’d suggest this: No actual printing of new currency. Make everything electronic. I’m NOT talking about bitcoin (which isn’t modernized in principle, as Mr Varoufakis has pointed out). I’m talking about people paying with credit cards (for large amounts) and smartphones using NFC (Near Field Communication) for up to 20 drachma (I’m assuming a parity with the Euro, at least in the beginning). Regarding notes, these can be in euros (since they’re already out there circulating anyway).

People WILL trust this new form of paying because of the same reason people in the States prefer to use credit cards: they don’t see the money, so they feel that they didn’t pay much for the goods. It’s a psychological trick. In fact, the NFC trick, where you simply touch your phone to pay a micro-payment up to 20 drachma/euro, is even more painless. People will use that, even if for convenience. Most people already have Android smartphones in Greece, and one can buy such a phone for 100 Euros today (Archos brand). Within a few months, people will be using this new system, and they will even like it. They won’t even realize that there was a change in currency. Overtime, all the euros will end up in banks, especially if you offer them a sweet buy-back program.

Now, regarding corruption in Greece: Since all businesses and micropayments in shops would have to happen via NFC/cards, it would be very unlikely that people will be able to not pay their taxes. In fact, the taxes could be calculated automatically in such a system. Who needs an accountant (that most people can’t even afford), when the system takes care of that for you?

And regarding the rest kind of corruption (e.g. fakellaki, rousfetia etc), since everyone would now have these smartphones, they should be fully supported by law to RECORD every chat they have with civil servants, taxi drivers, doctors, police etc etc. And I don’t mean just audio recording, but full video recording. Who ever doesn’t accept the new kind of currency, or wants “extra”, they should be admitted to court, given the video proof. Russia does the same with the dash cams for car insurance purposes. It works.

I would like to re-iterate: you have an old Europe in new lipstick, and an old style financial problem (debt). The problems Greece has are not novel, they are in fact the status quo throughout history. So in order to go around these problems, you need the tools of today, not old tactics of the yesteryear (be it diplomacy, negotiations, politics). It needs swift action, and mobilization of technology.

published by (Greg Sabino Mullane) on 2015-07-15 18:36:00 in the "database" category

Being able to disable Postgres triggers selectively can be an important skill when doing tasks like bulk updates, in which you only want a subset of the triggers on the table to be fired. Read below for the long explanation, but the TL;DR version of the best solution is to set a WHEN clause on the trigger you wish to skip, making it conditional on a variable such as session_replication_role, or application_name

  ROW WHEN (current_setting('session_replication_role') <> 'local') EXECUTE PROCEDURE myfunc();
SET LOCAL session_replication_role = 'local';
UPDATE foobar SET baz = 123;

I decided to spin up a free Heroku "Hobby Dev" database to illustrate the solutions. Generating a test table was done by using the Pagila project, as it has tables which contain triggers. Heroku gives you a randomly generated user and database name. To install the Pagila schema, I did:

$ export H="postgres://"
$ cd pagila-0.10.1
$ psql $H -q -f pagila-schema.sql
$ psql $H -q -f pagila-data.sql

Errors appeared on the import, but they can be safely ignored. One error was because the Heroku database does not have a user named "postgres", and the other error was due to the fact that the Heroku user is not a superuser. The data, however, was all intact. The sample data is actually quite funny, as the movie titles were semi auto-generated at some point. For example, seven random movie descriptions:

  • A Brilliant Panorama of a Madman And a Composer who must Succumb a Car in Ancient India
  • A Touching Documentary of a Madman And a Mad Scientist who must Outrace a Feminist in An Abandoned Mine Shaft
  • A Lackluster Reflection of a Eskimo And a Wretch who must Find a Fanny Pack in The Canadian Rockies
  • A Emotional Character Study of a Robot And a A Shark who must Defeat a Technical Writer in A Manhattan Penthouse
  • A Amazing Yarn of a Hunter And a Butler who must Defeat a Boy in A Jet Boat
  • A Beautiful Reflection of a Womanizer And a Sumo Wrestler who must Chase a Database Administrator in The Gulf of Mexico
  • A Awe-Inspiring Reflection of a Waitress And a Squirrel who must Kill a Mad Cow in A Jet Boat

The table we want to use for this post is named "film", and comes with two triggers on it, 'film_fulltext_trigger', and 'last_updated':

heroku=> d film
                            Table ""
        Column        |            Type             |       Modifiers
 film_id              | integer                     | not null default 
 title                | character varying(255)      | not null
 description          | text                        | 
 release_year         | year                        | 
 language_id          | smallint                    | not null
 original_language_id | smallint                    | 
 rental_duration      | smallint                    | not null default 3
 rental_rate          | numeric(4,2)                | not null default 4.99
 length               | smallint                    | 
 replacement_cost     | numeric(5,2)                | not null default 19.99
 rating               | mpaa_rating                 | default 'G'::mpaa_rating
 last_update          | timestamp without time zone | not null default now()
    film_fulltext_trigger BEFORE INSERT OR UPDATE ON film FOR EACH ROW EXECUTE 
       PROCEDURE tsvector_update_trigger('fulltext', 'pg_catalog.english', 'title', 'description')
    last_updated BEFORE UPDATE ON film FOR EACH ROW EXECUTE PROCEDURE last_updated()

The last_updated trigger calls the last_updated() function, which simply sets the last_update column to CURRENT_TIMESTAMP, which is often seen as its shorter-to-type form, now(). This is a handy metric to track, but there are times when you want to make changes and not update this field. A typical example is some sort of bulk change that does not warrant changing all the rows' last_update field. How to accomplish this? We need to ensure that the trigger does not fire when we do our UPDATE. The way many people are familiar with is to simply disable all triggers on the table. So you would do something like this:

UPDATE film SET rental_duration = 10;

When using Heroku, you are given a regular user, not a Postgres superuser, so the above will generate an error that looks like this:

ERROR:  permission denied: "RI_ConstraintTrigger_a_88776583" is a system trigger.

This is caused by the failure of a normal user to disable the internal triggers Postgres uses to maintain foreign key relationships between tables. So the better way is to simply disable the specific trigger like so:

UPDATE film SET rental_duration = 10;

This works on Heroku, but there are two major problems with the ALTER TABLE solution. First, the ALTER TABLE will take a very heavy lock on the entire table, meaning that nobody else will be able to access the table - even to read it! - until your transaction is complete (although Postgres 9.5 will reduce this lock!). The other problem with disabling triggers this way is that it is too easy to accidentally leave it in a disabled state (although the check_postgres program has a specific check for this!). Let's take a look at the lock, and double check that the trigger has been disabled as well:

heroku=> SELECT last_update FROM film WHERE film_id = 123;
 2015-06-21 16:38:00.891019
heroku=> BEGIN;
heroku=> ALTER TABLE film DISABLE TRIGGER last_updated;
heroku=> SELECT last_update FROM film WHERE film_id = 123;
heroku=> UPDATE film SET rental_duration = 10;
-- We need the subselect because we share with a gazillion other Heroku databases!
heroku=> select relation::regclass,mode,granted from pg_locks where database = 
heroku->   (select oid from pg_database where datname = current_database());
 relation |        mode         | granted 
 pg_locks | AccessShareLock     | t
 film     | RowExclusiveLock    | t
 film     | AccessExclusiveLock | t  ## This is a very heavy lock!
## Version 9.5 and up will have a ShareRowExclusive lock only!
heroku=> ALTER TABLE film ENABLE TRIGGER last_updated;
heroku=> COMMIT;

-- This is the same value, because the trigger did not fire when we updated
heroku=> select last_update FROM film WHERE film_id = 123;
 2015-06-21 16:38:00.891019

What we really want is to use the powerful session_replication_role parameter to safely disable the triggers. The problem is that the canonical way to disable triggers, by setting session_replication_role to 'replica', will disable ALL triggers and rules, for ALL tables. This is not wanted. In our example, we want to stop the last_updated trigger from firing, but also want all the other user triggers to fire, as well as the hidden system triggers that are enforcing foreign key referential integrity.

You can set session_replication_role to one of three values: origin (the default), local, and replica. Setting it to "replica" is commonly used in replication systems such as Bucardo and Slony to prevent all rules and triggers from firing. It can also be used for careful bulk loading. Only triggers explicitly set as "replica triggers" will fire when the session_replication_role is set to 'replica'. The local setting is a little harder to understand, as it does not have a direct mapping to a trigger state, as 'origin' and 'replica' do. Instead, it can be thought of as an alias to 'origin' - same functionality, but with a different name. What use is that? Well, you can check the value of session_replication_role and do things differently depending on whether it is 'origin' or 'local'. Thus, it is possible to teach a trigger that it should not fire when session_replication_role is set to 'local' (or to fire only when it is set to 'local').

Thus, our previous problem of preventing the last_updated trigger from firing can be solved by careful use of the session_replication_role. We want the trigger to NOT fire when session_replication_role is set to 'local'. This can be accomplished in two ways: modification of the trigger, or modification of the underlying function. Each has its strengths and weaknesses. Note that session_replication_role can only be set by a superuser, which means I'll be switching from Heroku (which only allows connecting as a non-superuser) to a local Pagila database.

For the modify-the-function route, add a quick block at the top to short-circuit the trigger if the session_replication_role (srr) is set to 'local'. An advantage to this method is that all triggers that invoke this function will be affected. In the pagila database, there are 14 tables that have a trigger that calls the last_updated function. Another advantage is that the exception to the function firing is clearly visible in the functions definition itself, and thus easy to spot when you examine the function. Here is how you would modify the last_updated function to only fire when in 'local' srr mode:

CREATE OR REPLACE FUNCTION public.last_updated()
LANGUAGE plpgsql
AS $bc$
  IF current_setting('session_replication_role') = 'local' THEN

  NEW.last_update = CURRENT_TIMESTAMP;

To invoke it, we change session_replication_role (temporarily!) to 'local', then make our changes. Observe how the value of last_update does not change when we are in 'local' mode:

pagila=# show session_replication_role tg

pagila=# begin;
pagila=# select last_update from film where film_id = 203;
 2015-06-21 16:38:00.711411

pagila=# update film set rental_duration = 10 WHERE film_id = 203;
pagila=# select last_update from film where film_id = 203;
 2015-06-21 16:38:03.543831
pagila=# commit;

pagila=# begin;
pagila=# set LOCAL session_replication_role = 'local';
pagila=# select last_update from film where film_id = 203;
 2015-06-21 16:38:03.543831
pagila=# update film set rental_duration = 10 WHERE film_id = 203;
pagila=# select last_update from film where film_id = 203;
 2015-06-21 16:38:03.543831
pagila=# commit;

pagila=# show session_replication_role;

The second method for skipping a trigger by using session_replication_role is to modify the trigger definition itself, rather than changing the function. This has the advantage of not having to touch the function at all, and also allows you to see that the trigger has been modified when doing a d of the table. Using ALTER TRIGGER only allows a rename, so we will need to drop and recreate the trigger. By adding a WHEN clause to the trigger, we can ensure that it does NOT fire when session_replication_role is set to 'local'. The SQL looks like this:

pagila=# begin;
pagila=# drop trigger last_updated ON film;
pagila=# create trigger last_updated before update on film for each row 
pagila-#   when (current_setting('session_replication_role') <> 'local') execute procedure last_updated();
pagila=# commit;

Voila! As before, we can test it out by setting session_replication_role to 'local' and confirming that the function does not modify the last_update column. Before doing that, let's also change the function back to its original form, to keep things honest:

-- Restore the original version, with no session_replication_role logic:
pagila=# CREATE OR REPLACE FUNCTION public.last_updated() RETURNS TRIGGER LANGUAGE plpgsql

-- Normal update will change the last_update column:
pagila=# select last_update from film where film_id = 203;
 2015-06-21 16:38:00.121011

pagila=# update film set rental_duration = 10 WHERE film_id = 203;
pagila=# select last_update from film where film_id = 203;
 2015-06-21 16:38:03.011004

pagila=# begin;
pagila=# set LOCAL session_replication_role = 'local';
pagila=# update film set rental_duration = 10 WHERE film_id = 203;
pagila=# select last_update from film where film_id = 203;
 2015-06-21 16:38:03.011004

-- Show that we are not holding a heavy lock:
pagila=# select relation::regclass,mode,granted from pg_locks where relation::regclass::text = 'film';
 relation |       mode       | granted 
 film     | AccessShareLock  | t
 film     | RowExclusiveLock | t

pagila=# commit;

Those are the three main ways to selectively disable a trigger on a table: using ALTER TABLE to completely disable it (and invoking a heavy lock), having the function check session_replication_role (affects all triggers using it, requires superuser), and having the trigger use a WHEN clause (requires superuser). Sharp readers may note that being a superuser is not really required, as something other than session_replication_role could be used. Thus, a solution is to use a parameter that can be changed by anyone, that will not affect anything else, and can be set to a unique value. Here is one such solution, using the handy "application_name" parameter. We will return to the Heroku database for this one:

heroku=> drop trigger last_updated on film;
heroku=> create trigger last_updated before update on film for each row 
  when (current_setting('application_name') <> 'skiptrig') execute procedure last_updated();

heroku=> select last_update from film where film_id = 111;
 2015-06-21 16:38:00.365103
heroku=> update film set rental_duration = 10 WHERE film_id = 111;
heroku=> select last_update from film where film_id = 111;
 2015-06-21 16:38:03.101115

heroku=> begin;
heroku=> set LOCAL application_name = 'skiptrig';
heroku=> update film set rental_duration = 10 WHERE film_id = 111;
heroku=> select last_update from film where film_id = 111;
 2015-06-21 16:38:03.101115

-- Show that we are not holding a heavy lock:
heroku=> select relation::regclass,mode,granted from pg_locks where database = 
heroku->   (select oid from pg_database where datname = current_database());
 relation |       mode       | granted 
 film     | AccessShareLock  | t
 film     | RowExclusiveLock | t

heroku=> commit;

So there you have it - four solutions to the problem of skipping a single trigger. Which to use depends on your circumstances. I prefer the WHEN + session_replication_role option, as it forces you to be a superuser, and is very visible when looking at the trigger via d.

published by (Greg Sabino Mullane) on 2015-07-01 18:22:00 in the "postgres" category

Back in the old days, upgrading Postgres required doing a pg_dump and loading the resulting logical SQL into the new database. This could be a very slow, very painful process, requiring a lot of downtime. While there were other solutions (such as Bucardo) that allowed little (or even zero) downtime, setting them up was a large complex task. Enter the pg_upgrade program, which attempts to upgrade a cluster with minimal downtime. Just how fast is it? I grew tired of answering this question from clients with vague answers such as "it depends" and "really, really fast" and decided to generate some data for ballpark answers.

Spoiler: it's either about 3.5 times as fast as pg_dump, or insanely fast at a flat 15 seconds or so. Before going further, let's discuss the methodology used.

I used the venerable pgbench program to generate some sample tables and data, and then upgraded the resulting database, going from Postgres version 9.3 to 9.4. The pgbench program comes with Postgres, and simply requires an --initialize argument to create the test tables. There is also a --scale argument you can provide to increase the amount of initial data - each increment increases the number of rows in the largest table, pgbench_accounts, by one hundred thousand rows. Here are the scale runs I did, along with the number of rows and overall database size for each level:

Effect of --scale
--scaleRows in pgbench_accountsDatabase size
10010,000,0001418 MB
15015,000,0002123 MB
20020,000,0002829 MB
25025,000,0003535 MB
30030,000,0004241 MB
35035,000,0004947 MB
40040,000,0005652 MB
45045,000,0006358 MB
50050,000,0007064 MB
55055,000,0007770 MB
60060,000,0008476 MB

To test the speed of the pg_dump program, I used this simple command:

$ pg_dump postgres | psql postgres -q -p 5433 -f -

I did make one important optimization, which was to set fsync off on the target database (version 9.4). Although this setting should never be turned off in production - or anytime you cannot replace all your data, upgrades like this are an excellent time to disable fsync. Just make sure you flip it back on again right away! There are some other minor optimizations one could make (especially boosting maintenance_work_mem), but for the purposes of this test, I decided that the fsync was enough.

For testing the speed of pg_upgrade, I used the following command:

$ pg_upgrade -b $BIN1 -B $BIN2 -d $DATA1 -D $DATA2 -P 5433

The speed difference can be understood because pg_dump rewrites the entire database, table by table, row by row, and then recreates all the indexes from scratch. The pg_upgrade program simply copies the data files, making the minimum changes needed to support the new version. Because of this, it will always be faster. How much faster depends on a lot of variables, e.g. the number and size of your indexes. The chart below shows a nice linear slope for both methods, and yielding on average a 3.48 increase in speed of pg_upgrade versus pg_dump:

pg_dump versus pg_upgrade
--scaleDatabase sizepg_dump
1001.4 GB210.074.72.82
1502.1 GB305.079.43.86
2002.8 GB457.6122.23.75
2503.5 GB636.1172.13.70
3004.2 GB832.2215.13.87
3504.9 GB1098.8320.73.43
4005.7 GB1172.7361.43.25
4506.4 GB1340.2426.73.15
5007.1 GB1509.6476.33.17
5507.8 GB1664.0480.03.47
6008.5 GB1927.06073.17

If you graph it out, you can see both of them having a similar slope, but with pg_upgrade as the clear winner:

I mentioned earlier that there were some other optimizations that could be done to make the pg_dump slightly faster. As it turns out, pg_upgrade can also be made faster. Absolutely, beautifully, insanely faster. All we have to do is add the --link argument. What this does is rather than copying the data files, it simply links them via the filesystem. Thus, each large data file that makes up the majority of a database's size takes a fraction of a second to link to the new version. Here are the new numbers, generated simply by adding a --link to the pg_upgrade command from above:

pg_upgrade --link is crazy fast
--scaleDatabase sizepg_upgrade --link
1001.4 GB12.9
1502.1 GB13.4
2002.8 GB13.5
2503.5 GB13.2
3004.2 GB13.6
3504.9 GB14.4
4005.7 GB13.1
4506.4 GB13.0
5007.1 GB13.2
5507.8 GB13.1
6008.5 GB12.9

No, those are not typos - an average of thirteen seconds despite the size of the database! The only downside to this method is that you cannot access the old system once the new system starts up, but that's a very small price to pay, as you can easily backup the old system first. There is no point in graphing these numbers out - just look at the graph above and imagine a nearly flat line traveling across the bottom of the graph :)

Are there any other options that can affect the time? While pgbench has a handy --foreign-keys argument I often use to generate a more "realistic" test database, both pg_dump and pg_upgrade are unaffected by any numbers of foreign keys. One limitation of pg_upgrade is that it cannot change the --checksum attribute of a database. In other words, if you want to go from a non-checksummed version of Postgres to a checksummed version, you need to use pg_dump or some other method. On the plus side, my testing found negligible difference between upgrading a checksummed versus a non-checksummed version.

Another limitation of the pg_upgrade method is that all internal stats are blown away by the upgrade, so the database starts out in a completely unanalyzed state. This is not as much an issue as it used to be, as pg_upgrade will generate a script to regenerate these stats, using the handy --analyze-in-stages argument to vacuum. There are a few other minor limitations to pg_upgrade: read the documentation for a complete list. In the end, pg_upgrade is extraordinarily fast and should be your preferred method for upgrading. Here is a final chart showing the strengths and weaknesses of the major upgrade methods.

Postgres upgrade methods compared
  • Always works
  • Battle tested
  • Slowest method
  • Maximum downtime
  • Requires lots of disk space
  • Very fast
  • --link mode super fast
  • Cannot always be used (finicky)
  • Stats are lost
  • Minimal but non-zero downtime
  • Handles complex cases
  • Zero-downtime possible
  • Complex to setup
  • Requires primary keys on large tables
  • Requires lots of disk space

(As an addendum of sorts, pg_upgrade is fantastic, but the Holy Grail is still out of sight: true in-place upgrades. This would mean dropping in a new major version (similar to the way revisions can be dropped in now), and this new version would be able to read both old and new data file formats, and doing an update-on-write as needed. Someday!)

published by (Muhammad Najmi Ahmad Zabidi) on 2015-07-01 06:52:00 in the "python" category
Recently I worked on a program which required me to filter hundred of lines of blog titles. Throughout the assignment I stumbled upon a few interesting problems, some of which are outlined in the following paragraphs.

Non Roman characters issue

During the testing session I missed one title and investigating why it happened, I found that it was simply because the title contained non-Roman characters.

Here is the code's snippet that I was previously using:

for e in results:                                                                                                                        
    if freqs.get(simple_author,0) < 1:                                                                                               
        print parse(e['published']).strftime("%Y-%m-%d") , "--",simple_author, "--", e['title']

And here is the fixed version

for e in results:                                                                                                                        
    if freqs.get(simple_author,0) < 1:                                                                                               
        print parse(e['published']).strftime("%Y-%m-%d") , "--",simple_author, "--", e['title'].encode('UTF-8') 

To fix the issue I faces I added .encode('UTF-8') in order to encode the characters with the UTF-8 encoding. Here is an example title that would have been otherwise left out:

2014-11-18 -- Unknown -- Novo website do Liquid Galaxy em Português!

Python 2.7 uses ASCII as its default encoding but in our case that wasn't sufficient to scrape web contents which often contains UTF-8 characters. To be more precise, this program fetches an RSS feed in XML format and in there it finds UTF-8 characters. So when the initial Python code I wrote met UTF-8 characters, while using ASCII encoding as the default sets, it was unable to identify them and returned an error.

Here is an example of the parsing error it gave us while fetching non-roman characters while using ASCII encoding:
UnicodeEncodeError: 'ascii' codec can't encode character u'xea' in position 40: ordinal not in range(128)

Right and Left text alignment

In addition to the error previously mentioned, I also had the chance to dig into several ways of formatting output.
The following format is the one I used as the initial output format:

Name                                                     Age

Using "ljust" and "rjust" method

I want to improve the readability in the example above by left-justify "Name" by 30 characters and "Age" by another 30 characters distance.

Let's try with the '*' fill character. The syntax is str.ljust(width[, fillchar])

Name**************************                           Age

And now let's add .rjust:


By using str, it counts from the left by 30 characters including the word "Name" which has four characters
and then another 30 characters including "Age" which has three letters, by giving us the desired output.

Using "format" method

Alternatively, it is possible to use the same indentation approach with the format string method:

print("{!s:<{fill}}{!s:>{fill}}".format("Name", "Age",fill=30))
Name                                                     Age

And with the same progression, it is also possible to do something like:

print("{!s:*<{fill}}{!s:>{fill}}".format("Name", "Age",fill=30))
Name**************************                           Age
print("{!s:*<{fill}}{!s:#>{fill}}".format("Name", "Age",fill=30))

"format" also offers a feature to indent text in the middle. To put the desired string in the middle of the "fill" characters trail, simply use the ^ (caret) character:

Feel free to refer the Python's documentation on Unicode here:

And for the "format" method it can be referred here:

published by (Jeff Boes) on 2015-06-26 13:30:00 in the "ajax" category

Perl POD is a handy, convenient, but low-tech approach to embedded documentation. Consider a web service in Dancer:

get time => sub {
  return scalar(localtime());

(Disclaimer: my actual use-case of this technique was even more legacy: I was documenting Interchange Actionmaps that returned images, JSON, etc.)

Your application might have several, or even dozens of these, with various parameters, returning data in JSON or TXT or CSV or who-knows-what. I chose to document these in Perl POD (Plain Old Documentation) format, e.g.,


=head1 time

Retrieves the current time

=over 3

=item Parameters


=item Example

=begin html

=end html



This block gets inserted right in-line with the web service code, so it's immediately obvious to anyone maintaining it (and thus has the best chance of being maintained if and when the code changes!). Now I can generate an HTML page directly from my Perl code:

$ pod2html

Your output looks something like this (excerpted for clarity):


Retrieves the current time




Where the magic comes in is the Javascript code that allows an in-line example, live and accurate, within the documentation page. You'll actually get something more like this:


Retrieves the current time



(results appear here)

Note that the code I have below is not factored by choice; I could move a lot of it out to a common routine, but for clarity I'm leaving it all in-line. I am breaking up the script into a few chunks for discussion, but you can and should construct it all into one file (in my example, "js/example-time.js").

/* example-time.js */
" + /* Note 1 */ "" + "" + "
" + "
" );

Note 1: This being a painfully simple example of a web service, there are no additional inputs. If you have some, you would add them to the HTML being assembled into the <form> tag, and then using jQuery, add them below to the url parameter, or into the data structure as required by your particular web service.

This step just inserts a simple <form> into the document. I chose to embed the form into the Javascript code, rather than the POD, because it reduces the clutter and separates the example from the web service.

    var $form = $('form[action="/time"]');
          'url': $form.attr('action') /* Note 1 also */,
          'data': {},
          'dataType': 'text',
          'async': false,

Here we have a submit handler that performs a very simple AJAX submit using the form's information, and upon success, inserts the results into a result <div> as a pre-formatted block. I added a "json" class which just tweaks the font and other typographic presentation a bit; you can provide your own if you wish.

I'm aware that there are various jQuery plug-ins that will handle AJAX-ifying a form, but I couldn't get the exact behavior I wanted on my first tries, so I bailed out and just constructed this approach.

                 $('#time-result').html('Error retrieving data!')
/* */

(That stray-looking comment above is just a work-around for the syntax highlighter.)

Error handling goes here. If you have something more comprehensive, such as examining the result for error codes or messages, this is where you'd put it.

      return false;

And just a bit of UI kindness: we have a "hide" button to make the example go away. Some of my actual examples ran to dozens of lines of JSON output, so I wanted a way to clean up after the example.

published by (Kannan Ponnusamy) on 2015-06-18 18:14:00 in the "ipython" category
Recently I have been working on Python automation scripts. Very often I use IPython to develop/debug the code.
IPython is an advanced interactive python shell. It is a powerful tool which has many more features. However, here I would like to share some of the cool tricks of IPython.

Getting help

Typing object_name? will print all sorts of details about any object, including docstrings, function definition lines (for call arguments) and constructor details for classes.
In [1]: import datetime
In [2]: datetime.datetime?
datetime(year, month, day[, hour[, minute[, second[, microsecond[,tzinfo]]]]])

The year, month and day arguments are required. tzinfo may be None, or an
instance of a tzinfo subclass. The remaining arguments may be ints or longs.
File:      /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/
Type:      type

Magic commands


This will bring up an editor to type multiline code and execute the resulting code.
In [3]: %edit
IPython will make a temporary file named: /var/folders/xh/2m0ydjs51qxd_3y2k7x50hjc0000gn/T/ipython_edit_jnVJ51/
In [3]: %edit -p
This will bring up the editor with the same data as the previous time it was used or saved. (in the current session)

Run a script

This will execute the script and print the results.
In [12]: %run
Current date and time:  2015-06-18 16:10:34.444674
Or like this:  15-06-18-16-10
Week number of the year:  24
Weekday of the week:  4


Activate the interactive debugger.
In [15]: %run
Current date and time:  2015-06-18 16:12:32.417691
Or like this: ---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/Users/kannan/playground/ in ()
      4 print "Current date and time: " ,
----> 5 print "Or like this: " ,datetime.datetime.strftime("%y-%m-%d-%H-%M")
      6 print "Week number of the year: ","%W")
      7 print "Weekday of the week: ","%w")

TypeError: descriptor 'strftime' requires a '' object but received a 'str'

In [16]: %debug
> /Users/kannan/playground/
      4 print "Current date and time: " ,
----> 5 print "Or like this: " ,datetime.datetime.strftime("%y-%m-%d-%H-%M")
      6 print "Week number of the year: ","%W")

I made a error in the line number 5, it should have to look like this. So %debug command took me into the Python debugger.
print "Or like this: " ,"%y-%m-%d-%H-%M")


This will save the specified lines to a given file. You can pass any number of arguments separated by space.
In [21]: %save 1-2 2-3
The following commands were written to file ``:
import datetime
%edit -p


Repeat a command, or get command to input line for editing.
In [28]: %recall 21

In [29]: import datetime


Time execution of a Python statement or expression
It can be one line or multiline statement. In a one liner we can pass through multiple ones separated by semicolon.
In [33]: %timeit range(100)
1000000 loops, best of 3: 752 ns per loop

Shell Commands

Basic UNIX shell integration (you can run simple shell commands such as cp, ls, rm, cp, etc. directly from the ipython command line)

To execute any other shell commands we just need to add '!' beginning of the command line. We can assign the result of the system command to a Python variable to further use.
In [38]: list_of_files = !ls

In [39]: list_of_files


Print input history, with most recent last.
In [41]: %history 20-22
import datetime
%history ~1/4 #Line 4, from last session
This will list the previous session history.


This will upload the specifed input commands to Github?s Gist paste bin, and display the URL
It will upload the code as anonymous user
In [43]: %pastebin [-d ?Date Example?] 20-23
Out[43]: u''

For more info on this topic:

published by (Marina Lohova) on 2015-06-17 11:00:00 in the "database" category

If you need to dump the production database locally Heroku has a nice set of tools to make this as smooth as humanly possible. In short, remember these two magic words: pg:pull and pg:push. This article details the process

However, when I first tried it I had to resolved few issues.

My first problem was:

pg:pull not found

To fix this:

1. Uninstall the 'heroku' gem with

gem uninstall heroku (Select 'All Versions')

2. Find your Ruby 'bin' path by running

gem env

3. Cd to the 'bin' folder.

4. Remove the Heroku executable with

rm heroku

5. Restart your shell (close Terminal tab and re-open)

6. Type

heroku version
you should now see something like:
heroku-toolbelt/2.33.1 (x86_64-darwin10.8.0) ruby/1.9.3

Now you can proceed with the transfer:

1. Type

heroku config --app production-app

Note the DATABASE_URL, for example let's imagine that the production database url is HEROKU_POSTGRESQL_KANYE_URL, and the staging database url is HEROKU_POSTGRESQL_NORTH

2. Run

heroku pg:pull HEROKU_POSTGRESQL_KANYE rtwtransferdb --app production-app
heroku config --app staging-app
heroku pg:push rtwtransferdb HEROKU_POSTGRESQL_NORTH --app rtwtest

This is when I hit the second problem:

database is not empty

I fixed it by doing:


Happy database dumping!

published by (Greg Davidson) on 2015-06-11 12:14:00 in the "html5" category

Debugging Broken Maps

A few weeks ago I had to troubleshoot some Google Maps related code that had suddenly stopped working. Some debugging revealed the issue: the code adding markers to the page was attempting to access properties that did not exist. This seemed odd because the latitude and longitude values were the result of a geocoding request which was completing successfully. The other thing which stood out to me were the property names themselves:

var myLoc = new google.maps.LatLng(results[0].geometry.location.k, results[0].geometry.location.D);

It looked like the original author had inspected the geocoded response, found the 'k' and 'D' properties which held latitude and longitude values and used them in their maps code. This had all been working fine until Google released a new version of their JavaScript API. Sites that did not specify a particular version of the API were upgraded to the new version automatically. If you have Google Maps code which stopped working recently this might be the reason why.

The Solution: Use the built-in methods in the LatLng class

Screen Shot 2015 06 10 at 3 47 32 PM

I recalled there being some helper methods for LatLng objects and confirmed this with a visit to the docs for the LatLng class which had recently been updated and given the Material design treatment — thanks Google! The lat() and lng() methods were what I needed and updating the code with them fixed the issue. The fixed code was similar to this:

var myLoc = new google.maps.LatLng(results[0], results[0].geometry.location.lng());

Digging Deeper

I was curious about this so I mapped out the differences between the three latest versions of the API:

API Version Latitude Property Longitude Property Constructor Name
3.21.x (experimental) A F rf
3.20.x (release) A F pf
3.19.x (frozen) k D pf

It seems to me that the property name changes are a result of running the Google Maps API code through the Closure Compiler. Make sure to use the built-in lat() and lng() methods as these property names are very likely to change again in future!

published by (Zdenek Maxa) on 2015-06-09 19:40:00 in the "Chef" category

This post describes some of our experiences at End Point in designing and working on comprehensive QA/CI facilities for a new system which is closely related to the Liquid Galaxy.

Due to the design of the system, the full deployment cycle can be rather lengthy and presents us with extra reasons for investing heavily in unit test development. Because of the very active ongoing development on the system we benefit greatly from running the tests in an automated fashion on the Jenkins CI (Continuous Integration) server.

Our Project's CI Anatomy

Our Jenkins CI service defines 10+ job types (a.k.a. Jenkins projects) that cover our system. These job types differ as far as source code branches are concerned, as well as by combinations of the types of target environments the project builds are executed on.

The skeleton of a Jenkins project is what one finds under the Configure section on the Jenkins service webpage. The source code repository and branch are defined here. Each of our Jenkins projects also fetches a few more source code repositories during the build pre-execution phase. The environment variables are defined in a flat text file:

Another configuration file is in the JSON format and defines variables for the test suite itself. Furthermore, we have a preparation phase bash script and then a second bash script which eventually executes the test suite. Factoring out all degrees of freedom into two pairs of externally managed (by Chef) concise files allows for pure and simple Jenkins job build definition:

It?s well possible to have all variables and content of the bash scripts laid out directly in the corresponding text fields in the Jenkins configuration. We used to have that. It?s actually a terrible practice and the above desire for purity comes from a tedious and clumsy experience that changing a variable (e.g. an URL or such) in 10+ job types involves an unbearable amount of mouse clicking through the Jenkins service webpage. Performing some level of debugging of the CI environment (like when setting up ROS stack which the project depends on) one is in for repetitive strain injury.

In essence, keeping knowledge about job types on the Jenkins server itself at a minimum and having it managed externally serves us well and is efficient. Another step forward would be managing everything (the entire job type definition) by Chef. We have yet to experiment with the already existing Chef community cookbooks for Jenkins.

The tests themselves are implemented in Python using pytest unit testing envelope. The test cases depend on Selenium - the web automation framework. Python drives the browser through Selenium according to testing scenarios, sometimes rather complex. The Selenium framework provides handles by which the browser is controlled - this includes user data input, clicking buttons, etc.

We use Selenium in two modes:
local mode: selenium drives a browser running on the Jenkins CI machine itself, locally. The browser runs in the Xvfb environment. In this case everything runs on the Jenkins master machine.
remote mode: the remote driver connects to a browser running on a remote machine (node A, B) and drives the browser there, as described in the diagram below. The test cases are run on the Jenkins slave machine located on a private network. The only difference between browser A and B is that they load their different respective Chrome extensions.

The usual unit testing assertions are made on the state or values of HTML elements in the web page.

Custom dashboard

Our Jenkins server runs builds of 10+ various job types. The builds of each type are executed periodically and the builds are also triggered by git pushes as well as by git pull requests. As a result, we get a significant number of builds on daily basis.

While Jenkins CI is extensible with very many plugins available out there, enabling and configuring a plugin gets cumbersome as the number of job types to configure rises. This is just to explain my personal aversion to experimenting with plugins on Jenkins for our project.

The Jenkins service webpage itself does not offer creating a simple aggregated view across a number of job types to allow for a simple, concise, single page view. Natively, there is just the single job type trends $JOB_URL/buildTimeTrend page (see below).

A view which immediately tells whether there is an infrastructure problem (such as loss of connectivity) or conveys straight away that everything passes on Jenkins,  seems to be missing. Such a view or feature is even more important in an environment suffering from occasional transient issues. Basically, we wanted a combination of JENKINS/Dashboard+View and JENKINS/Project+Statistics+Plugin, yet a lot simpler (see below).
So yes, we coded up our own wheel, circular just according to our liking and thus developed the jenkins-watcher application.


The application is freely available from this repository, deploys on the Google App Engine platform and so utilizes certain platform features like Datastore, Cron jobs, TaskQueue and Access Control. A single configuration file contains mainly Jenkins CI server access credentials and job type names we are interested in. The above repository merely provides a template of this (secret) config file. AngularJS is used on the frontend and a smashing Jenkins API Python library is used to communicate from Python to the Jenkins CI server through its REST API. See below the result view it provides, the screenshot is cropped to show only 5 job types and their builds within the last 24 hours:

Colour coding in green (passed), red (failed) and grey (aborted) shows a build status and is in fact just standard Jenkins colour coding. Each table row corresponds to 1 build of the build ID, build timestamp (start of the build), build duration, number of test cases which passed (P), failed (F), were skipped (S), or suffered from errors (E). The last item in the row is a direct link to the build console output, very handy for immediate inspection. In my experience, this is enough for a Jenkins babysitter?s swift daily checks. This is nothing fancy: no cool stats, graphs or plots. It is just a brief, useful overview.

The application also performs periodic checks and aborts builds which take too long (yes, a Jenkins plugin with this functionality exists as well).

For example, at a glance it?s obvious that the following failed builds suffer from some kind of transient infrastructure problems: no tests were run, nothing failed, the builds were marked as failure since some command in either their prep or build scripts failed:

Or let?s take a look at another situation proving how simple visualisation can sometimes be very useful and immediately hint-providing. We observed a test case, interestingly only on just one particular job type, which sometimes ended up with a ?Connection refused? error between the Selenium driver and the web browser (in the remote mode):

Only after seeing the failures visualized, the pattern struck us. We immediately got an idea that something is rotten in the state of Denmark shortly after midnight: from that point on, the previously mysterious issue boiled down to an erroneous cronjob command. The killall command was killing everything and not just what it was supposed to (bug filed here):

killall --older-than 2h -r chromedriver

Once we fixed the cronjob with a more complex but functional solution, without the killall command this time, so that the builds had not the chromedriver blanket pulled from under them while running, the mysterious error disappeared.

Summary, conclusion

Jenkins CI proved in general very useful for our Portal project. Keeping its configuration minimal and handling it externally worked most efficient. The custom jenkins-watcher application provides useful, aggregated, dashboard-like view. It is very easily configurable and not in any way dependent on the base project - take it for free, configure a bit and push as your own Google App Engine project. The visualisation can sometimes be a useful debugging tool.