R annoyances

R seems to be, hmm, quirky, from what I've seen of it so far... For instance, ddply has different behavior using the American vs. British spellings of "summarize". Seriously?!

> g <- ddply(m, c("QGroup","Income.Group"), summarize, All = length(CountryCode))
Error: argument "by" is missing, with no default

> g <- ddply(m, c("QGroup","Income.Group"), summarise, All = length(CountryCode))

Yes, that's right - "summarise" works fine but "summarize" apparently takes different arguments.

Trouble installing jpeg R package on OSX?

I'm taking the Coursera "Getting and Cleaning Data" class in R and one of the quizzes requires you to use the jpeg package. The R package install was failing for some reason though:

install.packages("jpeg")

In file included from read.c:1:
./rjcommon.h:11:10: fatal error: 'jpeglib.h' file not found
#include 
         ^

I use homebrew, so I tried the usual...

brew install jpeg
Warning: jpeg-8d already installed

brew unlink jpeg && brew link jpeg

But that didn't fix the issue. The solution was to install the XCode command line tools with:

xcode-select --install

(As a side note, this is my development machine so I already had XCode command line tools but for whatever reason I had to install them again, maybe an XCode upgrade?)

Sometimes the best code is that which is never written

Someone wrote some moderately over-complicated code to provide functionality in a service. That developer has moved on. A little time passes and suddenly there are a bunch of stack traces popping up in the logs indicating that a status message isn't getting published.
First reaction - dig into the error, figure out exactly what's wrong and fix it, ASAP!

Here's the error:  Invalid parameter: Message too long (Service: AmazonSNS; Status Code: 400; Error Code: InvalidParameter; Request ID: <redacted>)

Seems pretty straightforward; two seconds of Googling turns up that SNS messages have a limit of 256KB. Go into the appropriate class and add something detect how many bytes are in the message. Hmmm... maybe these messages are really large, and our app has a ton of threads - don't want to use up too much memory. A quick check on StackOverflow suggests using a CountingInputStream (there's one in the Apache commons-io, but it's also pretty trivial to write). Add it. Write a couple tests to make sure that it counts bytes correctly for UTF-8, UTF-16, etc... Looking good and have only spent 20mins so far fixing this problem.

Now that we can reliably detect the message size, we need to deal with ones that are too large for SNS... Uh-oh, the message body is actually something getting serialized into a JSON payload, we can't simply truncate or the message won't be valid JSON any more. This will require some thought, but I'm totally up for the challenge!

STOP! Take a step back from the technical details and think about what problem you're trying to solve.

Basically, the point of this code was to "make sure a notification goes out that something failed to process". There's nothing in there to suggest that the entire response needs to be serialized and sent on the wire (although that's what it was doing originally). Think to yourself, "what's the simplest thing that could work?". In this case, all that really needs to be sent is an ID and a status of success or failure:

{ id: 1234, status: "failure" }

There's NO way that message will ever approach 256KB, so do I need to worry about truncating this message? No. Looking back on it, did I need a CountingOutputStream even with the truncating solution? Probably not, it was a JSON payload in UTF-8; UTF-8 handles the ASCII range in 1 byte per character so simply getting the length of the String should have been "close enough".

In the end, I had to write a negligible amount of code to create that new JSON payload, and I removed a bunch of complexity from the original codebase along the way.

The take-away from this exercise? Step back and understand the problem-space from the start and resist the impulse to dive in and fix the errors immediately.

What's important to me, career-wise

(Continuing on from "What's important to you, career-wise?", here's my personal list of top 5 needs from a job, ranked from most important to least)

1 - Interesting people

It's important to me to work with interesting people who are more experienced/skilled/intelligent than I am. As Chad Fowler said in The Passionate Programmer: Always be the worst guy in every band you’re in. - so you can learn. The people around you affect your performance. Choose your crowd wisely. It is NOT a good idea to surround yourself with people who think identically or have had the same life experiences. Just because everyone around you hates node.js doesn't mean there aren't good ideas in it, but if there's no one championing it then it's easy to get mired in your (mis-)conceptions. Different viewpoints can challenge your perceptions and make you grow. Learn from their experiences and challenge dogma wherever you go.

In the almost 6 yrs I've been here, there's only been one person I've disliked working with; that's almost unheard of. It lends credence to the idea that hiring decisions based on culture can result in a better working environment. Unfortunately, niceness doesn't help you grow. There are people here I would love to work on projects with, but the reality is that there are no projects big or complex enough to justify us working together. Those people are a scarce resource that gets spread out to make the most use of their time - 1/5

2 - Challenge

You grow from experience and from solving problems, not just putting in time. I think you can find something to challenge you in most projects/situations, whether that's solving scaling, latency, algorithm selection or just plain figuring out how to meet the business needs. But there's a corollary to that, it has to be something meaningful. Sure, "reduced latency from 1,023ms to 45ms" looks great on a resume, but if it's an operation that gets called once a day from a cron job, did you achieve anything? If you spent the last 6 months "configuring" a system owned by your parent company or fighting with byzantine build tools, but don't have any means to improve that process, did you grow?

I can't remember the last project that involved an actual technical challenge. A true craftsman doesn't blame their tools, but trying to build a skyscraper with a plastic hammer isn't a challenge, it's foolish. Most of the "challenges" here are self-inflicted. - 0/5

3 - Growth

I'm huge on growth and learning. This is third on my list but there are aspects of it in pretty much everything I choose to do (or not do). I attend a lot of conferences, I find it's a great way to keep up with the industry, see what's hot and upcoming and get new ideas or inspiration. I read a ton of books and articles. I also take advantage of MOOCs (Coursera/Udacity/etc) - I'm taking some courses on R right now. Elasticsearch? Docker? Kafka? TDD? Continuous delivery? Sign me up!! I'm happy learning new languages and being a polyglot (java/ruby/go/perl/whatever) programmer. It helps me to select the best language for the job at hand and team situation. That being said, there's a vast difference between HelloWorld and the invaluable experience of running something in production. If you don't have anywhere to apply all these ideas, concepts and languages to something real, you won't grow nearly as much.

Zappos funds one conference a year and lets you expense work-related books. 'Pursue growth and learning' is actually a core value. There's a quarterly hackathon to explore and try to solve any problem you want. Ultimately though, it boils down to "Do I have somewhere to apply what I'm learning?" and the answer has been almost uniformly no. All the time and effort I invest in outside learning barely offsets the complete lack of mental stimulation offered from the projects available. - 2/5

4 - Balance

Work hard, play hard. I love thinking about tough problems, and like the fable of Archimedes, having that Eureka moment. But those moments don't necessarily occur at work. They occur at the gym or lounging by the pool. They happen at a bar with friends or coworkers or on a trip to an exotic beach with crystal clear water. What's important is that I have the challenging problem to puzzle over in the first place. I don't enjoy purely remote work, I like being in contact with other human beings and coworkers. Some of the best times in my life have been during grueling projects with a small group of awesome people, doing something that mattered.

You definitely have the freedom to play hard here, and the freedom to joust at windmills if you choose. But work hard? Not doing something that matters. 2/5

5 - Money

Last on the list is salary, for a reason. This is more of an enabler than an end goal. As Henry David Thoreau said, Wealth is the ability to fully experience life. Once I have enough to live the way I want, and to do the things I want, then it stops being a motivating factor.

I feel more than fairly compensated for this market, which is all I ask. - 5/5

I heard the Arctic Monkeys playing 'U R Mine' on the way to work this morning:

And I go crazy cause here isn't where I wanna be
And satisfaction feels like a distant memory

What's important to you? (career-wise)

Should I stay or should I go now?
If I go, there will be trouble
And if I stay it will be double
- The Clash

I normally stay away from blogging about career advice, or advice in general, but sometimes you get thrown a curveball - something like "The Offer" and have to come up with a way of dealing with it.

It's a very generous offer. It also shows strong commitment by leadership to an idea or ideal; especially when the ramifications are large enough to potentially reinvigorate, or decimate, a fortune 500 company. Unfortunately, is it too good of an offer? I'm not going to go into the tech landscape at Zappos, but there are many confounding factors.

What I want to talk about though is, "As a software engineer, how could I process a major decision like "stay or go"?

Do we:
Break out the some spreadsheets and calculate financial outcomes using future value of money formulas?
Mash it up with the expected probabilities of various events?
Go with a gut feeling?
Just blindly invest in someone else's evolving vision?
Ask your friends what they're doing or for advice?

Is it even necessary to evaluate the attempted transition to self-management/self-organization? Or, is there an easier heuristic for this decision in the first place?

Lets look at Maslow's work on hierarchy of needs, usually pictured as a pyramid. Roughly paraphrased, Maslow posited that there are two groupings: deficiency needs and growth needs. Deficiency needs are things that must be satisfied before you can move to the next higher level. Growth needs can only be acted upon after deficiency needs are satisfied.

From the bottom of the pyramid to the top, the first four are deficiency needs: Physiological -> Safety/security -> Belongingness/Love -> Esteem.
After that are the growth needs: Cognitive -> Aesthetic -> Self-actualization -> Self-transcendence

(Self-actualization is something like "finding self-fulfillment and realizing one's potential", while self-transcendence is something like "helping others find self-fulfillment and realize their potential")

Is there something similar to this hierarchy of needs for a career? Where would self-management and self-organization fit in?

I'm going to suggest that these ideas that we're being asked to buy into fall somewhere between self-actualization and self-transcendence, as I don't think anyone can argue they're a deficiency need.
Since it's not a deficiency need, and deficiency needs must be satisfied first, what are your deficiency needs? Or put more plainly, "what's important to you?"

Figure out what is important to you in your career/profession and what you need from your place of employment

That might be earning enough money to live comfortably, job security, a good work/life balance allowing you to spend time with your family or traveling to exotic locales, a short commute, growth opportunities, etc. It doesn't matter WHAT those are, just figure out what goals or things are important to YOU in your career. Once you do that, it may help you evaluate large decisions like this.

Integration tests in golang

In an ideal world, I want to run `go test` to run unit tests (which should be fast, not rely on external resources/network/etc) and add something to the command to specify that I want to run integration tests. You can definitely do it with ENV variables and/or some flags inside the test package, but what I've settled on for now is to use build tags and separate out my integration tests.

Create a file 'integration_test.go' and add the following build tag to the file before the package (note: you must put a blank line after the build tag to distinguish it from a comment):

// +build integration

Then just write your tests as usual.

When it comes time to run them, you still use `go test` for your unit tests but now you can type `go test -tags=integration` to only run your integration tests.

3node-kafka

Installing 3-node Zookeeper + 3-node Kafka cluster on Ubuntu instances in AWS

Build a Zookeeper cluster

Log into your AWS account
Create a m3.medium ubuntu 64-bit instances with a 256GB EBS instance
Create a security group allowing you some TCP ports - ssh (22) and a zookeeper port (2181)
Wait for the instance to boot up and log into it
Set up your disk space - since it’s an EBS instance, it’s not attached yet. See: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html

$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdb 202:16 0 256G 0 disk

As you can see, we have to attach /dev/xvdb. Since it’s new, we’ll also have to put an ext4 filesystem on it, make a data directory and then mount the EBS volume

$ sudo mkfs -t ext4 /dev/xvdb
$ sudo mkdir /data
$ sudo mount /dev/xvdb /data

If all went well, running df -h should show ~250 GB available on /data
Edit /etc/fstab to add the following line for /dev/xvdb:

$ sudo vi /etc/fstab
/dev/xvdb /data ext4 defaults,nofail,nobootwait 0 2

Double check (if you don’t see any output from this command, your fstab is ok)

$ sudo mount -a

Fix your ~/.bash_profile - add this to the first line

if [ -f ~/.bashrc ]; then . ~/.bashrc; fi

Now install Zookeeper. We’ll try the easy way first and see what we get:

$ sudo apt-get install zookeeper zookeeper-bin

The Ubuntu zookeeper package puts a configuration file at /etc/zookeeper/conf/zoo.cfg, we need to edit it to set the data dir before starting up zookeeper.

$ sudo vi /etc/zookeeper/conf/zoo.cfg

Change the data dir line to dataDir=/data/zookeeper and save it (ignore the replicated part, we’ll come back to that once we’re sure this instance works)

$ cd /usr/share/zookeeper && sudo bin/zkServer.sh start

In preparation for setting up the cluster, go ahead and create the ‘myid’ file that Zookeeper uses in the data directory you specified in the conf. NOTE: this must be a single integer and increment on each host (i.e. 1 is the first, 2 is the second, 3 is the third host in the cluster)

$ sudo sh -c ‘echo “1” > /etc/zookeeper/conf/myid’

Now, repeat for two more servers (so that we have 3 running Zookeeper).

Important: you will probably want/need to edit your AWS security group to allow the instances to communicate amongst themselves. I just opened up TCP ports 0 - 3888 to the security group, but you might want to lock that down further. (if you have a security group sg-12345, you can put that in the IP address field instead of specifying an actual IP address or range).

Once you have all three instances running, stop the Zookeeper on each. It’s time to make the cluster.

Edit your zookeeper config again, ON EACH INSTANCE, this time uncommenting the section:

server.1=your-ec2-host-1.compute.amazonaws.com:2888:3888
server.2=your-ec2-host-2.compute.amazonaws.com:2888:3888
server.3=your-ec2-host-3.compute.amazonaws.com:2888:3888

Start up each of the instances again.

Once they’re all up, test them out. Fire up the CLI bundled with Zookeeper.

bin/zkCli.sh -server 127.0.0.1:2181

Run these commands on the first instance (and you should see similar output)

[zk: 127.0.0.1:2181(CONNECTED) 0] ls /
[zookeeper]
[zk: 127.0.0.1:2181(CONNECTED) 1] create /foo mydata
Created /foo
[zk: 127.0.0.1:2181(CONNECTED) 2] ls /
[foo, zookeeper]

Now start up the CLI on the other instances and run ls / - you should see foo in the results. Replication works, woohooo!

Build the Kafka cluster

Similar to the Zookeeper instances, but a little beefier, I created three m3.large instances with 256GB of SSD EBS. Same process for mounting the EBS, etc.

Install java

sudo apt-get install -y openjdk-7-jdk

Download Kafka from a mirror and install it

$ curl -O http://supergsego.com/apache/kafka/0.8.1.1/kafka_2.8.0-0.8.1.1.tgz
$ sudo mkdir -p /usr/local/kafka && sudo mv kafka_2.8.0-0.8.1.1.tgz /usr/local/kafka/ && cd /usr/local/kafka/
$ sudo tar -xvf kafka_2.8.0-0.8.1.1.tgz
$ cd kafka_2.8.0-0.8.1.1

Edit the server.properties file in config/server.properties
* change broker.id to an integer (just like with myid in zookeeper) - e.g. broker.id=1 for the first instance, broker.id=2 for the second, etc
* change the zookeeper.connect property to a comma-separated list of the zookeeper instances and ports (e.g. zookeeper.connect=your-zks-host-1.compute.amazonaws.com:2181,your-zks-host-2.compute.amazonaws.com:2181,your-zks-host-3.compute.amazonaws.com:2181)
* change the advertised.host.name to the PUBLIC dns entry for your AWS host - this is super important or (depending on your language/library) you’ll get cryptic failure to resolve DNS/connect errors

NOTE: You’ll probably want to store that zookeeper string somewhere for convenience

Make your life easier, add some properties to your ~/.bash_profile

export KAFKA_HOME=/usr/local/kafka/kafka_2.8.0-0.8.1.1
export PATH=$ZK_HOME/bin:$KAFKA_HOME/bin:$PATH
export ZKS=your-zks-host-1.compute.amazonaws.com:2181,your-zks-host-2.compute.amazonaws.com:2181,your-zks-host-3.compute.amazonaws.com:2181
export KAFKA_NODES=your-kafka-host-1.compute.amazonaws.com:9092,your-kafka-host-2.compute.amazonaws.com:9092,your-kafka-host-3.compute.amazonaws.com:9092

Make sure to source the file so that your changes take effect in this session

$ source ~/.bash_profile

Start up all the instances individually (watch for errors!)

$ sudo $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &

Once they’re all up, test it out

$ $KAFKA_HOME/bin/kafka-topics.sh –list –zookeeper $ZKS
$ $KAFKA_HOME/bin/kafka-topics.sh –create –zookeeper $ZKS –replication-factor 2 –partitions 2 –topic test-topic-1
# check it again
$ $KAFKA_HOME/bin/kafka-topics.sh –list –zookeeper $ZKS

Kafka provides some handy scripts in the bin/ directory, make use of them =)

# run this on host 2 (just to keep things interesting)
$KAFKA_HOME/bin/kafka-console-producer.sh –topic test-topic-1 –broker-list $KAFKA_NODES

# run this on hosts 1 and 3
$ $KAFKA_HOME/bin/kafka-console-consumer.sh –topic=test-topic-1 –zookeeper=$ZKS

Anything you type in the console on host 2 should be echoed on hosts 1 and 3

success, you have a working 3-node kafka cluster.