Friday, November 13, 2015

updatedb on OSX

Mainly because I always forget, on OSX you can run the script the updates the locate db automatically, just run:
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.locate.plist
(you only have to do it once)

Before you do, edit /System/Library/LaunchDaemons/com.apple.locate.plist Get rid of the following portion so that it runs daily instead of just once a week though.

                <key>Weekday</key>
                <integer>6</integer>

Monday, November 2, 2015

A refresher on what Java 7 brought...

Java 7 was released way back in 2011, but it's unfortunately not THAT uncommon for a large organization to get set in its ways (having found years ago, those battle-tested, magical, set of exact JVM tuning arguments that work for *their particular* application stack on a certain version of a 1.6 JVM) so that you don't get exposed to "newer" features. And to only get dragged kicking and screaming into JDK8 when Oracle deprecated certain versions.

So, what did Java 7 bring us? As developers, it seemed to be mostly about cleaning up syntax/getting rid of some of the boilerplate...

  • Strings in switch statements - sure, other languages had it for years, but you weren't able to switch on Strings until Java 7. Nice to have, not especially mind-blowing.
  • try with resources - This was a welcome addition - no longer did you have the super-awkward "declare your variable outside the block just so you have access to it in a second try/catch so you can close it..." e.g.
    Connection conn = null;
    try{
      conn = getConnectionFromSomewhere();
      // do work with the resource
    } catch (Exception e){
      // handle
    } finally {
      if (conn != null){
        try {
          conn.close();
        } catch (Exception someOtherExceptionYouProbablyIgnore) {
          // yah, you probably ignore this
        }
      }
    }
    
    Nope, now you can just do
    try(Connection conn = getConnectionFromSomewhere();){
      // do work with the resource
    } catch (Exception e) {
      // handle
    }
    You can also have your own objects work this way, look into java.lang.AutoClosable
  • Multi-catch - Another nice little cleanup, rather than listing out all the exceptions you want to handle (even if you handle some of them the same), you can now use the | pipe operator to union the exceptions you are handling. Instead of this:
    } catch (IOException ex) {
         logger.log(ex);
         throw ex;
    } catch (SQLException ex) {
         logger.log(ex);
         throw ex;
    }
    write this:
    } catch (IOException|SQLException ex) {
        logger.log(ex);
        throw ex;
    }
    
  • Binary literals - "0b" prefix, e.g. 0b10100111001 is 1337
  • left-to-right Improved type inference (<>) Removed the necessity of writing tripe like:
    Map<String, List<Integer>> m = new HashMap<String, List<Integer>>();
    simplifying it slightly to:
    Map<String, List<Integer>> m = new HashMap<>();
    (This seems slightly backwards to me, we're essentially specifying the details on the interface and waving our hands over the concrete implementation, instead of
    Map<> m = new HashMap<String, List<Integer>>(); // does not work
    but I'm not the designer - you must use the first version =)
  • Underscores in numeric literals What's easier to understand at a glance? Counting the zeros in 1000000000 or 1_000_000_000?
  • A whole slew of new APIs for NIO - notably, java.nio.Path and the WatchService, which allows you to be notified of changes to a path you're watching (there are a ton of applications for this)

Those are the big ones in my opinion. Did I miss any of your favorites?

Saturday, October 31, 2015

Changing default prefix keys for tmux

tmux uses Ctrl-b as the default command prefix, that's ok on Windows but less than desirable on a MacBook Pro. Turns out that it's super easy to remap to something easier to type, but it's a two-step process.

First, let's get some usage out of the vestigial 'caps lock' key. Go to 'System Preferences' -> 'Keyboard' -> 'Modifier Keys' and then change caps lock to do something (marginally more) useful, the Control key.

Now that that's out of the way, remap the prefix in tmux by editing (or creating) your ~/.tmux.conf file. We only need one line in there to remap the prefix to Ctrl-a (which are handily right next to each other):

set -g prefix C-a

Two things to note: first, changing the caps lock key to be control affects ALL applications on your Mac, if you're like me and never use caps lock, that's probably ok - just something to remember. Secondly, sometimes you might want to be able to send Ctrl-a to an application that you're using inside tmux. If you think you will, just add another line to your ~/.tmux.conf file:

bind C-a send-prefix

This will allow you to send Ctrl-a to the application running inside tmux by pressing it TWICE.

Friday, September 4, 2015

Compiling a Go app, that uses Cgo, to run on an Edison

[TLDR - Use a Docker container, install the 32-bit compiler chain and 32-bit dev libraries on the build machine, include the flags to enable CGO, set the target GOARCH, set ldflags to statically link]

So, originally I thought I'd be able to easily get my (trivial) Go app to cross-compile for the Edison (which is running a 32-bit linux variant) but I was quickly disabused of that notion. My app uses the `gopacket` library, which in turn uses C bindings (Cgo) to `libpcap-dev` to do the actual packet capture.

I had originally thought it was just a matter of adding "GOOS=linux GOARCH=386" to compile from my OSX box to target a 32-bit linux binary. It works fine for most apps, BUT it doesn't work for apps that are using Cgo. Ah, just forgot the "CGO_ENABLED=1" flag, right? Nope. That causes all sorts of different errors. Googling/Stack Overflow didn't really turn up anything helpful, but there's *probably* a way to do it. (there were a few projects out there, including gonative that seemed promising, but `gonative` only addresses Cgo-enabled versions of the stdlib packages, not if your project uses Cgo)

Rather than dig into the intricacies of cross-compiling on a Mac, I just pivoted to using a Docker container. I couldn't find an official Yocto linux container prebuilt, which is what the Edison runs, so I just went with a standard Ubuntu image.

root@ubuntu-docker-instance# go build listen.go 

Sweet, compiled. All set, that was easy. Let's just check the binary to make sure all is good with the world.

root@ubuntu-docker-instance# file listen
listen: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=11b30b279fd6580392e72bac70a5be034e12b2a7, not stripped

Oops, dynamically linked. Let's fix that by setting the correct ld flags, and check it again.

root@ubuntu-docker-instance# go build --ldflags '-extldflags "-static"' listen.go 
root@ubuntu-docker-instance# file listen
listen: ELF 64-bit LSB  executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.24, BuildID[sha1]=8a5a63b0151c2cef02399d091ea7339b2ca1d30b, not stripped

Ok, statically linked now but it's (still) 64-bit. The Edison is 32-bit, which means we'll have to fix that... should be cake, just use the "GOARCH" flag...

root@ubuntu-docker-instance/util# GOARCH=386 go build --ldflags '-extldflags "-static"' identify.go 
# command-line-arguments
./identify.go:19: undefined: pcap.FindAllDevs
./identify.go:23: undefined: pcap.OpenLive
./identify.go:23: undefined: pcap.BlockForever

What?! Oh, yes, need to tell the compiler about the Cgo code... (CGO_ENABLED=1)

root@ubuntu-docker-instance/util# GOARCH=386 CGO_ENABLED=1 go build --ldflags '-extldflags "-static"' identify.go 
# runtime/cgo
In file included from /usr/include/errno.h:28:0,
                 from /usr/local/go/src/runtime/cgo/cgo.go:50:
/usr/include/features.h:374:25: fatal error: sys/cdefs.h: No such file or directory
 #  include 
                         ^
compilation terminated.

C'mon... now what? Some quick googling turns up the fact that you need the 32-bit gcc stuff, since the host architecture is 64-bit.

root@ubuntu-docker-instance/util# apt-get install libx32gcc-4.8-dev libc6-dev-i386

root@ubuntu-docker-instance/util# GOARCH=386 CGO_ENABLED=1 go build --ldflags '-extldflags "-static"' identify.go 
# github.com/google/gopacket/pcap
/usr/bin/ld: cannot find -lpcap
collect2: error: ld returned 1 exit status

Now what? Oh, although I have the 32-bit compiler chain, I DON'T have the 32-bit version of libpcap-dev. Let's fix that.

root@ubuntu-docker-instance/util# apt-get install libpcap-dev:i386
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package libpcap-dev:i386 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'libpcap-dev:i386' has no installation candidate

Huh? furious googling ensues... Ok, add the i386 architecture and try again...

root@ubuntu-docker-instance/util# sudo dpkg --add-architecture i386
root@ubuntu-docker-instance/util# sudo apt-get update
root@ubuntu-docker-instance/util# apt-get install libpcap0.8-dev:i386

NOW we're cooking with fire. Let's give it one more try...

root@ubuntu-docker-instance/util# GOARCH=386 CGO_ENABLED=1 go build --ldflags '-extldflags "-static"' identify.go 

Looks promising, it's a 32-bit statically compiled binary at this point. No obvious errors.

identify: ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.24, not stripped

SCP the binary onto the Edison and test it...

root@edison:~# ./identify -inf=wlan0
2015/09/04 14:11:02 Starting up...
2015/09/04 14:11:03 Listening for ARP packets on en0 to identify dash button.
Press the dash button and watch the screen. You will want the 'Source MAC'
2015/09/04 14:11:09 ARP packet, Source MAC[74:75:48:a4:59:a8], Destination MAC[ff:ff:ff:ff:ff:ff]
2015/09/04 14:11:24 ARP packet, Source MAC[74:75:48:29:a8:7c], Destination MAC[ff:ff:ff:ff:ff:ff]

SUCCESS!!!

Tuesday, September 1, 2015

Playing with Amazon Dash buttons

Amazon is experimenting with wireless buttons (for $5 each) called Dash Buttons. The intended usage seem to be for Prime members to stick the button near a product that needs periodic refilling, e.g. laundry soap, so that when you're almost out of the product, you press the button and it (almost) magically reorders for you.

These buttons are nifty little pieces of technology, especially for $5. Someone has already figured out how to use them for something other their intended usage - read his post on Medium, it's very well written. He has source code in Python for accomplishing the hack.

I wanted to take a slightly different route to accomplishing the same type of thing. I have a bunch of hardware laying around that would be fun to play around with, including an Intel Edison board. I could write my code in Go, cross-compile it on my Mac, and just SCP it onto the Edison. Sounded like a fun evening or two, so that's what I did and threw it up on github.

Tested the code on OSX and it works. Attempted to compile it for Linux, so I can deploy it onto the Edison, and compilation fails...

$ GOOS=linux GOARCH=386 go build identify.go
# command-line-arguments
./identify.go:19: undefined: pcap.FindAllDevs
./identify.go:23: undefined: pcap.OpenLive
./identify.go:23: undefined: pcap.BlockForever

I'm pretty sure it uses CGO and there are some details about cross-compiling that I'll need to figure out... but that will be for another post.

Wednesday, July 22, 2015

Getting RStudio to work with RWeka and the previous java 1.8 problem

So, from the previous post, installing Java 1.6 and recompiling rJava is enough to get everything working from a terminal window. But, if you start up RStudio and try to use RWeka (or other libraries using JNI, probably) you'll see an error message similar to:

Error : .onLoad failed in loadNamespace() for 'rJava', details:
  call: dyn.load(file, DLLpath = DLLpath, ...)
  error: unable to load shared object '/usr/local/Cellar/r/3.2.0_1/R.framework/Versions/3.2/Resources/library/rJava/libs/rJava.so':
  dlopen(/usr/local/Cellar/r/3.2.0_1/R.framework/Versions/3.2/Resources/library/rJava/libs/rJava.so, 6): Library not loaded: @rpath/libjvm.dylib
  Referenced from: /usr/local/Cellar/r/3.2.0_1/R.framework/Versions/3.2/Resources/library/rJava/libs/rJava.so
  Reason: image not found
Error: package or namespace load failed for ‘RWeka’

A workaround is to start RStudio from a terminal window by running the following line:

LD_LIBRARY_PATH=$(/usr/libexec/java_home)/jre/lib/server: open -a RStudio

Monday, July 20, 2015

Getting various R packages to work with Java 1.8 and rJava on OSX

If you attempt to install various R packages (e.g. RWeka) on OSX using a Java 1.8 runtime, you're likely to run into the following error message (even though you have a perfectly good installation of Java):
"No Java runtime present, requesting install.
ERROR: loading failed"

There are a ton of threads out there about this problem, but it seems to boil down to this unclosed (and untouched since 2014-01) JDK issue -> https://bugs.openjdk.java.net/browse/JDK-7131356

The "solution" can be found on the last comment on this rJava github issue: https://github.com/s-u/rJava/issues/37

  1. install JDK 1.6 from Apple (http://support.apple.com/kb/DL1572)
  2. open a new terminal window and make sure you're still using 1.8 - "java -version"
  3. run "sudo R CMD javareconf -n" from a terminal window
  4. start up R and install rJava again from source - install.packages("rJava", type="source")

Tuesday, May 12, 2015

Continuous integration: Rmd to md to html

I'm taking one of the Coursera courses right now on Reproducible Research and the first assignment requires you to build an Rmd file and convert it to html (using knitr). They do a really good job showing how to use RStudio to accomplish this task, but what if you don't want to bother with RStudio?

Here's (https://gist.github.com/slowteetoe/7b8426f567ac3df1d8f9) a simple bash script that watches an Rmd file. When you save that file, it runs knitr to convert it to markdown, and then converts the markdown to html and opens it in a browser... sorta a continuous integration for R markdown.

Couple caveats: you'll need to install the Ruby gem 'kicker', the r packages 'knitr' and 'markdown', and put this script somewhere in your path. It will also have to be made executable, 'chmod 755 knitr'. Also, it opens a different browser tab each time - but I couldn't figure out how to change this behavior and it was worth it (to me at least).

Thursday, April 16, 2015

R annoyances

R seems to be, hmm, quirky, from what I've seen of it so far... For instance, ddply has different behavior using the American vs. British spellings of "summarize". Seriously?!
> g <- ddply(m, c("QGroup","Income.Group"), summarize, All = length(CountryCode))
Error: argument "by" is missing, with no default

> g <- ddply(m, c("QGroup","Income.Group"), summarise, All = length(CountryCode))
Yes, that's right - "summarise" works fine but "summarize" apparently takes different arguments.

Trouble installing jpeg R package on OSX?

I'm taking the Coursera "Getting and Cleaning Data" class in R and one of the quizzes requires you to use the jpeg package. The R package install was failing for some reason though:
install.packages("jpeg")
In file included from read.c:1:
./rjcommon.h:11:10: fatal error: 'jpeglib.h' file not found
#include 
         ^
I use homebrew, so I tried the usual...
brew install jpeg
Warning: jpeg-8d already installed

brew unlink jpeg && brew link jpeg
But that didn't fix the issue. The solution was to install the XCode command line tools with:
xcode-select --install
(As a side note, this is my development machine so I already had XCode command line tools but for whatever reason I had to install them again, maybe an XCode upgrade?)

Tuesday, April 7, 2015

Sometimes the best code is that which is never written

Someone wrote some moderately over-complicated code to provide functionality in a service. That developer has moved on. A little time passes and suddenly there are a bunch of stack traces popping up in the logs indicating that a status message isn't getting published.
First reaction - dig into the error, figure out exactly what's wrong and fix it, ASAP!

Here's the error:  Invalid parameter: Message too long (Service: AmazonSNS; Status Code: 400; Error Code: InvalidParameter; Request ID: <redacted>)

Seems pretty straightforward; two seconds of Googling turns up that SNS messages have a limit of 256KB. Go into the appropriate class and add something detect how many bytes are in the message. Hmmm... maybe these messages are really large, and our app has a ton of threads - don't want to use up too much memory. A quick check on StackOverflow suggests using a CountingInputStream (there's one in the Apache commons-io, but it's also pretty trivial to write). Add it. Write a couple tests to make sure that it counts bytes correctly for UTF-8, UTF-16, etc... Looking good and have only spent 20mins so far fixing this problem.

Now that we can reliably detect the message size, we need to deal with ones that are too large for SNS... Uh-oh, the message body is actually something getting serialized into a JSON payload, we can't simply truncate or the message won't be valid JSON any more. This will require some thought, but I'm totally up for the challenge!

STOP! Take a step back from the technical details and think about what problem you're trying to solve.


Basically, the point of this code was to "make sure a notification goes out that something failed to process". There's nothing in there to suggest that the entire response needs to be serialized and sent on the wire (although that's what it was doing originally). Think to yourself, "what's the simplest thing that could work?". In this case, all that really needs to be sent is an ID and a status of success or failure:

{ id: 1234, status: "failure" }

There's NO way that message will ever approach 256KB, so do I need to worry about truncating this message? No. Looking back on it, did I need a CountingOutputStream even with the truncating solution? Probably not, it was a JSON payload in UTF-8; UTF-8 handles the ASCII range in 1 byte per character so simply getting the length of the String should have been "close enough".

In the end, I had to write a negligible amount of code to create that new JSON payload, and I removed a bunch of complexity from the original codebase along the way.

The take-away from this exercise? Step back and understand the problem-space from the start and resist the impulse to dive in and fix the errors immediately.

What's important to me, career-wise

(Continuing on from "What's important to you, career-wise?", here's my personal list of top 5 needs from a job, ranked from most important to least)

1 - Interesting people

It's important to me to work with interesting people who are more experienced/skilled/intelligent than I am. As Chad Fowler said in The Passionate Programmer: Always be the worst guy in every band you’re in. - so you can learn. The people around you affect your performance. Choose your crowd wisely. It is NOT a good idea to surround yourself with people who think identically or have had the same life experiences. Just because everyone around you hates node.js doesn't mean there aren't good ideas in it, but if there's no one championing it then it's easy to get mired in your (mis-)conceptions. Different viewpoints can challenge your perceptions and make you grow. Learn from their experiences and challenge dogma wherever you go.
In the almost 6 yrs I've been here, there's only been one person I've disliked working with; that's almost unheard of. It lends credence to the idea that hiring decisions based on culture can result in a better working environment. Unfortunately, niceness doesn't help you grow. There are people here I would love to work on projects with, but the reality is that there are no projects big or complex enough to justify us working together. Those people are a scarce resource that gets spread out to make the most use of their time - 1/5

2 - Challenge

You grow from experience and from solving problems, not just putting in time. I think you can find something to challenge you in most projects/situations, whether that's solving scaling, latency, algorithm selection or just plain figuring out how to meet the business needs. But there's a corollary to that, it has to be something meaningful. Sure, "reduced latency from 1,023ms to 45ms" looks great on a resume, but if it's an operation that gets called once a day from a cron job, did you achieve anything? If you spent the last 6 months "configuring" a system owned by your parent company or fighting with byzantine build tools, but don't have any means to improve that process, did you grow?
I can't remember the last project that involved an actual technical challenge. A true craftsman doesn't blame their tools, but trying to build a skyscraper with a plastic hammer isn't a challenge, it's foolish. Most of the "challenges" here are self-inflicted. - 0/5

3 - Growth

I'm huge on growth and learning. This is third on my list but there are aspects of it in pretty much everything I choose to do (or not do). I attend a lot of conferences, I find it's a great way to keep up with the industry, see what's hot and upcoming and get new ideas or inspiration. I read a ton of books and articles. I also take advantage of MOOCs (Coursera/Udacity/etc) - I'm taking some courses on R right now. Elasticsearch? Docker? Kafka? TDD? Continuous delivery? Sign me up!! I'm happy learning new languages and being a polyglot (java/ruby/go/perl/whatever) programmer. It helps me to select the best language for the job at hand and team situation. That being said, there's a vast difference between HelloWorld and the invaluable experience of running something in production. If you don't have anywhere to apply all these ideas, concepts and languages to something real, you won't grow nearly as much.
Zappos funds one conference a year and lets you expense work-related books. 'Pursue growth and learning' is actually a core value. There's a quarterly hackathon to explore and try to solve any problem you want. Ultimately though, it boils down to "Do I have somewhere to apply what I'm learning?" and the answer has been almost uniformly no. All the time and effort I invest in outside learning barely offsets the complete lack of mental stimulation offered from the projects available. - 2/5

4 - Balance

Work hard, play hard. I love thinking about tough problems, and like the fable of Archimedes, having that Eureka moment. But those moments don't necessarily occur at work. They occur at the gym or lounging by the pool. They happen at a bar with friends or coworkers or on a trip to an exotic beach with crystal clear water. What's important is that I have the challenging problem to puzzle over in the first place. I don't enjoy purely remote work, I like being in contact with other human beings and coworkers. Some of the best times in my life have been during grueling projects with a small group of awesome people, doing something that mattered.
You definitely have the freedom to play hard here, and the freedom to joust at windmills if you choose. But work hard? Not doing something that matters. 2/5

5 - Money

Last on the list is salary, for a reason. This is more of an enabler than an end goal. As Henry David Thoreau said, Wealth is the ability to fully experience life. Once I have enough to live the way I want, and to do the things I want, then it stops being a motivating factor.
I feel more than fairly compensated for this market, which is all I ask. - 5/5

I heard the Arctic Monkeys playing 'U R Mine' on the way to work this morning:

And I go crazy cause here isn't where I wanna be
And satisfaction feels like a distant memory

Monday, March 30, 2015

What's important to you? (career-wise)

Should I stay or should I go now?
If I go, there will be trouble
And if I stay it will be double
- The Clash
I normally stay away from blogging about career advice, or advice in general, but sometimes you get thrown a curveball - something like "The Offer" and have to come up with a way of dealing with it.

It's a very generous offer. It also shows strong commitment by leadership to an idea or ideal; especially when the ramifications are large enough to potentially reinvigorate, or decimate, a fortune 500 company. Unfortunately, is it too good of an offer? I'm not going to go into the tech landscape at Zappos, but there are many confounding factors.

What I want to talk about though is, "As a software engineer, how could I process a major decision like "stay or go"?

Do we:
Break out the some spreadsheets and calculate financial outcomes using future value of money formulas?
Mash it up with the expected probabilities of various events?
Go with a gut feeling?
Just blindly invest in someone else's evolving vision?
Ask your friends what they're doing or for advice?

Is it even necessary to evaluate the attempted transition to self-management/self-organization? Or, is there an easier heuristic for this decision in the first place?

Lets look at Maslow's work on hierarchy of needs, usually pictured as a pyramid. Roughly paraphrased, Maslow posited that there are two groupings: deficiency needs and growth needs. Deficiency needs are things that must be satisfied before you can move to the next higher level. Growth needs can only be acted upon after deficiency needs are satisfied.

From the bottom of the pyramid to the top, the first four are deficiency needs: Physiological -> Safety/security -> Belongingness/Love -> Esteem.
After that are the growth needs: Cognitive -> Aesthetic -> Self-actualization -> Self-transcendence

(Self-actualization is something like "finding self-fulfillment and realizing one's potential", while self-transcendence is something like "helping others find self-fulfillment and realize their potential")

Is there something similar to this hierarchy of needs for a career? Where would self-management and self-organization fit in?

I'm going to suggest that these ideas that we're being asked to buy into fall somewhere between self-actualization and self-transcendence, as I don't think anyone can argue they're a deficiency need.
Since it's not a deficiency need, and deficiency needs must be satisfied first, what are your deficiency needs? Or put more plainly, "what's important to you?"

Figure out what is important to you in your career/profession and what you need from your place of employment

That might be earning enough money to live comfortably, job security, a good work/life balance allowing you to spend time with your family or traveling to exotic locales, a short commute, growth opportunities, etc. It doesn't matter WHAT those are, just figure out what goals or things are important to YOU in your career. Once you do that, it may help you evaluate large decisions like this.