Playing with Amazon Dash buttons

Amazon is experimenting with wireless buttons (for $5 each) called Dash Buttons. The intended usage seem to be for Prime members to stick the button near a product that needs periodic refilling, e.g. laundry soap, so that when you're almost out of the product, you press the button and it (almost) magically reorders for you.

These buttons are nifty little pieces of technology, especially for $5. Someone has already figured out how to use them for something other their intended usage - read his post on Medium, it's very well written. He has source code in Python for accomplishing the hack.

I wanted to take a slightly different route to accomplishing the same type of thing. I have a bunch of hardware laying around that would be fun to play around with, including an Intel Edison board. I could write my code in Go, cross-compile it on my Mac, and just SCP it onto the Edison. Sounded like a fun evening or two, so that's what I did and threw it up on github.

Tested the code on OSX and it works. Attempted to compile it for Linux, so I can deploy it onto the Edison, and compilation fails...

$ GOOS=linux GOARCH=386 go build identify.go
# command-line-arguments
./identify.go:19: undefined: pcap.FindAllDevs
./identify.go:23: undefined: pcap.OpenLive
./identify.go:23: undefined: pcap.BlockForever

I'm pretty sure it uses CGO and there are some details about cross-compiling that I'll need to figure out... but that will be for another post.

Getting RStudio to work with RWeka and the previous java 1.8 problem

So, from the previous post, installing Java 1.6 and recompiling rJava is enough to get everything working from a terminal window. But, if you start up RStudio and try to use RWeka (or other libraries using JNI, probably) you'll see an error message similar to:

Error : .onLoad failed in loadNamespace() for 'rJava', details:
  call: dyn.load(file, DLLpath = DLLpath, ...)
  error: unable to load shared object '/usr/local/Cellar/r/3.2.0_1/R.framework/Versions/3.2/Resources/library/rJava/libs/rJava.so':
  dlopen(/usr/local/Cellar/r/3.2.0_1/R.framework/Versions/3.2/Resources/library/rJava/libs/rJava.so, 6): Library not loaded: @rpath/libjvm.dylib
  Referenced from: /usr/local/Cellar/r/3.2.0_1/R.framework/Versions/3.2/Resources/library/rJava/libs/rJava.so
  Reason: image not found
Error: package or namespace load failed for ‘RWeka’

A workaround is to start RStudio from a terminal window by running the following line:

LD_LIBRARY_PATH=$(/usr/libexec/java_home)/jre/lib/server: open -a RStudio

Getting various R packages to work with Java 1.8 and rJava on OSX

If you attempt to install various R packages (e.g. RWeka) on OSX using a Java 1.8 runtime, you're likely to run into the following error message (even though you have a perfectly good installation of Java):

"No Java runtime present, requesting install.
ERROR: loading failed"

There are a ton of threads out there about this problem, but it seems to boil down to this unclosed (and untouched since 2014-01) JDK issue -> https://bugs.openjdk.java.net/browse/JDK-7131356

The "solution" can be found on the last comment on this rJava github issue: https://github.com/s-u/rJava/issues/37

1. install JDK 1.6 from Apple (http://support.apple.com/kb/DL1572)
2. open a new terminal window and make sure you're still using 1.8 - "java -version"
3. run "sudo R CMD javareconf -n" from a terminal window
4. start up R and install rJava again from source - install.packages("rJava", type="source")

Continuous integration: Rmd to md to html

I'm taking one of the Coursera courses right now on Reproducible Research and the first assignment requires you to build an Rmd file and convert it to html (using knitr). They do a really good job showing how to use RStudio to accomplish this task, but what if you don't want to bother with RStudio?

Here's (https://gist.github.com/slowteetoe/7b8426f567ac3df1d8f9) a simple bash script that watches an Rmd file. When you save that file, it runs knitr to convert it to markdown, and then converts the markdown to html and opens it in a browser... sorta a continuous integration for R markdown.

Couple caveats: you'll need to install the Ruby gem 'kicker', the r packages 'knitr' and 'markdown', and put this script somewhere in your path. It will also have to be made executable, 'chmod 755 knitr'. Also, it opens a different browser tab each time - but I couldn't figure out how to change this behavior and it was worth it (to me at least).

R annoyances

R seems to be, hmm, quirky, from what I've seen of it so far... For instance, ddply has different behavior using the American vs. British spellings of "summarize". Seriously?!

> g <- ddply(m, c("QGroup","Income.Group"), summarize, All = length(CountryCode))
Error: argument "by" is missing, with no default

> g <- ddply(m, c("QGroup","Income.Group"), summarise, All = length(CountryCode))

Yes, that's right - "summarise" works fine but "summarize" apparently takes different arguments.

Trouble installing jpeg R package on OSX?

I'm taking the Coursera "Getting and Cleaning Data" class in R and one of the quizzes requires you to use the jpeg package. The R package install was failing for some reason though:

install.packages("jpeg")

In file included from read.c:1:
./rjcommon.h:11:10: fatal error: 'jpeglib.h' file not found
#include 
         ^

I use homebrew, so I tried the usual...

brew install jpeg
Warning: jpeg-8d already installed

brew unlink jpeg && brew link jpeg

But that didn't fix the issue. The solution was to install the XCode command line tools with:

xcode-select --install

(As a side note, this is my development machine so I already had XCode command line tools but for whatever reason I had to install them again, maybe an XCode upgrade?)

Sometimes the best code is that which is never written

Someone wrote some moderately over-complicated code to provide functionality in a service. That developer has moved on. A little time passes and suddenly there are a bunch of stack traces popping up in the logs indicating that a status message isn't getting published.
First reaction - dig into the error, figure out exactly what's wrong and fix it, ASAP!

Here's the error:  Invalid parameter: Message too long (Service: AmazonSNS; Status Code: 400; Error Code: InvalidParameter; Request ID: <redacted>)

Seems pretty straightforward; two seconds of Googling turns up that SNS messages have a limit of 256KB. Go into the appropriate class and add something detect how many bytes are in the message. Hmmm... maybe these messages are really large, and our app has a ton of threads - don't want to use up too much memory. A quick check on StackOverflow suggests using a CountingInputStream (there's one in the Apache commons-io, but it's also pretty trivial to write). Add it. Write a couple tests to make sure that it counts bytes correctly for UTF-8, UTF-16, etc... Looking good and have only spent 20mins so far fixing this problem.

Now that we can reliably detect the message size, we need to deal with ones that are too large for SNS... Uh-oh, the message body is actually something getting serialized into a JSON payload, we can't simply truncate or the message won't be valid JSON any more. This will require some thought, but I'm totally up for the challenge!

STOP! Take a step back from the technical details and think about what problem you're trying to solve.

Basically, the point of this code was to "make sure a notification goes out that something failed to process". There's nothing in there to suggest that the entire response needs to be serialized and sent on the wire (although that's what it was doing originally). Think to yourself, "what's the simplest thing that could work?". In this case, all that really needs to be sent is an ID and a status of success or failure:

{ id: 1234, status: "failure" }

There's NO way that message will ever approach 256KB, so do I need to worry about truncating this message? No. Looking back on it, did I need a CountingOutputStream even with the truncating solution? Probably not, it was a JSON payload in UTF-8; UTF-8 handles the ASCII range in 1 byte per character so simply getting the length of the String should have been "close enough".

In the end, I had to write a negligible amount of code to create that new JSON payload, and I removed a bunch of complexity from the original codebase along the way.

The take-away from this exercise? Step back and understand the problem-space from the start and resist the impulse to dive in and fix the errors immediately.