Decrypting something encrypted with OpenSSL passphrase in Golang

For various reasons, it's easiest to deploy configuration settings at our company as java properties files.  Java properties files usually contain lines in the form of {key}={value}, with a few quirks around encoding, but nothing too difficult to handle.

My app is written in Go, so it's a little strange to use java-style properties, but they're easy enough to parse and load into a map[string]string.  

Things get a little more interesting when those keys and/or values are sometimes encrypted with OpenSSL using AES-256-CBC and a passphrase.

Golang already has quite a few packages available in crypto including AES, so I thought this should be straightforward, especially since the password to decrypt would be provided by the user.  

First question is regarding aes.NewCipher - it takes a []byte, the length of which is used to determine if you use AES-128, AES-192 or AES-256.   I know the properties were encrypted using AES-256-CBC, therefore I need a 32 byte key,  but all I have is a password. You can easily get the key, salt and initialization vector (iv) by using the OpenSSL command-line tool, but it took quite a bit of digging to figure out how to do that programmatically.

# Using the openssl CLI
$ echo "this is a test" | /usr/bin/openssl enc -e -aes-256-cbc -pass pass:password -base64
U2FsdGVkX1+ywYxveBnekSnx6ZP25nyPsWHS3oqcuTo=

$ echo "U2FsdGVkX1+ywYxveBnekSnx6ZP25nyPsWHS3oqcuTo=" | /usr/bin/openssl enc -d -aes-256-cbc -pass pass:password -base64 -p
salt=B2C18C6F7819DE91
key=98350CE8088E5A4F4E8B31830D20E20DAFD7881970E1487837FDA01F86B9166C
iv =2436176AB793A101FC7427DB662B84A2

$ echo "U2FsdGVkX1+ywYxveBnekSnx6ZP25nyPsWHS3oqcuTo=" | /usr/bin/openssl enc -d -aes-256-cbc -pass pass:password -base64
this is a test

A couple posts/stackoverflow questions point to EVP_BytesToKey which sorta implies that you just need to MD5 the passphrase and salt until you have enough bytes to make up the key and iv.

The next question, "how do I get the salt?", turned out to be super easy.  In the OpenSSL implementation, you just have to turn the encrypted base64 string into a []byte and grab the first 16 bytes.  Those bytes are always the string "Salted_" followed by the actual 8 bytes of salt.

Now that we have the passphrase and salt, we just have to repeatedly generate MD5 sums from the previous MD5 sum (start with an empty []byte for the first iteration) concatenated with the passphrase concatenated with the salt.  Each iteration gives you 16 bytes so for the 32 byte AES-256 key plus the 16 byte iv it needs to be run it three times.

After you have the key and iv, it's easy to decrypt using Go - the tricky part was figuring out how to go from a passphrase to the key and iv.

Anyway, here's the final code, hope it's helpful to someone: http://play.golang.org/p/r3VObSIB4o

Set complement between two files

Set Complement (or what elements from file1 are not in file2)

$ sort file2 file2 file1 | uniq -u

Serving up static files from Go / Gorilla mux

I've been playing with Go lately, just serving up a non-trivial API to kick the tires a bit.

One of the things I wanted to do was use the Swagger (https://helloreverb.com/developers/swagger) framework to document that API (I've done this before in Ruby with grape and it was a breeze).  (BTW - it was considerably more tedious to write the specification (even using the awesome https://github.com/wordnik/swagger-editor) but that's a different post...)

I needed to host the swagger-ui files (which they helpfully bundle into a 'dist' folder) and serve them up from my Go application.  The directory structure looked like this:

├── server.go

├── swagger-ui/

│   ├── css/

│   ├── images/

│   ├── index.html

│   ├── lib/

It ended up being fairly trivial to serve static files but took a little experimentation to get the Gorilla mux piece working correctly.  Here's what I ended up doing, inside 'server.go'

func main() {
router = mux.NewRouter()
router.HandleFunc("/", indexHandler)
router.HandleFunc("/api-docs", apiDocHandler)
router.PathPrefix("/swagger-ui/").Handler(http.StripPrefix("/swagger-ui/", http.FileServer(http.Dir("./swagger-ui/"))))
http.Handle("/", router)
err := http.ListenAndServeTLS(":8443", "cert.pem", "key.pem", router)
if err != nil {
log.Fatalln(err)
}
}

Important parts to realize:
1) you should use the PathPrefix so that the mux picks up everything inside that folder
2) you need to strip out the path prefix so that your files are served up from the correct directory, otherwise you'll end up making requests for /swagger-ui/swagger-ui/ which won't work.

Docker and Google Compute Engine

Participated in a hackathon (hackforchange.org) this past weekend and there were some Google credits available so I figured I'd see how easy it was to get Docker running on Compute Engine. Turns out, very easy thanks to this post from Google: Containers on Google Cloud Platform.
The only real gotcha that I ran into was that I ran out of disk space almost immediately after installing Docker registry. What?! Yup, since Compute Engine starts you out with an 8GB disk you need to:

1. Provision a bigger disk and attach it
2. Move Docker onto the bigger disk

Attaching more disk is thoroughly documented by Google (see: Disks) but it took quite a bit of searching to figure out how to move Docker so I figured I'd post with what I did.

We're going to assume I added a bigger disk and mounted it to /docker-storage.

1. Stop docker - 'service docker stop'
2. Edit /etc/default/docker - you're looking for the line that sets DOCKER_OPTS.  You want to add a -g flag and where you want Docker to run from.  In my example, I prepended

   -g=/docker-storage/docker

3. Move the files from the default Docker location (/var/lib/docker) to the new mount:  'cd /var/lib; mv docker /docker-storage/'
4. Restart docker - 'service docker start'

That should be all it takes.

Quick tidbits

One of the reasons I like Project Euler is that it gives you a chance to expand your knowledge of whatever language you're using...

For instance, today I learned about Fixnum#fdiv (forces float division instead of integer division) in Ruby:

2/4       # 0
2.div(4)  # 0
2.fdiv(4) # 0.5

Sure there's a ton of other ways to do it, but this was succinct and to the point.

And the RSpec =~ operator for array equality - it's basically the same thing as ==, except that order doesn't matter.

[1,2] == [2,1] # false
[1,2] =~ [2,1] # true

Two simple things that I'd not had a reason to use before. Will I use them all the time, nope. But I'm glad to have found them.

Logging sql queries from korma

Wanted to see what sql is actually getting executed from a webapp running korma - seems fairly easy since I stumbled upon blackwater.
Inside your project.clj, add blackwater and clj-time...

(defproject foo "0.1.0-SNAPSHOT" 
  :description "Zap bug tracker"
  :dependencies [
    ;; other deps omitted
    [blackwater "0.0.9"]
    [clj-time "0.6.0"]]

Inside your ns:

(ns.foo
  (:use korma.db korma.core)
  (:require [black.water.korma :refer [decorate-korma!]]))

(decorate-korma!)

You'll get nice output like:

SELECT "project".* FROM "project"| took:31ms
SELECT "project".* FROM "project" WHERE ("project"."id" = ?)| took:4ms

Reloading lein repl

So, I'm working through the book Seven Web Frameworks in Seven Weeks from the awesome Pragmatic Programmers and I'm on the Clojure section (keep in mind that I don't know Clojure ^^) but quitting/restarting to get the repl to pick up changes just plain sucks.

Here's a solution I found on SO:

Add tools.namespace to your project.clj dependencies.

e.g.

:dependencies [
  # ...
  [org.clojure/tools.namespace "0.2.4"]]

After you start up the lein repl:

user=> (use '[clojure.tools.namespace.repl :only (refresh)])
nil

... make some changes to a clj file ...

user=> (refresh)
:reloading (hello.core hello.core-test zap.models)
:ok

It's not 100% automatic, but it beats quitting/restarting. Though it does appears you have to type that use statement every time after you refresh...