While you're reading this, keep in mind that I'm available for hire stupid!
Last week I wrote a tool for reverse engineering protobuf data called protofudger. I wanted to show it off to a few people, but didn’t want them to have to download it and run an unsigned binary on their system, or set up a whole go toolchain just to build one program. If only there were a better way!
TL;DR: take a look at this if you want to look at the end result. If you’d like a file to test with, try this one from node-protobuf’s test suite.
Enter GopherJS. If you haven’t heard of it, here’s an excellent description right from their home page:
GopherJS compiles Go code (golang.org) to pure JavaScript code. Its main purpose is to give you the opportunity to write front-end code in Go which will still run in all browsers.
This is perfect! I can take my command line program, compile it to JavaScript, put it on a page, and be done by dinner. Simple. Except, you know, it’s JavaScript. Which means you have to spend at least two hours getting your tooling set up right. This is a recollection of the steps I went through, the approaches I abandoned, and some problems I faced. Overall it wasn’t so bad, but I did run into some interesting roadblocks!
First task: get the program to compile nicely.
This was trivial. I ran gopherjs build
in my project directory, and it gave
me a protofudger.js
file, along with a source map. GopherJS scores major
points for being pain-free. Five stars, easy transaction, would download
again.
So now I have a command line program compiled to JavaScript. That’s pretty cool, but not actually very useful. It ran under Node.js just fine, which was again cool, but not immediately useful to me. When I put it into a web page, it ran and printed some output to the console. But… That’s not really a web UI. How do I get data into the program? How do I put the output on the page? Well, read on!
Second task: refactor the program to operate more like a library.
I had a typical go command line thing going on, using the flag
package for
arguments, the fmt
package to write stuff to the terminal, and everything
happening from start to end in a main
function. This… Did not transfer so
well to the web. You know, obviously. Makes sense in hindsight.
The meat of the matter was in refactoring all the Printf
statements out of the
parsing functions, and instead returning strings to be printed by the main
function. This was pretty straightforward, so I won’t go into too much detail.
Take a look at this commit
if you’d like to see the actual changes. I also had to make sure the parsing
functions weren’t reading stuff from the environment (e.g. command line args),
instead receiving them as parameters.
Third task: somehow bind the program to the UI.
I decided that I wanted the UI to remain largely unchanged. This means text output, some very limited binary options, and an input file.
On the command line, the most obvious way to do the “input file” bit was to
read from standard input, e.g. ./protofudger < file.dat
. I figured the best
conversion to the web would be to drop a file onto the web page. This meant
that I was going to have to get the data from a FileReader
to look like a
[]byte
for the go code. Again, most of the heavy lifting here is done by
GopherJS. If you define a go function as func([]byte)
, you can call it from
JavaScript with a Uint8Array
and it just works as you’d expect. In my case,
this looked like fn(new Uint8Array(reader.result))
.
The next thing would be the options. There are currently only two options for
protofudger - one to display all possible interpretations of numeric fields, and
one to display byte offsets alongside field numbers. These mapped pretty well to
checkboxes. It’s pretty easy to get the state of a checkbox, so I just do this:
parse(new Uint8Array(reader.result), showAll.checked, showOffsets.checked)
.
Output was easy. That parse
function returns an array of strings that I can
join together and shove into a <pre>
element for that fixed-width terminal
feel.
Fourth task: build a bundle that I can deploy.
This is where things got interesting! I went through a few different iterations here, and found some interesting limitations in the current best-of-breed JavaScript tools.
First I tried to build the protofudger
go code as a module that I could
import
from the main program. This actually works, but it’s not what I
settled on. This would be a boring post if I ended it with “yep, the first
thing I tried was perfect,” wouldn’t it?
GopherJS actually provides hooks to build CommonJS modules - in addition to a
global
handle, it provides a module
handle to go programs. This allows you
to do something like js.Module.Set("exports", ...)
, and whatever you expose
will be available via require('./file.js')
or import './file.js'
. This is
awesome, and it’s something that I think would be totally manageable if it
weren’t for some limitations in webpack and Babel. What it would work very
well for would be Node.js programs. With GopherJS you could very easily build
a wrapper around your go program for use in Node.
The actual problem that I had with this approach was memory exhaustion! GopherJS generates some pretty incredibly large JavaScript files - nearly a megabyte just for protofudger. It turns out that if you try to parse this with Babel, it’ll end up blowing through the 1.4GB heap limit in Node. That’s fair enough though, considering that most of the files Babel has to deal with are closer to several kilobytes than several hundred.
So, obvious solution: don’t run Babel over the GopherJS output. Yep! That worked. It built correctly, and I had a bundle that I could put into a page. But sourcemaps were broken!
Fifth task: get source maps working.
Okay, well, let’s take a look around at what’s available in webpack to work with this. I figure that I had some JavaScript, and an existing sourcemap file, so I could just feed the sourcemap in and webpack would be able to use it somehow. I know there’s some precedent for that in UglifyJS, so maybe there’s something for webpack too.
Well, yeah. There kind of is. There’s a package called source-map-loader, but it hasn’t seen any love for a while. There were a couple of bugs in it that I thought prevented it from operating at all (see issue 18), but even once I fixed them it still didn’t give me great results.
I really wanted to have working source maps, so I decided to go back to basics. When I included the GopherJS output directly in the page, they worked fine. The actual source code doesn’t show up, because GopherJS doesn’t include it in the source map, but all the line numbers and filenames were correct. I figured it wasn’t too horrible to have two files if it meant I had working source maps.
Conclusion: JS like it’s 2004.
What I ended up doing was treating the GopherJS output like a third party
library. I switched from setting module.exports
to setting
window.protofudger
in the GopherJS entry file, then I access that explicitly
within the UI code. This has upsides and downsides. One downside is that there
are multiple files now. This makes deployment a tiny bit more complex, as I
need to put two <script>
tags in the HTML file. One upside is that there are
multiple files now (lol). The browser can download the UI file at the same
time as the GopherJS code, which has (anecdotally) resulted in slightly faster
load times. The UI code makes sure that window.protofudger
exists before it
tries to use it, so I could even load the GopherJS output asynchronously if I
wanted to optimise things further.
I hope this is helpful for someone out there, and I hope I’ve littered this post with enough keywords to catch people googling for “gopherjs webpack source maps,” to warn them off the path I tried to walk!