2017.04.10 VimWiki Gateway

I've been having lots of fun with lately! I've mixed in a bit of encryptfs to have a private directory of notes. Sometime during boot, I set it up with:

sudo mount -t ecryptfs \
  -o key=passphrase,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_passthrough=no,ecryptfs_enable_filename_crypto=yes,ecryptfs_fnek_sig=1a9380706fdf5bce \
  ~/docs/private/vimwiki ~/notes

I haven't done much customization yet. But one thing I'm playing with now is making it easy to post notes from VimWiki directly onto my website.

Add Comment

2017.01.11 Slack Hash Store

Last night at the DC Perl Mongers meetup we collaboratively built a proof-of-concept for storing key/value pairs as messages in a slack channel, This was made straightforward by great Slack API docs and WebService::Slack::WebApi. Example usage:

# writes "hello=world" into the #globalhash channel
$ globalhash set hello world

# reads the #globalhash channel for the most
# recent value of hello and prints it
$ globalhash get hello

There are of course a TON of things you could do from here. Some ideas:

  • More features: list of keys, delete, complex values, binary values, value history
  • Clean implementation and better usage/error messaging, docs
  • Port to other languages -- perl6 and still using the perl5 module would be fun
  • Register as a slack app that anyone can use, fix up OAuth or whatever

Add Comment

2016.09.23 All The Methods

A fun thing to do is to explore things using introspection/reflection. In Ruby and Perl6, for example, we can get a list of methods for a given object instance pretty easily:

# Ruby
"hello good people!".methods.each { |method| puts method.to_s }

# Perl6
for "hello good people!".^methods -> $method { say $method }

On the REPL (irb/pry or perl6) this is even shorter since it prints out lists of things by default, so you can do:

# Ruby
"hello good people!".methods

# Perl6
"hello good people!".^methods

One difference you see here is that Perl6 has a separate way to call meta-methods, whereas Ruby provides them directly. In some other languages you need to call into a separate reflection library and give it your object and what you want from it.


This is a great and devious way to answer the question "what can I do with this object?". In Ruby's pry you can use the "ls" command like "ls foo" to get an even prettier version.

But you know... why stop there? Let's call the methods.

All of them.

In Perl6 that looks like:

sub all-the-methods($thing) {
  for $thing.^methods -> $method {
    say "{$} => {$thing.clone.$method.gist}";
    CATCH { default { say "{$} => ERROR" } }

all-the-methods <this is words>;
all-the-methods "hello little fishies!";

Taking the list of methods, we loop over them and invoke them one at a time. Here we do ".clone" so that if it is a mutating function we'll work on a copy. Note that ".clone" is NOT a deep-copy, so while it works in my simple list and string examples it might not work so well for other things. The call to .gist gives us a nice printable version of the result.

We invoke the method with no parameters, and many methods don't like that. So here we provide a CATCH block that just prints out a generic message -- it's inside the loop so after catching an error it'll just go to the next method.

Here is some of the output:

# <this is words>
permutations => ((this is words) (this words is) (is this words) (is words this) (words this is) (words is this))
join => thisiswords
pick => is
roll => words
reverse => (words is this)
rotate => (is words this)
append => ERROR

# "hello little fishies!"
ords => (104 101 108 108 111 32 108 105 116 116 108 101 32 102 105 115 104 105 101 115 33)
wordcase => Hello Little Fishies!
flip => !seihsif elttil olleh
chop => hello little fishies
contains => ERROR

Lots of the functions just result in 'ERROR', especially for those pesky methods that take some sort of parameter. A lot of these methods, however, work ok with no params and return some value.

In Ruby this looks like:

def all_the_methods(thing)
  thing.methods.each do |method|
      if method =~ /^pry|byebug|debugger$/
        puts "#{method} => SKIPPING"
      puts "#{method} => #{ thing.clone.send(method) }"
      puts "#{method} => ERROR"

all_the_methods %w(this is words)
all_the_methods "hello little fishies!"

Here we had to explicitly skip the pry/byebug things, at least for my REPL execution, because those entered a subshell which isn't what I'm going for. Similar to Perl6 we also clone the object (also a shallow-clone) and had to handle exceptions, which are again almost entirely from methods that take more than zero parameters. In Ruby interpolated values automatically get .to_s called on them, so we don't need to do anything there.

One very notable thing is that operators in Ruby are methods directly on the object! This is nice for lots of reasons, including that you can easily override them. In Perl6 operators are standalone subs that do typed multi-dispatch (pattern matching) to decide what to execute, so the code isn't usually directly part of the class itself even if it ends up calling methods on the class or messing with the state.

Some of the output:

# %w(this is words)
shuffle! => ["words", "is", "this"]
include? => ERROR
permutation => #<Enumerator:0x00000002e48188>
combination => ERROR
sample => is
sort => ["is", "this", "words"]
count => 3
first => this
== => ERROR

# "hello little fishies!"
succ! => hello little fishiet!
downcase! => 
capitalize => Hello little fishies!
codepoints => [104, 101, 108, 108, 111, 32, 108, 105, 116, 116, 108, 101, 32, 102, 105, 115, 104, 105, 101, 115, 33]
reverse! => !seihsif elttil olleh
center => ERROR

Again most of the results are 'ERROR', especially where they expect more arguments. Another interesting thing is 'permutation', which returns an enumerator rather than a list, and isn't forced to evaluate upon rendering or anything. Also I notice here that 'downcase!' doesn't return the result of downcasing :)


Since I'm going all polyglot here anyway, I was starting to wonder which methods were equivalent. I could read through and guess, but that's no fun. Instead I made an alternate version of the above that does two things -- it reverses the method-result relationship so that we get a list of methods that return a specific result. Then I also output in a more language-neutral JSON format to compare across languages, and and turn that into a list of methods and their result prefixed with the language name.

perl6-rotate ruby-rotate ruby-rotate! => [ "is", "words", "this" ]

perl6-words ruby-split => [ "hello", "little", "fishies!" ]

perl6-ords perl6-NFC perl6-NFD perl6-NFKC perl6-encode perl6-NFKD ruby-codepoints ruby-bytes => [
  104, 101, 108, 108, 111, 32, 108, 105, 116, 116, 108, 101, 32, 102, 105, 115, 104, 105, 101, 115, 33

perl6-lines ruby-lines => [ "hello little fishies!" ]

perl6-reverse ruby-reverse! ruby-reverse => [ "words", "is", "this" ]

perl6-comb ruby-chars => [
  "h", "e", "l", "l", "o", " ", "l", "i", "t", "t", "l", "e", " ",
  "f", "i", "s", "h", "i", "e", "s", "!"

perl6-sort ruby-sort ruby-sort! => [ "is", "this", "words" ]

perl6-Slip perl6-values perl6-tree perl6-unique perl6-List perl6-Array
perl6-words perl6-squish perl6-item perl6-clone perl6-flat perl6-eager
perl6-Seq perl6-FLATTENABLE_LIST perl6-reification-target perl6-ZEN-KEY
perl6-ZEN-POS perl6-return-rw perl6-cache perl6-self perl6-lazy perl6-return
ruby-untaint ruby-unshift ruby-freeze ruby-dup ruby-compact ruby-uniq
ruby-taint ruby-itself ruby-to_ary ruby-untrust ruby-clone ruby-push
ruby-entries ruby-flatten ruby-to_a ruby-trust => [
  "this", "is", "words"

That is a lot fewer than I expected! I had to run the perl6 with .methods(:all) to get ".sort", which was surprising. This probably misses others where my JSON encoding normalization trick doesn't work.

Add Comment

2016.09.05 Test Suite Debugging Time

One of the codebases I work on regularly has a large and slow test suite. So slow that it typically only runs in totality in a continuous-integration (ci) environment, and there it uses the parallel_tests gem to slice it into pieces and run in parallel. This gets it to run in like 30 minutes instead of 2.5 hours.

But I want to run the whole thing linearly sometimes, especially so I can make sure tests aren't conflicting with each other. So I created a new ci job that runs it linearly ... and got a fail that I didn't get otherwise. Took forever to figure it out!

This was about a worker that I call a "sweeper" -- it's job is to run once a day and make sure nothing was missed by other realtime jobs during the day. Here is a simplified version.

class SweeperWorker
  INTERVAL = 25.hours.ago # Overlap the day by an hour
  def perform
    candidates.each(&:do_work) # something like that
  def candidates
    Thing.where('created_at > ?', INTERVAL)

# rspec
describe SweeperWorker do
  let(:new_thing) { 5.hours.ago) }
  let(:old_thing) { 26.hours.ago) }

  it "does pick up the new thing" do
    new_thing # force to exist
    expect( include(new_thing)

  it "does not pick up the old thing" do
    old_thing # force to exist
    expect( include(old_thing)

Not sure that is quite right, but you should get the idea.

The second example was failing -- both the "old thing" and the "new thing" were getting into the sweeper window -- but only when the test suite was run linearly. Run it in ci, no problem. Run just this test/file locally, no problem.

The suspect is thus "time" -- but where? I went down the wrong track for a while, looking into timezones. Rails vs MySQL on ci vs my machine and so on ... some of those things weren't quite matched up, but still. I ran the test over and over, adding more and more debugging so I could peer into the contents of variables and the DB. Everything looked fine -- all the timestamps were what I expect. So it must be the query itself that is somehow asking for a longer timespan.

Finally I got it! Right there in the first line of SweeperWorker:

INTERVAL = 25.hours.ago # Overlap the day by an hour

This constant is set when the class is loaded, and never changed! So by the time this test runs in the 2.5 hours of test time, "25 hours ago" is now "27.5 hours ago" -- so our "old" thing is in our window. The "25.hours.ago" certainly looks relative, but it is a calculation that outputs an absolute time which is then saved for later use.

Move that constant into the method as a variable and it is fixed. And now we can sing our victory song!


  • Beware of calculated constants
  • Make sure relative values are actually relative
  • If the speed of execution (slow vs fast) changes the outcome, look for things that are more or less spread out in the timeline, such as constant definition.

Add Comment

2016.08.12 ICFP Contest 2016 - Origami Folding

Challenge: Folding Origami

This year the ICFP will be held in Nara, Japan. The contest was organized by members of The University of Electro-Communications. I checked out their website since that sounded like a weird school name, and see that they are also hosting "The Joint Conference of the 8th International Conference on ESP in Asia & the 3rd International Symposium on Innovative Teaching and Research in ESP in Japan". Unfortunately "ESP" here is for "English for Specific Purposes" rather than the usual usage.

Unlike other years, the contest started at 00:00 UTC, which is 8:00pm Thursday, and lasted until 8:00pm Saturday. This was a nice schedule to start with, getting to roll into the contest right after work, but it did leave me jazzed up on Sunday night after it was over.

I was joined at my house by Mike, and Jason joined us remotely from Portland. We coordinated by google hangouts, irc, and github. Near the end I turned on the github-notification hook that posted to irc upon pushes, and should have done that a lot earlier.

First we read through the problem, which was very tricky to wrap our heads around as usual, despite it's somewhat minimalistic Japanese aesthetic. We drew on the whiteboards and drew on paper and folded some paper. Apparently I'm not supposed to use scissors to do origami, who knew?

The rough idea is that we are given the shadow, or "silhouette", of a folded origami flattened to 2D. We then need to reconstruct it by starting with an unfolded paper and then trying to fold it until we get the same shadow. During the lightning round we were given 101 problems of this form. After the lightning round we could submit new problems of our own invention and solve problems submitted by other contestants. We submit our solutions as we go and acquire points based on how well we solved a problem, how big in bytes was our solution, and how easy it was to solve (based on how many others solved it). They even provided a nice leaderboard.

I talked Mike into programming in Perl 6 :) . While he and Jason worked on understanding the rules of folding and solving a few problems by hand, I whipped up a problem parser using Perl 6 Grammars, which worked perfectly. I had fortunately practiced a bit before. I built a simple caching API layer to get problems from the contest server, just shelling out to curl basically.

After that I did a very basic visualization using Imager (via Inline::Perl5), which Mike then took over and worked on. Then (and we're talking Friday or Saturday) I started working on modeling folding of an origami. And then I got stuck.

g_polygon.jpg I didn't realize it until I was taking a shower on Monday morning after the contest, but I was attempting to implement too general of a solution for part of it. The rough idea is to consider an origami as a bunch of polygons. You start off with one, and then when you fold it you end up with two, one of which you reflect along the fold line.

So I set out to chop up a polygon given a line, and unfortunately didn't think through my goal clearly enough -- I tried to split ANY polygon. That includes funky looking concave polygons that when you cut with a single line might split into a bunch of pieces. What I realized after the contest was that a single cut starting from a solid piece of paper can never end up that way -- there is no way to cut it into a funky shape! So I lost a lot of time there.

The other thing that cost time was reflection. We cast about for some off the shelf libraries, but got mad at them, so I just started to implement reflection myself. I even thought I had it working, but sometime early on Sunday I realized something was fishy. I switched to an off-the-shelf implementation via Math::Geometry::Planar and presto! Worked much better. Should have used that from the beginning.

origami_fold.jpg Meanwhile Mike was both improving the visualization, especially around setting a useful viewport for shadows that had been moved far away from the axis origins. He also went through and generated images for the existing problem, which helped us pick out good test cases. Jason kept at it too, starting to hand-generate us some problems to submit once we got past the lightning round, and hand-solving some of the problems that were provided.

After visualizations were working, Mike connected the dots between problems and the origami modeling to calculate a score, making further use of the Planar library. This gave us the final ingredient we needed for automated exploration of simple folds in origami space.

Finally, late on Sunday, we got to where I wanted to be by Saturday morning. Given a problem we could start with an unfolded square and then randomly fold, keeping the improvements. Basic hill-climbing, in other words. I consider this stumbling across the finish line ... but we made it :)


Add Comment


META INFORMATION: This is the technical blog and wiki of Brock Wilcox (awwaiid). Entries focus on my current projects, interests, and sometimes life events. If you'd like you can check out the list of All Entries or the RSS Feed. I also have a LiveJournal syndication feed for LJ friends.


Blog Blog RSS Feed



Follow @awwaiid




Wiki Edits Wiki RSS Feed

... more changes