THE LACK THEREOF

2017.05.21 Core Logic

There are many tools in my programming toolbox, but one that I've felt is missing is constraint and logic programming. I've done a spot of Prolog and played with some inference systems of various types ... but even when I run into something where I think it's the right hammer... I don't have a go-to tool.

I got The Reasoned Schemer a while back, hoping to use it to learn clojure's core.logic or similar, but the writing still KILLS me. Hate it so much. It is like the worst extreme sort of Socratic Method that I can imagine, literally everything has two columns of question and answer. But the QUESTIONS are obtuse! Working through it that way is not the sort of enlightenment that I'm going for ... I don't want to apply a unification algorithm on my understanding of logic programming with the contents of the book! The programming system might be "backwards" that way, but my way of learning isn't.

Anyway. One dumb thing that I had a hard time getting over was things ending in -o, like conso and emptyo. But tonight I had a breakthrough! See those 'o' things, according to the internets, are supposed to be sort of like question-marks except describing a goal. So now if I read the 'o' as '-objective' (as in, "goal") then they sound much better. cons-objective, empty-objective. Declaring the goal of the statement.

I guess I should go back and try reading The Reasoned Schemer with that in mind, and see if it makes me less angry :)

Add Comment

2017.04.10 VimWiki Gateway

I've been having lots of fun with https://github.com/vimwiki/vimwiki lately! I've mixed in a bit of encryptfs to have a private directory of notes. Sometime during boot, I set it up with:

sudo mount -t ecryptfs \
  -o key=passphrase,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_passthrough=no,ecryptfs_enable_filename_crypto=yes,ecryptfs_fnek_sig=1a9380706fdf5bce \
  ~/docs/private/vimwiki ~/notes

I haven't done much customization yet. But one thing I'm playing with now is making it easy to post notes from VimWiki directly onto my website.

Add Comment

2017.01.11 Slack Hash Store

Last night at the DC Perl Mongers meetup we collaboratively built a proof-of-concept for storing key/value pairs as messages in a slack channel, https://github.com/plicease/globalhash. This was made straightforward by great Slack API docs and WebService::Slack::WebApi. Example usage:

# writes "hello=world" into the #globalhash channel
$ globalhash set hello world

# reads the #globalhash channel for the most
# recent value of hello and prints it
$ globalhash get hello
world

There are of course a TON of things you could do from here. Some ideas:

  • More features: list of keys, delete, complex values, binary values, value history
  • Clean implementation and better usage/error messaging, docs
  • Port to other languages -- perl6 and still using the perl5 module would be fun
  • Register as a slack app that anyone can use, fix up OAuth or whatever

Add Comment

2016.09.23 All The Methods

A fun thing to do is to explore things using introspection/reflection. In Ruby and Perl6, for example, we can get a list of methods for a given object instance pretty easily:

# Ruby
"hello good people!".methods.each { |method| puts method.to_s }

# Perl6
for "hello good people!".^methods -> $method { say $method }

On the REPL (irb/pry or perl6) this is even shorter since it prints out lists of things by default, so you can do:

# Ruby
"hello good people!".methods

# Perl6
"hello good people!".^methods

One difference you see here is that Perl6 has a separate way to call meta-methods, whereas Ruby provides them directly. In some other languages you need to call into a separate reflection library and give it your object and what you want from it.

Cool!

This is a great and devious way to answer the question "what can I do with this object?". In Ruby's pry you can use the "ls" command like "ls foo" to get an even prettier version.

But you know... why stop there? Let's call the methods.

All of them.

In Perl6 that looks like:

sub all-the-methods($thing) {
  for $thing.^methods -> $method {
    say "{$method.name} => {$thing.clone.$method.gist}";
    CATCH { default { say "{$method.name} => ERROR" } }
  }
}

all-the-methods <this is words>;
all-the-methods "hello little fishies!";

Taking the list of methods, we loop over them and invoke them one at a time. Here we do ".clone" so that if it is a mutating function we'll work on a copy. Note that ".clone" is NOT a deep-copy, so while it works in my simple list and string examples it might not work so well for other things. The call to .gist gives us a nice printable version of the result.

We invoke the method with no parameters, and many methods don't like that. So here we provide a CATCH block that just prints out a generic message -- it's inside the loop so after catching an error it'll just go to the next method.

Here is some of the output:

# <this is words>
permutations => ((this is words) (this words is) (is this words) (is words this) (words this is) (words is this))
join => thisiswords
pick => is
roll => words
reverse => (words is this)
rotate => (is words this)
append => ERROR

# "hello little fishies!"
ords => (104 101 108 108 111 32 108 105 116 116 108 101 32 102 105 115 104 105 101 115 33)
wordcase => Hello Little Fishies!
uc => HELLO LITTLE FISHIES!
flip => !seihsif elttil olleh
chop => hello little fishies
contains => ERROR

Lots of the functions just result in 'ERROR', especially for those pesky methods that take some sort of parameter. A lot of these methods, however, work ok with no params and return some value.

In Ruby this looks like:

def all_the_methods(thing)
  thing.methods.each do |method|
    begin
      if method =~ /^pry|byebug|debugger$/
        puts "#{method} => SKIPPING"
        next
      end
      puts "#{method} => #{ thing.clone.send(method) }"
    rescue
      puts "#{method} => ERROR"
    end
  end
end

all_the_methods %w(this is words)
all_the_methods "hello little fishies!"

Here we had to explicitly skip the pry/byebug things, at least for my REPL execution, because those entered a subshell which isn't what I'm going for. Similar to Perl6 we also clone the object (also a shallow-clone) and had to handle exceptions, which are again almost entirely from methods that take more than zero parameters. In Ruby interpolated values automatically get .to_s called on them, so we don't need to do anything there.

One very notable thing is that operators in Ruby are methods directly on the object! This is nice for lots of reasons, including that you can easily override them. In Perl6 operators are standalone subs that do typed multi-dispatch (pattern matching) to decide what to execute, so the code isn't usually directly part of the class itself even if it ends up calling methods on the class or messing with the state.

Some of the output:

# %w(this is words)
shuffle! => ["words", "is", "this"]
include? => ERROR
permutation => #<Enumerator:0x00000002e48188>
combination => ERROR
sample => is
sort => ["is", "this", "words"]
count => 3
first => this
== => ERROR

# "hello little fishies!"
succ! => hello little fishiet!
upcase => HELLO LITTLE FISHIES!
downcase! => 
capitalize => Hello little fishies!
codepoints => [104, 101, 108, 108, 111, 32, 108, 105, 116, 116, 108, 101, 32, 102, 105, 115, 104, 105, 101, 115, 33]
reverse! => !seihsif elttil olleh
center => ERROR

Again most of the results are 'ERROR', especially where they expect more arguments. Another interesting thing is 'permutation', which returns an enumerator rather than a list, and isn't forced to evaluate upon rendering or anything. Also I notice here that 'downcase!' doesn't return the result of downcasing :)

BONUS

Since I'm going all polyglot here anyway, I was starting to wonder which methods were equivalent. I could read through and guess, but that's no fun. Instead I made an alternate version of the above that does two things -- it reverses the method-result relationship so that we get a list of methods that return a specific result. Then I also output in a more language-neutral JSON format to compare across languages, and and turn that into a list of methods and their result prefixed with the language name.

perl6-rotate ruby-rotate ruby-rotate! => [ "is", "words", "this" ]

perl6-words ruby-split => [ "hello", "little", "fishies!" ]

perl6-ords perl6-NFC perl6-NFD perl6-NFKC perl6-encode perl6-NFKD ruby-codepoints ruby-bytes => [
  104, 101, 108, 108, 111, 32, 108, 105, 116, 116, 108, 101, 32, 102, 105, 115, 104, 105, 101, 115, 33
]

perl6-lines ruby-lines => [ "hello little fishies!" ]

perl6-reverse ruby-reverse! ruby-reverse => [ "words", "is", "this" ]

perl6-comb ruby-chars => [
  "h", "e", "l", "l", "o", " ", "l", "i", "t", "t", "l", "e", " ",
  "f", "i", "s", "h", "i", "e", "s", "!"
]

perl6-sort ruby-sort ruby-sort! => [ "is", "this", "words" ]

perl6-Slip perl6-values perl6-tree perl6-unique perl6-List perl6-Array
perl6-words perl6-squish perl6-item perl6-clone perl6-flat perl6-eager
perl6-Seq perl6-FLATTENABLE_LIST perl6-reification-target perl6-ZEN-KEY
perl6-ZEN-POS perl6-return-rw perl6-cache perl6-self perl6-lazy perl6-return
ruby-untaint ruby-unshift ruby-freeze ruby-dup ruby-compact ruby-uniq
ruby-taint ruby-itself ruby-to_ary ruby-untrust ruby-clone ruby-push
ruby-entries ruby-flatten ruby-to_a ruby-trust => [
  "this", "is", "words"
]

That is a lot fewer than I expected! I had to run the perl6 with .methods(:all) to get ".sort", which was surprising. This probably misses others where my JSON encoding normalization trick doesn't work.

Add Comment

2016.09.05 Test Suite Debugging Time

One of the codebases I work on regularly has a large and slow test suite. So slow that it typically only runs in totality in a continuous-integration (ci) environment, and there it uses the parallel_tests gem to slice it into pieces and run in parallel. This gets it to run in like 30 minutes instead of 2.5 hours.

But I want to run the whole thing linearly sometimes, especially so I can make sure tests aren't conflicting with each other. So I created a new ci job that runs it linearly ... and got a fail that I didn't get otherwise. Took forever to figure it out!

This was about a worker that I call a "sweeper" -- it's job is to run once a day and make sure nothing was missed by other realtime jobs during the day. Here is a simplified version.

class SweeperWorker
  INTERVAL = 25.hours.ago # Overlap the day by an hour
  def perform
    candidates.each(&:do_work) # something like that
  end
  def candidates
    Thing.where('created_at > ?', INTERVAL)
  end
end

# rspec
describe SweeperWorker do
  let(:new_thing) { Thing.new(created_at: 5.hours.ago) }
  let(:old_thing) { Thing.new(created_at: 26.hours.ago) }

  it "does pick up the new thing" do
    new_thing # force to exist
    expect(SweeperWorker.new.candidates).to include(new_thing)
  end

  it "does not pick up the old thing" do
    old_thing # force to exist
    expect(SweeperWorker.new.candidates).not_to include(old_thing)
  end
end

Not sure that is quite right, but you should get the idea.

The second example was failing -- both the "old thing" and the "new thing" were getting into the sweeper window -- but only when the test suite was run linearly. Run it in ci, no problem. Run just this test/file locally, no problem.

The suspect is thus "time" -- but where? I went down the wrong track for a while, looking into timezones. Rails vs MySQL on ci vs my machine and so on ... some of those things weren't quite matched up, but still. I ran the test over and over, adding more and more debugging so I could peer into the contents of variables and the DB. Everything looked fine -- all the timestamps were what I expect. So it must be the query itself that is somehow asking for a longer timespan.

Finally I got it! Right there in the first line of SweeperWorker:

INTERVAL = 25.hours.ago # Overlap the day by an hour

This constant is set when the class is loaded, and never changed! So by the time this test runs in the 2.5 hours of test time, "25 hours ago" is now "27.5 hours ago" -- so our "old" thing is in our window. The "25.hours.ago" certainly looks relative, but it is a calculation that outputs an absolute time which is then saved for later use.

Move that constant into the method as a variable and it is fixed. And now we can sing our victory song!

Lessons:

  • Beware of calculated constants
  • Make sure relative values are actually relative
  • If the speed of execution (slow vs fast) changes the outcome, look for things that are more or less spread out in the timeline, such as constant definition.

Add Comment

More...


META INFORMATION: This is the technical blog and wiki of Brock Wilcox (awwaiid). Entries focus on my current projects, interests, and sometimes life events. If you'd like you can check out the list of All Entries or the RSS Feed. I also have a LiveJournal syndication feed for LJ friends.

Navigation

Blog

https://thelackthereof.org/pics/16px-Feed-icon.svg.png Blog RSS Feed

Tweets


2017-05-23

2017-05-19

2017-05-18

2017-05-15

Code

Follow @awwaiid

Wiki Edits

https://thelackthereof.org/pics/16px-Feed-icon.svg.png Wiki RSS Feed

... more changes