2015.10.18 My First Useful Perl6 Grammar

Tags: Perl6

In my recent talks about perl6, one thing that I keep neglecting is Grammars. Part of that is because I hadn't used them yet!

I was putting together a talk on some clojure code, and was getting mad at my slide software not syntax-highlighting with magical rainbow parenthesis. Clojure being a LISP-like, it lacks some visual clues for structure and is WAY easier to glance at (in my opinion) when parenthesis are highlighted. I have vim doing this for me... but no dice in my slides.

So! Slice up slides, run them through vim, stitch them back together, and presto!

Complete code is at

I use Pinpoint as my go-to slide software. Simple and easy both to develop a deck and to present it. Each slide is a header and a body done in some Pango markup (more or less simplified HTML). Here is a 3 slide deck:

#!/usr/bin/env pinpoint

# Defaults for all slides
[font=Sans 90]


Some other slide content.

-- [font=monospace 99] [code=sh]


\# A lovely cgi application

echo "Content-type: text/html"
echo "Hello, world! Query:"

-- [font=monospace 99] [code=python]

\# Python WSGI

def application(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/plain')])
    yield 'Hello, world!'

We'll do this in two parts.

Part 1: The Grammar

#!/usr/bin/env perl6

grammar Pinpoint::Grammar {
  token TOP {

  token header { .*? <?before ^^ "--" > }

  token slide {

  token slide-header { ^^ "--" (\s* <slide-setting>)* $$ \n }

  token slide-setting { '[' .+? ']' }

  token slide-content { .*? <?before ^^ "--" > || .* $ }

Using 'grammar' is kinda like some funky cross between a regex and a class. That is -- everything is declared like a named regex, but the result is effectively a class that has a ".parse" method we can use. We're giving the whole grammar a name, and then declaring the file structure in the body.

'TOP' is the start of the parse, which will be an entire file for me. There is one special slide that is the header-slide, and then after that some content slides. Though each bit has a name, you do a lot of things just like other regexes -- so "<slide>+" means one-or-more slides.

After that it just gets broken down further and further. Probably the trickiest bit is the

<?before ^^ "--" >
pattern, which we see a few times. This says that if we see a "--" on the start of it's own line, then we've gone too far. Slide content has the "||" to say if we NEVER find the "--" then we should instead continue to the end of the file.

Part 2: The Processor

Now that we have a handy dandy grammar, we'll just slurp in whatever file/stdin we are given, parse, and the traverse the parse tree. There is a way to do this with Action objects (another class with methods for each of these tokens), but I didn't do that.

my $match = Pinpoint::Grammar.parse(slurp());

print $match<header>;

for $match<slide>.flat -> $slide {
  my $h = ~$slide<slide-header>;
  $h ~~ s/\s*\[code\=(.*?)\]//;
  print $h;

  if ~$slide<slide-header> ~~ /\[code\=(.*?)\]/ {
    my $cmd = "code-htmlize $0";
    my $filter = shell $cmd, :in, :out;
    say $filter.out.slurp-rest;
  } else {
    print $slide<slide-content>;

To make this easy, I print out the bits of the parse tree as I go. the '$h' is a chopped up version of a slide-header, removing my new and magical [code=foo] slide setting. Now that I'm looking at this, I see that I didn't actually traverse into the slide-setting list for each slide... I guess that would be better than the search/replace I'm doing here.

A specific thing to note is that "~" is forcing string-context here, so "~$slide<slide-header>" is taking the object for the slide-header node we are on and dumps it out as a string of the content. This makes sense, though I initially found the "~" usage a bit ugly.

Once we get down to a slide that has a "[code=foo]" style setting in the header, instead of printing the slide content like we do otherwise we'll run it through an external shell command. The "code-htmlize" command is some other script I wrote that takes in code and a language and outputs an HTML-colorized version using vim. Here we do the equivalent of Open2, getting a filehandle for the input and output of the external command, printing to it's input, reading from its output.

That's it! A pretty readable thing that parses and traverses my slide software file format, filtering the results as it goes.

Add Comment

2015.07.26 Exploring Clojure in the REPL

Tags: Clojure, REPL

When I first learned Linux, my friend taught me a few commands. How to to list files, how to change directories and see your current directory, how to run things, how to ask a program what parameters it takes, and how to look at a file (ls, cd, pwd, ./foo or /usr/bin/foo or foo, foo --help, cat foo or more foo). After that I just... ran a lot of stuff. I eventually discovered 'man' -- but going through /bin and running ALL the commands was really educational.

Let's get some basic tools to do that with the Clojure REPL! The major things we want to do are find things (mostly functions), and then learn all about them individually.

Finding Things

So Clojure has namespaces, which are kinda like directories, using "." as a way to indicate nesting. Vars (and thereby named functions) are kinda like files. So the full "path" to a var looks like "clojure.string/split" where "clojure.string" is the namespace and "split" is the function.

First let's get a list of all the loaded namespaces. The "ns-all" gives us this, but we want a nice sorted printout, so we'll add a bit of fancy.

user=> (doseq [n (sort (map ns-name (all-ns)))] (println n))
; ... trimmed ...
; ... trimmed ...
; ... trimmed ...

Use "dir" to look at all the exposed vars from a namespace

user=> (dir clojure.string)

Search for functions with "split" in their name. There is also "apropos" but it doesn't show namespaces -- "apropos-better" does, and "find-name" is an alias of that, so is better.

user=> (find-name "split")
(split-at split-with clojure.string/split clojure.string/split-lines net.cgrand.parsley.grammar/split-empty-prods)

You can also use a regex! Not sure how useful this is... but here we are getting all the functions that start with an "s" and in with a "t".

user=> (find-name #"^s.*t$")
(set short sort sorted-set spit split-at struct clojure.set/select clojure.string/split clojure.test/set-test net.cgrand.sjacket/shift net.cgrand.sjacket/shift-right net.cgrand.sjacket/str-pt net.cgrand.sjacket/subedit)

Learning About Things

Once you've found your function, you'll want to learn more about it and maybe give it a try.

"doc" can be used to get the documentation associated with a function. This is pretty cool! It shows the full name of the function, its signature, and it's documentation.

user=> (doc print)
([& more])
  Prints the object(s) to the output stream that is the current value
  of *out*.  print and println produce output for human consumption.

Note that you can tab-complete

user=> (doc print<tab>
print          print-ctor     print-dup      print-method   print-simple   print-str
printf         println        println-str

If you don't know what you're looking for, you can also try find-doc. This will search both the name and the documentation itself for your string or regex.

user=> (find-doc "split")
; ..... too long to include ;)

THE ULTIMATE: Get the source for a function!

user=> (source print)
(defn print
  "Prints the object(s) to the output stream that is the current value
  of *out*.  print and println produce output for human consumption."
  {:added "1.0"
   :static true}
  [& more]
    (binding [*print-readably* nil]
      (apply pr more)))

Not everything is a function. Heck, sometimes you might not know what sort of thing you're looking at. But we can find out.

user=> (type 5)
user=> (type 5.2)
user=> (type "hello")
user=> (type 'hello)
user=> (type {:a 5, :b 6})
user=> (type [1 2 3])
user=> (type type)
user=> (type dir)
CompilerException java.lang.RuntimeException: Can't take value of a macro: #'clojure.repl/dir, compiling:(/tmp/form-init5987247545872948247.clj:1:1) 
user=> (type clojure.string/split)
user=> (defn say-hi [] (println "hi!"))
user=> (type say-hi)

Getting GUI

Clojure comes with clojure.inspector, which gives some Swing GUI for exporing data structures.

user=> (use 'clojure.inspector)
user=> (inspect-tree {:a 1 :b 2 :c [1 2 3 {:d 4 :e 5 :f [6 7 8]}]})

NOTE: I had to do this to get jdk-1.7 swing apps to display while using xmonad (and other tiling window managers I guess). I now dropped this in my ~/.zshrc. Uhg.


Doing Horrible Things

So let's have some fun. First, let's get all the string function names. "dir-fn" is like dir, but returns the results instead of printing them.

user=> (def string-funcs (clojure.repl/dir-fn 'clojure.string))

Let's turn them into their actual functions.

user=> (def real-string-funcs (map #(resolve (symbol (str "clojure.string/" %))) string-funcs))

Given a function, here is how to tell how many parameters it takes (well... it might take more or less than this, but this will tell us how many params for the first definition. Good enough).

user=> (defn arg-count [f] (count (first (:arglists (meta f)))))

Here is a handy thing that tells us if the given function can deal with a single arg.

user=> (defn single-arg? [f] (= 1 (arg-count f)))

Now let's filter our string functions down to the ones that just take one argument.

user=> (def one-arg-funcs (filter single-arg? real-string-funcs))

So. We if have a one-argument function in the clojure.string namespace, let's assume it takes a string and pass in a string to it :) . The "pr-str" bit is so that we can pretty-print this with escaped newlines and such later.

user=> (defn one-arg-result [f] (pr-str (apply f ["Hello\nworld\n"])))

Now let's try this on all the one-argument string functions, printing out the function name and the result pretty like.

user=> (map #(println (:name (meta %))  "\"Hello\\nworld\\n\" ->" (one-arg-result %)) one-arg-funcs)
blank? "Hello\nworld\n" -> false
capitalize "Hello\nworld\n" -> "Hello\nworld\n"
join "Hello\nworld\n" -> "Hello\nworld\n"
lower-case "Hello\nworld\n" -> "hello\nworld\n"
re-quote-replacement "Hello\nworld\n" -> "Hello\nworld\n"
reverse "Hello\nworld\n" -> "\ndlrow\nolleH"
split-lines "Hello\nworld\n" -> ["Hello" "world"]
trim "Hello\nworld\n" -> "Hello\nworld"
trim-newline "Hello\nworld\n" -> "Hello\nworld"
triml "Hello\nworld\n" -> "Hello\nworld\n"
trimr "Hello\nworld\n" -> "Hello\nworld"
upper-case "Hello\nworld\n" -> "HELLO\nWORLD\n"


Add Comment

2015.06.04 Polyglot Bridges

Here are the slides for Polyglot Bridges (pdf) lightning talk that I gave today at RubyNation.

Mad Science: Polyglot Bridges from awwaiid

The most important bit:

# Shell
sudo apt-get install matplotlib
gem install rubypython
# pending a small patch for debian/ubuntu 64bit

# Ruby; even works in the REPL!
require 'rubypython'
plt = RubyPython.import('matplotlib.pyplot')

plt.ylabel('some numbers');


Or in other words: pull in this rubypython gem, magically get all the python libs callable from Ruby. In this case, you get all the nice charts and graphs from pyplot. I originally played with this in Perl6/Rakudo Inline::Python, where it also works fantastically.

I love the idea of one community NOT completely re-writing a library in their language when it can just be used "directly" instead.

Add Comment

2013.12.26 Overtone Adventures 4

Today I'm playing with Open Sound Control (OSC), which is more general-purpose than midi but kinda similar idea. I have an android app on my phone named Control which has a few existing UIs, in my case I'm playing with the multi-touch. I set it to have two touch inputs.

In my Overtone REPL, I first set it up to listen for OSC events, and dump out whatever events it sees:

(def server (osc-server 44100 "osc-clj"))
(osc-listen server (fn [msg] (println msg)) :debug)
; Turn off with: (osc-rm-listener server :debug)

I got both my laptop and phone on the wifi (which allows peer-to-peer communication) and told the Control app to connect to my laptop server ( port 44100, in this case). Now when I touch the screen I get a bunch of messages about the generated OSC events. They look like:

{:src-port 45161, :src-host, :path /multi/1, :type-tag ff, :args (0.36825398 0.3961456)}

From this I see that the path I want to react to is "/multi/1" and "/multi/2". Dragging my fingers around, the args are the x and y coordinates normalized to (0..1). For now I'll just make it beep a bit when it gets an event. I'm hooking the first touch up so the X axis generates a frequency, and the second touch up so that the Y axis generates a frequency (just as a test), giving:

(osc-handle server "/multi/1" (fn [msg]
  (let [
      x (nth (:args msg) 0)
      y (nth (:args msg) 1)]
    (demo 0.1 (saw (+ 100 (* 100 x))))

(osc-handle server "/multi/2" (fn [msg]
  (let [
      x (nth (:args msg) 0)
      y (nth (:args msg) 1)]
    (demo 0.1 (saw (+ 100 (* 100 y))))

Each touch plays for 1/10th of a second, so if I drag around these overlap a bunch. In fact, if I drag around a lot then something gets backed up and the sound is delayed.

Add Comment

2013.12.24 Overtone Adventures 3

Good progress today -- I made a hack to get around that BUNCHES of midi issue. Needed that because when I would hit a single note on my jack-keyboard (virtual midi keyboard), I got a whole bunch of events. I'm guessing I was getting 64 of them.

I also got Overtone Vim Integration working. Well technically I already had it working (had fireplace.vim installed, etc), I just didn't know it. Now I can start up the REPL in one terminal and start up vim in another. In vim do ":Require" and it finds the running REPL and hooks into it. I also added "nnoremap <F1> :Eval (stop)<cr>" to my .vimrc, so that I can jam on F1 to make the noise stop.

Here is the file I've got now:

(ns noise.core)
(use '
(use 'overtone.inst.piano)

(on-event [:midi :note-on]
  (fn [e]
    (let [note (:note e)
          vel  (:velocity e)
          device-name (:name (:device e))]
      (if (= "VirMIDI [hw:1,0,1]" device-name)
        (piano note vel)

So that takes midi events specifically from that device (I picked one of my 64 virtual devices at random, seems to work) piped to the piano instrument. Next I'm going to make a new instrument of my own and hook that in!

Add Comment


META INFORMATION: This is the technical blog and wiki of Brock Wilcox (awwaiid). Entries focus on my current projects, interests, and sometimes life events. If you'd like you can check out the list of All Entries or the RSS Feed. I also have a LiveJournal syndication feed for LJ friends.


Blog Blog RSS Feed








Follow @awwaiid

Wiki Edits Wiki RSS Feed




... more changes