With Ghostwriter I inject pen input on the device to show the output of LLM (Claude, ChatGPT, Gemini, etc) on-device. But the result is a bit messy. So here I give Claude Code some direction and access to the device to iterate experimentally!

Here is my initial prompt:

I would like to perfect drawing on the reMarkable using the pen, which can then be used to more quickly draw SVG or images to the screen using the injected pen input. Toward this end, I want you to set up an experiment loop. I have the remarkable available via ssh rmpp where you can run remote commands, including test commands that we scp over. We could also set up a remote-control that can act as a tool, so like a cli debug version of ghostwriter that runs one command and exits or accepts commands over http or something like that. In any case, you can run code on the thing and treat it like a remote-computer-use situation -- take screenshots, inject input, take a screenshot to verify the result. You can send pen and touch input. You can get a new page by swiping from right to left and then tapping the new-page icon that comes up on the right side, for example. You can tap on the tools menu, the tiny circle in the upper-left, and use that to see tools. You can tap with two-fingers at once to undo.

So what I want is for you to bootstrap up some experiments to verify your capabilities, ultimately leading to an improvement in the drawing tools for ghostwriter -- especially the ability to accurately render SVG and png content using the injected pen input.

For example, you could iteratively (by a series of screenshots and pen/touch injection) verify that you can accurately:

• Draw a horizontal line
• Draw a vertical line
• Draw a diagonal line
• Draw a dot at any x/y coordinate
• Draw a triangle
• Draw a single dot
• Draw a png as a bunch of dots
• Trace an SVG

Between each you could send a bunch of undo-touches or try to go to a new page so that you can see what is happening. You can also experiment with being able to quickly change between tools. There are two pens, so one idea is that pen-1 is for the human and pen-2 is for ghostwriter.

From these series of built up experiments we should be able to document how to interact accurately and reliably with the screen to render content via pen input.

That got it going really well. I had to give it a bunch of permissions to get the loop going, but soon it was trying something, taking a screenshot, deciding on the result, and then trying something else. I also had to steer it a bit to be more precise... it kept concluding "give or take 5px is fiiinneeee".

It built an "experiment" binary that takes cli commands to draw. I had to coach it a bit on clearing the screen or getting a new screen, and how to avoid (or on purpose utilize) the tool menu. Now step by step it is getting better and better at pen drawing. We are working our way up to drawing smooth polylines from svgs and even outline fonts.

The general technique of giving Claude Code a way to observe results and an overall goal (even a vague one like "be better" or "increase accuracy") works really well. Now it is chomp chomp chomping on tokens and doing small experiments to find bugs and alternate techniques for operating the reMarkable.