[{"data":1,"prerenderedAt":163},["ShallowReactive",2],{"\u002Ftlt-2024.12.12-little-boxes-for-your-ghost":3},{"id":4,"title":5,"body":6,"createdAt":149,"description":150,"draft":151,"extension":152,"image":41,"mastodonThread":153,"meta":154,"navigation":155,"path":156,"seo":157,"stem":158,"tags":159,"updatedAt":149,"__hash__":162},"content\u002FTLT - 2024.12.12 - Little Boxes For Your Ghost.md","Little Boxes For Your Ghost",{"type":7,"value":8,"toc":144},"minimark",[9,19,24,35,42,50,61,65,87,109,120,123,129,132,135,141],[10,11,12,13,18],"p",{},"Drawing Xes in Boxes is harder than you might think! Teaching ",[14,15,17],"a",{"href":16},"\u002Fghostwriter","Ghostwriter"," a bit about space by pre-segmenting the input image gets us closer.",[20,21,23],"h2",{"id":22},"the-problem","The Problem",[10,25,26,27,31,32,34],{},"You may recall from ",[14,28,30],{"href":29},"\u002Ftlt-2024.11.23-a-ghostly-game","TLT - 2024.11.23 - A Ghostly Game"," that ",[14,33,17],{"href":16}," is ... not good with spatial awareness. For example, we can ask it to draw an X in a box and see how close it gets...",[10,36,37],{},[38,39],"img",{"alt":40,"src":41},"Ghostwriter X not in box","img\u002FGhostwriter_X_not_in_box.png",[10,43,44,45,49],{},"Not great. That is even ",[46,47,48],"em",{},"after"," we tried the \"plz think harder\" approach, asking the LLM to write out where it thinks things are with bounding boxes and planning where things should go and all that. It still can't hit the box that it can see.",[10,51,52,53,56,57,60],{},"It can ",[46,54,55],{},"read handwriting",", it ",[46,58,59],{},"knows there is a box more or less in the middle",". So close!",[20,62,64],{"id":63},"step-outside-the-box","Step Outside The Box",[66,67,68],"blockquote",{},[10,69,70,74,75,78,81,82,84],{},[71,72,73],"strong",{},"Vivian Jaffe:"," Have you ever transcended space and time?",[76,77],"br",{},[71,79,80],{},"Albert Markovski:"," No. Yes. Uh, time not space. No. I have no idea what you're talking about.",[76,83],{},[46,85,86],{},"(I Heart Huckabees, 2004)",[10,88,89,90,96,97,102,103,108],{},"So with a little help from my friend ",[14,91,95],{"href":92,"rel":93},"https:\u002F\u002Fsourcegraph.com\u002Fcody",[94],"nofollow","Cody"," I whipped up a relatively dumb segmenter. It uses ",[14,98,101],{"href":99,"rel":100},"https:\u002F\u002Fdocs.rs\u002Fimage\u002Flatest\u002Fimage\u002F",[94],"image"," and ",[14,104,107],{"href":105,"rel":106},"https:\u002F\u002Fdocs.rs\u002Fimageproc\u002Flatest\u002Fimageproc\u002F",[94],"imageproc"," to find contours, snaps boxes around them, and then looks for the big ones. So in the case of our box, here is a bit of extra that we put into the prompt:",[110,111,116],"pre",{"className":112,"code":114,"language":115},[113],"language-text","Here are interesting regions based on an automatic segmentation algorithm. Use\nthem to help identify the exact location of interesting features.\n\nRegion: x=400, y=572, width=640, height=613\nRegion: x=31, y=32, width=41, height=41\nRegion: x=811, y=96, width=133, height=127\nRegion: x=279, y=162, width=83, height=29\nRegion: x=178, y=128, width=69, height=86\nRegion: x=1014, y=138, width=43, height=47\nRegion: x=36, y=37, width=31, height=31\nRegion: x=979, y=96, width=42, height=85\nRegion: x=549, y=118, width=56, height=76\nRegion: x=439, y=149, width=53, height=56\n","text",[117,118,114],"code",{"__ignoreMap":119},"",[10,121,122],{},"And now it gets much closer!",[10,124,125],{},[38,126],{"alt":127,"src":128},"Ghostwriter X more in box","img\u002FGhostwriter_X_more_in_box.png",[10,130,131],{},"This is a pretty coarse-grained way to give a sense of space, but it is working much better with this segmentation hint. There are a few other things that might work as well -- we could give it an image of what each of the segments has inside of it, we could merge segments with some center-of-gravity sort of thing. Etc.",[10,133,134],{},"Let's try more boxes!",[10,136,137],{},[38,138],{"alt":139,"src":140},"Ghostwriter x in boxes not great","img\u002FGhostwriter_x_in_boxes_not_great.png",[10,142,143],{},"Whelp... still more work to do! :)",{"title":119,"searchDepth":145,"depth":145,"links":146},2,[147,148],{"id":22,"depth":145,"text":23},{"id":63,"depth":145,"text":64},"2024-12-12","Drawing Xes in Boxes is harder than you might think! Teaching Ghostwriter a bit about space by pre-segmenting the input image gets us closer.",false,"md",null,{},true,"\u002Ftlt-2024.12.12-little-boxes-for-your-ghost",{"title":5,"description":150},"TLT - 2024.12.12 - Little Boxes For Your Ghost",[160,161],"blog","ghostwriter","Tj1bLYbdvyOlxJ9NnaKymvJO7kJeJcv8F2lTSinNKGs",1778988314981]