Sometimes the art is the journey - Data Rabbit to Flowmaps

Ideas are a funny thing. We ponder, plan, visualize, iterate, and then (hopefully) execute. But often in creative works what you end up with is not what you set out to create - for better or worse.

You might open new doors - but also break windows, fall through holes in the floor, discover gold nuggets behind the drywall, etc.

Seriously though. Your understanding of the problem changes but so does reality and you roll with the punches as best you can. Technical limitations, unexpected issues, tangents, personal tastes change, you skill level goes up, ambition boosts/wavers, etc.

Tech is interesting because I view it as creative work as much (if not more) as it is technical work - and your output may be defined by 5,000 micro-decisions you made along the way - only a fraction of which were pre-meditated / pre-work decisions.

Decisions you might not even be able to verbalize & quantify after the fact - earned intuition, the deepest kind.

Less Basquiat-y types will likely take issue with this, but sometimes you take things in a certain direction because it "felt right" for the problem at hand in that moment. Time will tell if it was the right decision, and the more things that get stacked on it (more "concretions") - the more you might be stuck with it.

But the perspective you gain and what you learn is 10x more impactful than learning to "make better decisions" as smug detractors might say. "Bro, just do it right the first time!". As if.

Even a misguided decision can make ripples in the water that illuminate the edges of the pond.

Remember being a kid - adults & teachers giving you all kinds of advice about your life. Unsolicited. Things you don't even really have proper context for. Imagine just taking it all at face value and being like "Ok, don't do X, got it! Thanks, Mr. Peanut".

What kind of hollow "lesson" is that? A blind appeal to authority? Barf.

Granted, some of their advice might turn out to be sound - but only in retrospect - the real lessons we learn are from things we try (fail or not). Blindly following authoritative takes leads to a hollow understanding, if you could call it understanding at all.

Advice is a sentence or two, trial and error writes entire books.

But I digress.

Riffing on an idea

I was recently asked to give a Data Rabbit talk at Amperity's Seajure Clojure meet-up (awesome folks, btw) - they mentioned it would be cool to talk about Data Rabbit & then some of the LLM stuff I've been experimenting with (Rabbit Remix).

Problem is, while sharing a subset set of ideas, those are 2 different products - and what I really wanted to talk about was again a 3rd product (Flowmaps) - based on similar principals, but a different aim.

So I came up with a talk outline "Riffing on an idea: Data Rabbit to Flowmaps", how these things were each stepping stones to the next - intentional or not. Forcing me to think about all these things in context of time and lineage of ideas - what sifts through and what remains.

Data Rabbit

• The "big idea", ambitious, tries lots of stuff, some work well / some less so
• Inspires & provides ground-work for the others
• A "kitchen sink" for experimentation (and UI ideas)
• (Over)Complicated execution pipeline
• *Continues development as a REPL as well as a learning platform

(above from - "Using Data Rabbit as a 'data rich' Clojure REPL")

Rabbit Remix

• The application of "Data Rabbit ideas" to a text/string flow world with LLMs as "operators" to enable agents & complex reactive workflows
• A niche application of ideas... on to another niche
• A basic execution pipeline, less capable, makes lots of assumptions
• *Continues development as an OpenAI / ElevenLabs sandbox - future unknown

Remix was a break from Rabbit development burn-out (almost 4 years on and off, 5 prototypes deep) - and I was curious to look into the AI space and see what all the noise was about.

But it was also an attempt to (consciously or unconsciously) make a "sharper focused Rabbit" with "way less stuff" in it. At that point I was fully aware of what I knew, in my own space. A new downstream idea. From the minimalist look of the blocks the the paired down interaction models, I feel this is fairly evident.

Turns out I was looking the wrong direction.

Remix works well under certain scenarios - and is capable of some very cool things IMHO, but it still suffers from one core problem that keeps Data Rabbit from being not 100% a thing you'd want to use in production.

The browser.

Both Rabbits force Chrome to do a LOT of heavy lifting, all the expensive execution and orchestration of the flows happens in the browser. Now, this makes some sense since they are first and foremost "front end" tools - but it also means that none of these flows you create can really ever be run legit headlessly - from a cron job - or god forbid inside ANOTHER application. No way, man.

“Reasoning from first principles”

I got to thinking, knowing what I know now - how would I have designed it differently - and what would that look like? I drew up a new set of reqs - this time starting at the bottom of the stack instead of the top-down.

• need a solid backend core that can execute our flows w/o a front-end, or embedded in other codebases
• with enough hooks and instrumentation that a richly featured front-end can view/drive/orchestrate them
• a simple DSL for the flows that is ALSO legit Clojure code and not some horrible macro-read salad shooter
• don't re-invent the wheel with data passing, be smart by being dumb

Which leads to Flowmaps

Flowmaps is essentially a backend-focused Data Rabbit (at least a portion of it), but with an optional UI for viewing and debugging your flows. Your flow is just a Clojure map being passed to a function - with an optional return atom or channel.

This gives me all the new reqs and more - time-travel, production usage, embedded usage, small footprint, etc - and better yet, it's just a scaffolding for new things to sit atop - things that won't be constrained.

I won't wax poetic on it here, I made a whole post on it, or check out the GitHub docs...

Seems that the "Rabbit suite" is ever expanding. ;)

Point is, no effort is ever truly wasted - as long as your destination remains consistent. Try things, break shit, do the wrong thing - see were it takes you - the future you will know much more as a consequence.