Flowmaps - Mapping Clojure core.async Flows Through Time & Space

"The real voyage of discovery consists not in seeking new landscapes, but in having new eyes."

~ Marcel Proust, In Search of Lost Time

Did you ever want to view an entire business logic flow as easy as you can view the logic of a threading macro?

(-> my-data do-thing1 do-thing2 send-emails done!)

What about branching paths, loops, conditions, viewers, time-travel, step
injection, speech notifications, REST endpoints? Oh, and let's visualize it as a flowchart so the team can better reason about it together? Hmmm.

Earlier this month I dropped an early build of my flowmaps library (0.31-SNAP as of this writing). It's part of a larger long-term effort to reverse-design Data Rabbit from the backend up as a set of libraries & modules, based on all I've learned on this journey thus far.

Flowmaps is a simple Clojure map-based DSL library for creating "flow-based-programs" (think 'function chains') - with an advanced optional debugger UI. This is based on an interpretation of the classic FBP model popularized in the 1970s by J. Paul Morrison.

FBP (Flow Based Programming) is a programming approach that defines applications as networks of "black box" processes, which exchange data packets over predefined connections. These connections transmit this data via predefined input and output ports. FBP is inherently concurrent and promotes high levels of component reusability and separation of concerns.

This is a perfect match with a functional, homoiconic language like Clojure.

FBP was intentionally modeled in a way to emphasize visualizing program flows as flowcharts - which also makes it a great match for my Rabbit-esque canvas based approach.

The term observability gets thrown around a lot, but I've come to see this kind of system as a step further, entering the realm of "interactive introspection".

Not only can we monitor & understand the behavior from the outside (cough, observe it), but also dynamically interact with & influence it in near real-time.

A more verbose set of examples & explanation can be found on the GitHub page, but here I wanted to lay out the higher points.

Being smart by being dumb

No need to reinvent the wheel. No need to create some novel data passing / execution pipeline. No crazy pseudo language via macro cleverness.

• Blocks (FBP's 'processes') are normal Clojure functions
• Connections are core.async channels (created, destroyed as needed)
• Ports are the pre-defined arities that marry connections & blocks
• Data... is just data. Inputs & outputs (FBP's 'information packets')

Backend First

I love my fancy UI tools. Drag and drop, scrubbable, canvas based, Bret Victor style feedback loops? Inject that shit straight into my veins!

But none of that matters if you can't actually go to production with it. For general flows (exceptions for things like BI tools & deep niches) the fanciness needs to be available for development and introspection - and then completely disappear if need be. Otherwise, it's destined to be a novelty or a leaning tool only.

Needs to be a headless, back-end, Open Source library - but with a UI twist.

Consider this...

We have some steps, a "seed" static value (10), and a forking step (2a) that starts it's own path. Each "block function" consumes it's input and sends it's output down the line like a conveyor belt. It doesn't know or care about anything else.

Let's call this map flow1 and start it up.
(flowmaps.core/flow flow1 {:flow-id "live-scratch-flow" my-return-channel).
This can be run in the REPL or inside your application and return it's end result to another channel or an atom. Encapsulated logic.

Perhaps we want to double check the output of our functions.
(flowmaps.core/flow-results). Here flow-results returns a basic map of all flows, blocks & their ultimate values.

{"live-scratch-flow" {[:step1 :step2 :step3 :step4] ([:step1 10] [:step2 20] [:step3 30] [:step4 "30!"])}}

Makes sense. Lineage and the last values. In this case "30!" gets shipped off to my-return-channel to sail the seas of the rest of your application. Bon voyage.

Ok, but we want to see more. Our eyeballs yearn for it. 👀

Adding "dimensionality" to our understanding

What if we were still developing this flow, it was much more complex, and needed to see its behavior at a deeper level? Well, before running it, we'd start up the Rabbit debugger  / visualizer (flowmaps.web/start!).

Now when running we also produce this.

What we have here is the visual representation of the steps we defined, as well as the values that were created and streamed through them.

We now are adding various dimensions on to the flow that can provide our senses with more information.

Space

Laying these execution steps out on a canvas - adds a spatial dimension to an otherwise abstract sequence of events. This gives it a tangible and material feel to a model that we would otherwise have to unpack & animate in our heads.

Time

We also have 2 Gantt charts on the bottom. The top being the channels themselves receiving data, and the bottom one being the blocks (functions) that processed those values.

Context

Toggling UI can enumerate the passed data values as well as channels & functions involved. Hovering over anything will reveal details about itself as well as related items (producers, consumers, etc).

Rewind

Therefore the next logical step would be to map time to space - seeing "as was" values as the flow was executed. Controlling time.

Views

Sometimes debugging involves a lot of println statements, what if we had something slightly more powerful? Each block has a optional :view function that gets run on the output of that block. Here we can use simple text... or hiccup, re-com, or vega / vega-lite to render in our flow view.

Which makes things... interesting. Again, this only applies to debugging web-view, in production it would be ignored.

Error? Did one of your blocks barf? A simple "error view" will be automatically created. Rewind time to see exactly what the input values were at the time.

Sub-flows

What if we have another flow map definition - that is going to act as a single step in my parent flow? A flow as a block, so to speak.

Yes.

Zooming out

Let's say we have lots of flows in process, ones running ad-hoc, some running on a schedule, some still "live", & some that have already closed their channels.

Interactivity

In development it's important to test your flows and how they will react to changing and dynamic data. In production you want options and flexibility.

Let's roll

Flowmaps takes a "Marble Madness approach" to interactive flow values. Take a new value and "drop" it into any channel and see how the flow reacts downstream.

This can be done several ways:

• Via the debugger UI directly
• Literally doing a Clojure async/put! to a channel
• Hitting that channel's REST endpoint with a new value (and "return scooping" with an optional :await channel)
• Overriding arbitrary block values at runtime (pre-execution)

Notice how this created a new "time segment" on our Gantt timeline? One that can be switched back to and scrubbed on it's own without being bothered by the original timeline from which it inherits all it's starting values from.

(Just for fun)

Listen

Perhaps we want our flow to say something when it hits a certain step. This could be for accessibility reasons - or perhaps to notify you of and ETL job finishing from across the room. Exactly like :view works - write a :speak fn that generates a string & use your ElevenLabs voice API key in the UI settings.

A more detailed step-by-step guide can be found on the GitHub page.

As obtuse as it sounds... not just writing code; we're painting a picture, telling a story of how data moves and interacts - but ONLY if you want it - that's the key.

A fancy front-end with no backend legs is limited - just as much as an unobservable backend can be. Trying to cross the barrier here.

Bringing clarity to complexity, making the invisible visible - that's my goal.