In case you didn’t see the news, I recently launched Gradients & Grit, a new publication at the intersection of software engineering, AI, and UX. In our first issue, we introduced PromptCanvas, an interactive spatial prompt experience. As the name says, building PromptCanvas involved building a canvas interface similar to what users have come to know from products like Figma, Miro, and other design and collaboration tools in the space.
In this post, I want to outline the design and engineering decisions made in building the application.
Build or buy npm install
Making the canvas look and feel intuitive to use requires putting in a lot of thought. From features like zooming and panning, to adding interactive multimedia content, a dynamic surface is much harder to build than a fixed-layout page. Because of this, we were naturally wary of building our own rendering engine and searched the ecosystem for an alternative customizable enough to meet our requirements.
We knew that we wanted to build a sidebar-heavy workflow of interacting with LLM prompts, and we wanted to use intuitive canvas features like arrows and sticky notes to provide a powerful palette of tools. Initially, we didn’t want to bring over other features like drawing or even rich customization of shapes and text, so we looked for the most barebones canvas library that would allow us to build our business logic on top.
Tldraw started as an open-source whiteboard alternative and gradually evolved into an open-source company building the tools to build canvas applications. I remember checking out their repository a while ago and much has changed in the meantime as they’re preparing their v2.0 launch.
Trying out the product demo, I decided to dig deeper and see how it easy it would be to take the editing and canvas core and bolt our prompt concepts on top.
Choosing a suitable fork depth
At first, I tried to remove everything opinionated Tldraw had to offer like the built-in tools and shapes. This proved to be harder than I imagined at first because I quickly uncovered more implicit dependencies of seemingly modular parts. I believe the team is working hard on adding ever more customization options but they naturally had to start with a fully-featured implementation to understand the design challenges. Most users probably expected shapes to be included as well, which makes sense if you’re simply trying to extend the canvas or integrate it into other environments like Jupyter Notebooks.
After some days of trying to force the Tldraw library to serve a purpose it wasn’t designed for, I changed the tactic and attempted to add a simple custom shape and remove the built-in UI elements.
Add changes on top, keep what works
We needed features like selecting shapes, adding sticky notes, and undoing/redoing changes anyway, so keeping the editing experience and standard shapes made sense and saved us hours of work.
Adding the prompt concept as a custom shape and storing all details like input, past messages, and error and loading states immediately allowed features like duplicating the shape and undoing deletions while preserving all „custom“ state Tldraw didn’t know about. The only downside to using Tldraw‘s store as source of truth was that we had to continuously sync changes to prompt inputs and outputs back to the canvas state, which then automatically updated the shape „preview“ on the canvas whenever sidebar content changed.
In our final design, we used many parts we didn’t initially think we needed, and keeping the Tldraw concept worked out great for our purpose. Even with this simple experience, it took roughly half a week to understand the system design including the hooks to respond to updated shapes and other events. If I ever need to build another canvas application, I’d choose Tldraw again, especially if it keeps removing use case assumptions and offers different layers of abstraction and features to serve as a plain canvas that can be modified to work for all kinds of experiences.