Those practices shaped how I approach software engineering. They shaped how I approach management. I still default to TDD with small commits 20 years later if I get a choice. But most interesting to me is that all those XP practices have made me a successful user of AI in software development as well.

Always Be Pairing

One of my dev jobs was on a team that was 100% pair programming. We started the day with a standup where we split into pairs, picked a story card from the board, and then spent our day working at one computer with two keyboards and two mice. At first, it was hard. It felt awkward and slow. I found being shoulder to shoulder with another person all day exhausting. But with practice, I learned. I learned to communicate with precision, not phrases like “you know” and “stuff”. I learned to be a useful navigator, pointing out our errors and planning our next step when I wasn’t the one typing. I learned to enjoy pairing as a default practice, rather than as a tool to use only when you have a tricky bug.

When I started using AI for coding, I quickly fell back into my old pairing patterns. I’ve customized my AI tools to use pair-programming principles. We discuss changes bit by bit before writing any code. I discuss design decisions with the AI. I ask it for feedback and ask it to point out errors not only in my code but also in my design. I find that working with AI as a pair-programming partner, rather than fully delegating tasks to it, improves the quality of the results. It also ensures that I understand the code’s design and the decisions that went into choosing that design.

Small Stories, Small PRs

A habit my early mentors taught me was to work in very small chunks. The first time I submitted a 1,500-line pull request that I’d spent 2 weeks on, I was politely but forcefully instructed never to do that again. I was taught to break down tasks into chunks that could be completed and merged in 1-2 days. If I objected and said something couldn’t be broken down into pieces that small, I was told, less politely, that I was incorrect. And when I combined feature work and a refactor into a single PR, I was told that no one would code-review it until I separated them. All this feedback turned me into someone who is uncomfortable keeping a branch alive for more than a few hours.

When I started writing code with AI, it was natural to work in small stories and small PRs. I break features into pieces I can finish in a few hours to a day. Each task is a new conversation and a new branch. I have some customizations in my setup that direct the AI to work only on the current feature. If it identifies additional work along the way, I’ve told it to put those ideas into a to-do list. Once the feature we’re focused on is working, I submit the PR. I also sometimes start a new conversation to critique the code and identify issues, such as incorrect comments, before I do a human review.

Test Driven Development

One of the most important things I took from my time in the Ruby community was the culture of test-driven development and testing as a critical component of software development. I was taught a particularly strict form of TDD that only tested behavior, forgoing mocks and assertions on what methods were called. I still do a tiny involuntary shudder when I see tests that assert new was called.

When I work with an AI, I find I’m most successful when I use a modified TDD approach. The biggest difference is that instead of starting each step with a unit test describing the end state, I start each step with a prompt describing the end state. The prompts are usually closer to functional tests than unit tests, but they aren’t at the level of an entire spec. And just like the tests I write, my prompts don’t describe implementation details; instead, they describe the user-facing behavior.

Self Documenting Code

One of the things I found hardest to adopt in XP was the approach of self documenting code. When my mentors said self-documenting code,, they didn’t mean header blocks on methods picked up by a documentation generation tool. They meant code that was written so clearly that those blocks were largely unnecessary. It took me years to habitually use plural variable names for collections and the singular version of that name for individuals in that collection. And I remember some debates during code review about the order of clauses in conditionals. Do you start with the positive condition or the more common condition? Are postfix conditionals too confusing?

Now that I’m writing with AI, I’m grateful I developed early habits of constantly reviewing code with an eye for readability. Readable code is easier for the AI to understand, and it’s easier for me to review the diffs for the AI-generated code. I have made one adjustment to better document my code for AI. I have a generated file in each directory labeled something like context.md that my AI assistant can use for context.

Conclusion

If you were an XPer back in the early 2000s, what practices have you found most helpful as we adopt AI as part of our coding toolbox?

The simple truth is that I don’t know what the future of DevRel is. I have some ideas, but we’re all on an AI journey together, and sometimes it seems like the tech changes every day. In that world I’m a bit suspicious of anyone who claims to know the future, unless they have a time machine. But I am comfortable talking about trends that I’m seeing in how I personally work and what I’m hearing from the folks I talk to. But upfront caveat, I can guarantee my sample is biased. I’d love to know where others have experiences that differ.

The Tale of the Reluctant Gopher

One thing I’m noticing in my own work is that AI-assisted development makes it much easier to try new things and experiment with new tools, languages, and frameworks. I’m seeing this in my own workflows. I put off learning Go (Golang) for years. While I value all the strengths of the language, I didn’t have any problems where I absolutely, 100% needed the power of Go, so I hadn’t been able to justify to myself dedicating the time to learn the language and become a Gopher.

Now, writing Go is as easy as writing any other language, so I’m preferentially picking Go for new projects. Why not? I can get high performance, native concurrency, and high readability with no additional friction for me. Additionally, because I find Go very readable I’m still comfortable doing code reviews on the code I generate for side projects. Are my code reviews as detailed as someone who knows the language well? Probably not. But also, the code that’s being generated isn’t a mystery to me.

Goals Not Syntax

Another thing I’m noticing in myself and in a lot of demos I’m seeing from my colleagues, is that the way we’re working with AI is changing. When I started using AI for development assistance I was asking questions about the argument order of specific methods in LangChain and what values an enum could have in a library I was trying to integrate into a demo.

This week I asked Gemini how to fix the images on social media links to my blog. I didn’t ask about X/Twitter cards or Open Graph or SEO. I just asked how to make the links to my blog look better. It was able to identify the issue and fix it. A lot of folks refer to this change in prompting style as “Spec-Driven Development”, but I’m still not sure how I feel about that term. I did end up having to nudge Gemini to use the Jekyll SEO Tag plugin rather than just adding the necessary fields itself. But that’s my design preference to use a maintained library rather than building integrations from scratch. The generated code worked in both cases.

No Really, Solve My Problem

One more trend I’m seeing is that folks want learning personalized to them. Either they want explanations that relate to their existing knowledge and strengths, or they want knowledge presented in the context of their existing project. So instead of getting a generic explanation of how concurrency works in Go, a web dev may want that explanation given in contrast to JavaScript. Or someone may want an LLM’s answer to take into account the libraries they’ve already imported into a project, rather than pulling in a completely new library.

I regularly ask Gemini to explain things to me, or more accurately I’ll say “tell me more…”. And at this point I pretty much expect that its explanation will take any additional context into account. Usually that’s my codebase, but sometimes that’s additional information I’ve shared about me and my previous knowledge. When it was teaching me Agent Development Kit (ADK) I asked it to use Go and told it I knew JavaScript. It explained critical Go concepts to me based on how they compared, or didn’t, to similar things in JavaScript. For me at least, it seems natural that answers would be customized to me and the things I’m currently working on.

What Does This Mean for DevRel

So what does this mean for DevRel? While I do think some things will change pretty significantly, I think a lot of what we already do is still relevant. Since it is easier for folks to try new languages and frameworks, we need to ensure the communities we belong to are welcoming to newcomers, especially newcomers who may not have followed a traditional path to this particular technology. I think my team is probably tired of me saying it, but everyone is welcome at my developer (and builder) party.

I also think we need to focus a larger percentage of our content around higher level tasks. Some DevRel teams do this really well already. I’ve always appreciated that the Firebase docs have great walk-throughs for common multi-product tasks. If folks are asking higher level questions we need to make sure that the tutorials and documentation exist to help achieve those higher level goals, especially when the solution is multi-product.

Finally, every day I feel more and more urgency to have great evaluation tools. If folks are expecting personalized answers, I want confidence that those answers are correct. I was a huge proponent of Test-Driven Development (Red, Green, Refactor for life). I liked TDD because it gave me confidence my code was right, or at least right within the context of the tests that I wrote. Evaluation seems to be the unit and integration testing of the AI age. And I want really great evaluation tools so I can be confident that personalized answers are correct.

Conclusion

I’m curious, what trends you are seeing in your communities? Are folks embracing personalized learning or do they prefer traditional documentation and tutorials? Are your colleagues trying new tools more readily now that the time needed to experiment is lower, or are folks sticking with what they know well? Please let me know in the comments.

It wasn’t that I disliked learning or got bad grades. I just never understood the way most schools work. Why should everyone who is the same age have to learn the exact same material at roughly the same rate? We were taught to embrace our strengths and weaknesses, and yet our school treated us like we were all the same when it came to learning.

To arrogant, teenage me, it was obvious that the solution was self-paced learning of some kind, likely driven by computers since most of my classes had over 30 students per teacher. So I tried to code a self-paced learning system. Sadly my programming skills weren’t up to the task back in the mid-90s. I did build a couple of very simple flashcard-style apps that my friends used to improve their grades on tests, and seeing how my work could help others was what got me hooked on tech. And so I got a degree in Computer Science.

Announcing Ursa Weaver

A few weeks ago, at a workshop, I got thinking about the personalized learning problem again and wondered if I could create something to address it using AI. I fired up Google Antigravity, picked something I wanted to learn, Agent Development Kit (ADK) for Go, and got to work. I’ve released what I created in open source as Ursa Weaver (https://github.com/GoogleCloudPlatform/ursa-weaver).

Ursa Weaver was entirely vibe coded in Antigravity. It took me about 60 minutes to build an initial proof of concept and a few hours over the following weeks to refine the idea based on feedback and do some initial testing. The initial release has no traditional code, just an AI Agent Skill and some data files. And yet, it can provide personalized guidance to someone who wants to learn ADK for Go.

Design Considerations

Before I started actually writing the code, I worked on refining the requirements. In the end there were three things that were high priorities for the initial version. First, the tutor needed to run in the environment where people actually worked. I don’t understand why folks should learn skills in one set of tools and then have to transfer that knowledge to their primary IDE.

Second, I wanted the curriculum to be both personalized and curated by an expert. There are plenty of examples of AI explaining concepts; what is usually missing from these demonstrations is how bits of skill and knowledge build on each other. For example, AI can teach you addition, but it may not know you need to understand quantity and numbers before you learn addition. Or that you could learn subtraction before you learn addition.

Finally, it was important to me that there was some verification that students had actually learned the material. I know that often just reading material isn’t enoug. You need to actually do something with the information to retain it.

High Level Implementation

Ursa Weaver is implemented as an AI Agent Skill, allowing it to run on AI integrated IDEs. This means that the tutorial happens in the context of your existing configuration and preferences and in the context of a tool you already know. In context learning means you can focus on the material being presented and not on learning a new interface or tool. It also means that Ursa Weaver has access to any additional skills, CLIs, or MCP servers you have installed.

To allow the AI to personalize the learning journey I broke down the topic I wanted to learn, ADK for Go, into a series of skills that could be learned in 5-10 minutes. I used Gemini to help me do this, but I verified its work. Once I had the core list of skills I then worked with Gemini to add any pre-requisite skills, especially implicit or implied skills. Things like having Go installed or understanding how asynchronous versus synchronous function calls worked.

Once I had the entire list of possible skills, I used Gemini to organize them in a JSON file that listed each skill and the dependencies it had on previous skills. This creates a skill graph, but you can think of it as a course catalog explicitly listing that you can’t take Advanced Algorithms until you complete the Introduction to Algorithms course.

To tackle the verification requirement, I added explicit details about how to verify each skill was completed or learned. I gave Gemini a list of potential ways to do verification including short answer questions, multiple-choice questions, reviewing or executing code the user wrote, and verifying a configuration with MCP or command-line commands. This verification ensures that if a user completes the “Go Setup Skill” later skills can assume that the user has Go and the GOPATH is set correctly.

Finally, to support personalization I made two changes. First, I added support for fast tracking through skills if the user can complete the verification or an equivalent assessment without needing additional guidance or explanation. I also added a user profile that has basic information about the user including their background, interests, and what skills they’ve already completed. This allows the AI to customize the learning to a learner’s existing knowledge base and allows you to work through the curriculum across sessions.

Future Work

I have a long to-do list for the project. I’ve tested it in Antigravity and Gemini CLI but I need to test it with more tools. I want to add full support for multiple languages so students can be taught in their preferred language. I’ve only committed one curriculum file to the GitHub project, but I have others I’ve tested myself. I want to both provide some additional example curriculum files and the prompts (and code) to create your own from documentation and other sources.

I want to build evaluation tools for curriculum files that simulate common user personas and common scenarios including longer sessions which in my testing have shown the highest tendency to drift from the established curriculum. Once I’m confident in evaluation, I want to create larger curricula with potentially hundreds or thousands of skill nodes. I’ve created a curriculum with 100 nodes in my testing, but without good testing and evaluation it is hard to have confidence it will provide a good learner experience consistently.

Finally, most learning apps have some sort of gamification, or at least recognition of the work learners finish. I’m not sure what to do here but potentially I could use one of Google’s image models to create merit badges or certificates of accomplishment at key points in someone’s learning journey.

Share Your Feedback

If you get a chance to try Ursa Weaver, let me know what you think, both good and bad. It is still a proof of concept. For me, the biggest surprise was how fast it came together without any traditional coding. In just a few hours using Antigravity I was able to build something that teenage me could imagine but didn’t have the ability to build. I know I would have been a happier student if I could have wound my own way through the curriculum rather than having to work on chapter 3 just because it was the first Thursday in November.

Some of my resistance was that things people were showing me AI could do, I already had automated. When someone showed me that the AI could create the boilerplate for a class, I wanted to respond with, “Rails calls that scaffold, and it does the tests, too.” I wasn’t excited because the demos weren’t using AI for things I needed.

However, my primary objection to AI-assisted development was accuracy or, rather, the lack of accuracy. In one of my first exchanges with a generative AI, I quickly got it to produce an (incorrect) proof for P == NP. I continued to experiment with AI with my tester hat on and let’s just say my early introduction to AI-assisted development did not inspire confidence.

Despite my strong initial skepticism, I’ve found myself using AI coding assistance much of the time over the last nine months.

So what changed?

Partly, the models got better. But also, I changed. I discovered how these tools can fit into my personal development workflow and save me time. I also found that the current level of accuracy wasn’t as big of an issue as I initially thought.

So, how do I use these tools?

First off, I pretty much only interact with AI through in-editor chat. I have years of pair programming experience, so treating the AI as a less skillful pairing partner feels natural to me. If I want to write a method, I don’t start in the editor; I begin by prompting the AI in the chat. It feels similar to asking a pairing partner, “How should we write this method?” I give feedback on the code the AI returns, which is sometimes specific, and sometimes I ask questions. I iterate back and forth with the AI like this until I’m happy the code works and is well-designed. Only then do I copy it over to my editor and test it. My workflow may not be a typical assistance workflow, but it works for me.

Also, I mostly use AI for research. It turns out a lot of time I spend “coding” is actually spent looking things up in docs and on the internet. I think we’ve all reflexively opened a browser tab to look up a method signature or a cryptic error message while debugging at some point. Instead of using the browser, I now use the AI chat in the editor. It seems like a small thing, but saving the context switch keeps me focused. The AI gives me answers faster than I could find them myself, and when relevant, it can give me answers in the context of my codebase.

I also don’t use AI all the time. It is obvious in retrospect, but one of my “a-ha” moments with AI was realizing it isn’t all or nothing. I can use the AI where it helps and ignore it when it doesn’t. Personally, I don’t use it for languages and frameworks I know well. I’m faster without it and enjoy coding with those tools, so I let myself have fun and don’t use the AI. By contrast, I use AI extensively when working in languages where I’m less skilled and when working on tasks that I find less enjoyable. For example, despite years of trying, I’m barely a mediocre front-end JavaScript developer, so I use AI for JavaScript. I also use AI when I’m learning a new framework or tool. The AI doesn’t get frustrated with me when I ask it to explain various bits of code over and over and doesn’t find the bizarre analogies I used to understand the world annoying.

But what if it’s wrong?

So, that’s how I use AI assistance, but how have I managed my objection around accuracy?

I’ll be honest, I have some cognitive dissonance around the fact that current accuracy levels are good enough to be helpful to me. I’ve spent a lot of time thinking about how I could find AI useful, and also be frustrated that it feels like it is often confidently wrong.

I won’t claim to know precisely why both these things can feel true, but I have two ideas that have helped me accept the cognitive dissonance. First, humans often remember negative things more than positive ones and extreme experiences more than mundane ones. So there’s a good chance I forget all the times the AI gave correct but not remarkable answers, and I remember all the completely off-the-wall hallucinations even when those are significantly less common.

The second thing that’s helped me accept the cognitive dissonance is thinking about time spent versus time saved. This is easiest to understand with an example, so let’s do a thought experiment with some round numbers. Let’s assume that using AI takes me 30 seconds, including submitting the prompt and assessing whether the response was useful. Let’s also assume that the AI I’m using is especially bad and is only correct 10% of the time. Coding assistants are correct far more often than this, but 10% makes the math easy. Finally, let’s assume that the AI saves me 10 minutes when it is correct.

Submitting 20 prompts to the AI takes me 10 minutes (20 * 30 / 60). Since it is wrong 90% of the time, 18 responses will be unhelpful, and 2 will be useful. The two helpful responses save me 20 minutes. If we subtract the time it took to use the AI (10 minutes) from the time saved (20 minutes), we can see that using the AI saved us 10 minutes even though it was wrong most of the time.

As I said above, this is how I’ve been thinking about it to help me make sense of my own observations. It may not accurately reflect my reality or anyone else’s. But thinking about things this way has helped me understand why I find AI useful and also regularly get frustrated with the answers it gives. Both can be true.

How about you?

I’d love to hear how others use AI assistance for dev tasks since I suspect my preferred workflow isn’t typical. Have other folks run into the same cognitive dissonance I have where the AI is both helpful and also it feels like it is constantly incorrect? How have you reconciled those two things?

But what do you do if a friction log isn’t convincing your stakeholders or they need to urgently address developer pain points? That’s when I set up an empathy session.

Empathy sessions, or empathetic engineering, aren’t one of my ideas. I learned this technique from Ray Tsang, Kelsey Hightower, and Kim Bannerman. When executed well, it is one of the most effective ways I’ve found to influence a product roadmap. I’ve seen well-done empathy sessions cause launches to slip while aspects of the product were redesigned to improve the developer experience.

What is an Empathy Session?

An empathy session is a group friction-logging activity for the team that builds a product or feature. Traditionally, you gather all the engineers, product managers, and others in a room and ask them to complete a realistic user task with their product in small groups. While they work on the task, they track their progress and feedback in a friction log. After an hour or so, each group shares some notes from their experience and identifies what they think are the most urgent friction points to address.

Seems simple, right?

A good empathy session can be simple to run. And many companies do similar activities. However, to get the most impact from a session, there are many subtle details to optimize. Let’s dig into the ones I believe are most important.

Important things to consider

For a successful empathy session, the most important thing you can do is ensure the right people are in the room. You need to make sure that at least a few decision-makers are present. I also want all the folks who write the code and design the product to participate, but the group needs to be empowered to act on what they learned. And usually that requires a decision-maker or two in the room. I also like to secure a promise from the decision-makers that the team has the capacity to address the top friction found.

The second most important thing to a successful empathy session is task selection. Your chosen task should highlight specific developer friction you already know about and want to draw attention to. The task also needs to be something that attendees agree should be doable in the time available. If participants think the task is unreasonable, they won’t be as receptive to making product changes.

Third, you need to set the rules of engagement. I set a rule for the sessions I run: “No internal resources.” That means no internal docs, no internal accounts with allow-list access, no checking internal reporting when you get an error, and no messaging the person who built the feature when you get stuck. The goal is for the participants to get an accurate understanding of the experience of external developers. And external devs don’t have access to internal docs. For some sessions, I’ve even had teams use loaner laptops so they don’t have access to their editors and any product-specific customizations they’ve made.

Tips for running effective empathy sessions

I learned a few tips through trial and error that can help you run effective empathy sessions that folks enjoy and want to repeat.

First, an empathy session isn’t a bug bash. Bug bashes focus on a particular feature or product. Empathy sessions focus on a single user story that likely uses multiple features and products. Bug bashes are usually open-ended, with people exploring different parts of a feature. In an empathy session, everyone works through the same task. But mostly, bug bashes are focused on finding bugs and technical issues. At a good empathy session, you are just as likely to identify missing documentation, UX improvements, or entirely missing functionality.

Second, it is essential to let folks get frustrated. An empathy session aims to develop empathy for user pain points by experiencing them personally. Frustration is by design. But it is tough for most folks to watch their co-workers struggle and not jump in to unblock them. After a couple of less successful sessions, I now remind any facilitators and TAs to offer advice or hints but do not prevent attendees from feeling friction.

However, you also can’t let folks get too frustrated. Folks only stay engaged if they feel like they are making progress. The best way to control frustration is with the task you select. I know I’ve hit the right level of frustration if folks ask for ten more minutes when I call time. You also need to make sure folks get frustrated in the right places. Consider doing any necessary setup ahead of time if the setup isn’t part of the user story. That way, attendees can get right to the task and not waste time running through setup steps. And pick technologies your attendees are comfortable with. If the attendees only know Java, don’t choose a Python task because they will find friction with Python instead of with the product.

Finally, keep it short and fun. I think the sweet spot for sessions is between 2 and 3 hours. A 2-3 hour session usually gives 60-90 minutes for running through the user story. The rest of the time inevitably goes to intros, sharing, breaks, and logistical hiccups. It is also easier to keep the energy high with short sessions. To make the experience fun, you can also provide snacks, play upbeat background music, or even offer not-so-serious prizes like “most obscure bug.”

Variations for remote teams

I much prefer to do these activities in person. I’ve worked with several teams that will work an empathy and friction session into an offsite. But I know that’s not always possible. So, if you want to run one of these sessions virtually, you have a couple of options.

Option one is to do the activity all together on a call. A good format is to do the intros and share on a single call, then use the Google Meet breakout rooms feature to divide people into teams of 3-4 to work through the task. TAs and facilitators can pop into different breakout rooms to check on folks and help as needed. In my experience, this works, but engagement is lower than in-person sessions. It is easier for someone to get distracted by other things. It also is hard to have teams use loaner hardware in these sessions.

Another option is to have folks work through the task independently during a specific week. Doing a session asynchronously like this gives participants much more flexibility in scheduling the empathy friction log around their other commitments. It also allows you to choose a task that takes more than 60-90 minutes to complete. The downside is that your TAs have to be available during the entire week to assist with issues. And there are usually more issues since folks work independently instead of in a team and can’t ask a teammate for help.

Closing thoughts

While empathy sessions aren’t as easy as standard friction logs, they can change product roadmaps. I’ve found them especially useful for calling attention to setup/installation issues and integration issues between products. These tend to be things that product teams don’t look at often but are crucial to a solid developer experience. If you end up running an empathy session at your company, let me know how it goes, and share your tips for having an effective session.

One of the more common techniques for preventing hallucinations is RAG or Retrieval Augmented Generation. It sounds complex, and it can be. But the basic idea is extremely simple. It is also extremely powerful, and while there are common architectures for RAG solutions, the underlying technique can be used in many different situations and ways.

The Basics

The basics of retrieval augmented generation are simple. To help the LLM not hallucinate or to produce better answers, you can retrieve some relevant information and add it to the prompt along with your question or generation task. The most common example I see is for customer service tasks, where you retrieve relevant pages from a user manual and ask the LLM to base its response on that information. For example,

Help me restart my cable modem. Here are some pages from the manual for my particular cable modem. Base your answer on the information in this manual, and do not add anything extra.

{Insert text from the relevant pages of the user manual here}

How do you figure out what pages from the user manual are relevant? Usually, that’s done using a technique called vector search. Vector search deserves its own blog post, but to simplify it, we can say we turn each page of a user manual into a list of numbers. Then, when a request comes in like “Help me restart my cable modem” we can also convert that sentence to numbers and then find the pages with the most similar numbers. Or, as I said to a colleague once, “It’s all just linear algebra.”

This example is perfectly adequate, and many cool RAG systems have been built using this basic technique with documents and vector search, but that’s not the limit of what RAG can do. I’ve found that thinking about RAG in terms of documents and vector search can make it so folks don’t easily see other uses for retrieval in grounding responses from an LLM.

RAG without documents

Many RAG systems break a corpus of large documents into chunks and then retrieve some of those chunks for grounding. But what if your data isn’t large documents? Can you still use RAG for smaller pieces of text like employee bios, product descriptions, or book summaries?

It turns out you can. It sometimes requires minor changes to your similarity algorithm, but you can vectorize smaller pieces of text and then do a similarity search on those the same way you do on larger documents. And with that you can create systems that identify good mentors for new employees, help users find products that complement their previous purchases, and identify the perfect beach read for their next vacation.

You can also skip documents altogether and do the retrieval step from databases or web services. For example, perhaps you retrieve the week’s weather forecast to help an LLM answer, “Which is the best day this weekend for a picnic?” Or maybe you add recent order information to the prompt when a customer has questions about when their order will arrive. Depending on the exact architecture, some folks may call these non-vector sources “tools.” But if you find relevant information (retrieval) and add it to the prompt (augmentation) to improve the response (generation), it is another form of RAG.

More fun with RAG

Another use of RAG I’ve found is for multi-shot prompting. Multi-shot prompting is where you add some examples of what you want from the LLM to the prompt. For example:

Give me examples of things I can cook with chicken thighs. Here are some things I like:

Chicken tacos, Chicken in dill sauce, Chicken sandwiches

And here are some things I don’t like:

Chicken salad, Chicken wings

If you work on an app that logs prompts, responses, and potentially user ratings of those responses, you could do something similar.

The other use of RAG I like is when you have an LLM help you with the prompt before you do the retrieval. User prompts can often be incomplete sentences, contain misspellings, use synonyms, or otherwise be problematic for an LLM. You can ask an LLM to improve the user’s prompt before you look for related content in a vector database to increase the chances of finding relevant data. The demo Aaron Wanjala gave at Next 2024 used this technique. Instead of taking a text prompt from the user, it took a photo, asked an LLM to describe the image, and then used vector search to find products with similar descriptions in the product catalog.

RAG is simple and powerful

RAG is everywhere, and it can be intimidating. Many of the tutorials I’ve seen combine concepts with implementation and can be hard to follow. At the end of the day, RAG is a simple technique where you fetch relevant information to help the LLM craft a better response, add that to the prompt you send the LLM, and then let the LLM generate a response. That’s all it is. And that basic three-step process can power a huge variety of applications and agents.

Step 1 - Review and feel your feelings

Step 1 is to read the data. Most managers I know read the results as soon as they come in. My manager feedback reports are a mix of survey questions with results on a Likert scale and anonymized free-text responses. I look at the survey responses first, then the free responses about areas for improvement, and then the free responses about what’s going well. There isn’t any particular reason for that; it just is the pattern I follow.

During this process, I often have emotional responses to what I read. Sometimes, some of the feedback is funny and makes me laugh. Other feedback may be hard-to-read truths about my areas for improvement. I let myself feel any sadness or defensiveness that may come up as I read. Emotions are human. If necessary, I grab time with a trusted colleague to discuss the data and my reactions. My goal is to be through whatever feelings I have before I talk to the team about the feedback. I remind myself regularly that feedback is a gift and that clear feedback is kind feedback.

Step 2 - Make a plan

It can be very demoralizing to spend time writing genuine and thoughtful feedback and feel like it is ignored. I don’t want folks who report to me to feel that way. So, after I’ve read the feedback and worked through any emotions, I sit down and make a plan for how to address it.

I do my best to address all the suggestions for improvement and negative feedback. Even if I disagree with it and even if it was just one person. Things that multiple people brought up get more attention, but I address everything. For practical reasons, I don’t address each piece of feedback directly. Instead, I group the feedback into themes or categories. For example, one of my management weak points is building team identity, and so that has come up as a theme several times over the years.

Once I have my themes, I decide on one or two concrete actions I can take to address each theme. Concrete is important. I won’t put “be more empathetic” or “do more team building activities” as an action. Instead, an action would be something like “work with a mentor on empathetic leadership techniques with three meetings before Aug 1”, “speak no more than 50% of the time in 1:1s and team meetings”, or “plan online team social activities quarterly, starting in Q2.”

Where something is out of my control, like layoffs or budgets, the action is usually to either pass that feedback up the chain or to create an environment where my team can give their feedback to leadership directly by inviting an exec from our team to a team meeting.

Step 3 - Communicate

When I finish my plan, I share it with my team and my manager. While I communicate with these audiences differently, I strive to be as objective and forward-looking as possible with both. Defensiveness and disagreement don’t help here and, in my opinion, make a manager look weak.

If possible, I share the plan with my team in the next team meeting. I want to show, with my actions, that their feedback is my highest priority. I start by thanking them for their feedback. Then, I summarize the themes in the feedback and review my action plan. I do this with slides, so there’s something written down to hold me accountable. After I’ve covered my plan, I open the floor for questions and more feedback. I do not ask the team for specific things I should do in response to their feedback. Some managers do, but I prefer not to. In my opinion, it is my job to figure out how to be a better manager, not my team’s job to teach me how to be a better manager.

I also share my feedback summary and action plan with my manager. My intent in sharing is to communicate to my manager, “I’m on top of this.” My manager could read the feedback and assign me development tasks based on their observations. And if they think my plan is insufficient, I need them to do that. But I hope they see I’m handling this well and let me continue to do things my way. I do my best work, even management, with some autonomy.

Closing Thoughts

This approach has worked for me for several years with largely positive manager feedback and also with manager feedback that wasn’t entirely positive. I’m aware that it may not work with a different team or at a different company, and I’m curious how other managers approach similar situations.

Lesson #1: Every Field Matters

In the Agent Builder experience, you see text fields like name, description, and company name. When I was first experimenting with Agent Builder, I did what I always do when testing software and put nonsense data into these fields—things like “test123” or “my description.” While the tool accepts this, I was struggling with inconsistent responses from the agents I created.

Reading through the docs to debug the issues I was seeing, I realized that all those text fields are used by the LLM in the prompts that it constructs. The company name helps the LLM understand the context it is in. The name and description of tools and specific agents help agents understand how to use tools or interact with other agents. Those fields are just as important as the ones that look more technical or “codey.”

Once I started putting accurate and detailed information in those fields, my agent performed much more consistently. I learned my lesson: the agent uses every bit of information, so take every field seriously. Put clear, accurate, and descriptive text into each and every field. Every field is used.

Lesson #2: Using APIs is easy

I was shocked at how easy Vertex AI Agent Builder made it to integrate my agent with an API. It was three simple steps: write the description, paste in OpenAPI spec for the API, and then set up auth. As someone who’s spent the last 20+ years coding, it feels slightly uncomfortable that I don’t have to tell the agent how to access the API or write the prompt myself. But you don’t.

For this demo, I stubbed out an API to get available reservations and make reservations and hosted the two endpoints on Cloud Functions. The code is available on GitHub. The API has two endpoints and returns hardcoded answers.

I used what I learned in lesson 1 and made sure the tool’s name was simple and descriptive, the description clear, and the verbs used were the same as those of the actual API endpoints.

Lists the available reservation times for picnic shelters at specific parks, on specific days. Allows users to make a reservation for a shelter at a specific park, day, and time.”

I generated the OpenAPI YAML using Gemini. The prompt was very similar to the description above but with the actual endpoints and URLs called out. It needed a few small tweaks to the description and how the URLs were specified, but using Gemini saved me a bunch of time. Once it was done, I copied and pasted it into the Agent Builder UI.

Lastly, I had to ensure the agent could access my API. Since everything was hosted in the same project, I just had to grant the Vertex Agent service account function invocation privileges for the two functions from the Cloud Functions permissions tab..

Overall, setting up the API tool took me about an hour. Most of that was making minor tweaks to the OpenAPI YAML and figuring out where to grant function invoker permissions in the Google Cloud UI. It really does “just work” and is much simpler than I imagined.

Lesson #3: Using data stores is harder

The original version of the demo had an additional component that allowed me to ask the agent free-form questions about the parks. Like “which parks have wading pools or splash pads.” To answer these questions, the agent was instructed to use a data store tool. The data store tool, in turn, was set up to look at a Cloud Storage bucket that contained a download of all the information on parks I could find in the City of Seattle’s Open Data Portal.

I ended up cutting this from the demo, partly for time but also because I couldn’t make it work consistently. Building a data-based agent that can answer arbitrary questions is harder than I thought it would be.

The first thing I had to tackle was creating the data store itself. The UI makes it pretty straightforward but you do need to ensure that your data store, agent, and app are all in compatible regions. I lost an hour or two debugging issues that turned out to be having things in the wrong places.

The second issue I ran into was that my data wasn’t in the correct format for the tool. The open data portal provided the data in structured formats: CSV or XML. The data store tool currently supports web pages, Q&A formatted structured data, or unstructured data. My CSVs weren’t web pages or Q&A formatted, so I had the tool treat them as unstructured data. It turns out that treating structured data as unstructured data doesn’t work well, especially when using a model primarily trained on natural language, not machine-readable formats like CSV. There are many ways to address this mismatch including reformatting the data, creating custom agents, or building this app as a RAG solution myself. I haven’t tried all these yet, if I do they’ll become part 2 of this blog.

While I could setup the agent to use the data store, the responses were inconsistent. On one run it would correctly tell me which parks had splash pads or wading pools. On a subsequent run it might give an incorrect answer. And on another run it would say “unable to process request” or similar. It was also unable to reliably answer questions that

On one run the tool would correctly respond that South Lake Union park has a splash pad. The next time I tried it, it would incorrectly tell me that Harrison Ridge Greenbelt had a wading pool. On a third try it would tell me it was unable to process the request. There didn’t seem to be any rhyme or reason to it .

This is almost certainly because the data I gave it wasn’t what it was designed for. That was my mistake, but it was also disappointing since my programmer brain I assumed that clean, structured data would actually be easier to work with than raw natural language text.

Overall Lesson Learned

I really liked learning Agent Builder and look forward to trying the new features they add. But the most important thing I learned through this process is that what I, as a programmer, think is easy is not necessarily what is easy for an LLM. I knew this intellectually, but it had been a while since I had experienced it firsthand.

The thing I like about bullet journaling is that it is flexible. I’ve tried several paper planners as an adult. They either provide very little structure or so much structure that they don’t work for my life where there’s no average day. With bullet journaling, I can change things when they stop working, try new things, or skip entire weeks or days, and it’s okay. The Bullet Journal system has a few key components, like rapid logging (writing things down) and migration (retrospectives for your planning). But it can be customized to fit your unique needs. If part of the system doesn’t work for you, just don’t do it. Over the past seven years I’ve customized and streamlined the bullet journal system into something that works well for me. Maybe my system will inspire you.

The Notebook & Supplies

I prefer a classic hard-cover A5 dot grid notebook for my bullet journal. I like the size because it doesn’t occupy much desk space. I know I won’t use a notebook unless it is open on the desk in front of me, and A5 is small enough I can always have it open. A5 is also easy to carry, and hard-cover A5 notebooks aren’t damaged when I bring them back and forth to work in my bag. I don’t know why I prefer dot grid; I just do. I’ve tried lined and blank, and neither feels right.

Bullet journaling has a reputation for being complex and requiring artistic skills, and that’s not true. Personally, I do occasionally use stickers or colored pencils in my journal, but it isn’t required and I have plenty of pages with no decoration. For me the stickers, fancy lettering, and other ornaments are purely about keeping things interesting and having an artistic outlet.

The Daily “Page”

The core of my organization method is half a page per day. Why half a page? If I have more space, I fill it up and overcommit myself, and then I feel like a failure at 8 PM when I haven’t gotten everything done. Trial and error has taught me that on most days, I can do 2-3 big things and 5-7 little ones. So I give myself 15 - 20 lines per day, 10ish for tasks and another 10 for notes and doodles.

At the start of each day, I write down the three things that have to happen that day to make it a success. These are not the three most important things; they are just three things that have to get done for me to feel like I did my best that day. Some days, the top 3 things are blog posts, code, and amazing technical work. And some days, the top three things are groceries, taking out the trash, and going outside for more than 3 minutes. Usually, it is a mix of 1-2 mundane life things, and the rest are tasks for work. Next to my daily top 3, I write three “5-minute wins”. These are things I can finish in five minutes that I can pick up between meetings or when my motivation is low and I want an easy win. Short chores, making appointments, and my regular DuoLingo practice often end up on this list.

Underneath those things I copy my meetings and appointments for the day from my calendar. In any remaining space I add the tasks I want to get done. Copying my appointments over first makes it clear how much free time I’ll have for other tasks. If my meetings fill most of the half page for that day I shouldn’t plan on doing an additional 15 things. This is just another personal hack to force myself to be honest about my capabilities. Throughout the day I’ll add notes, ideas, and new tasks on any remaining lines in each days half-page.

Here is typical page from earlier this year with sensitive info obscured.

The Weekly Page & Migration

The other component of my spin on the bullet journal system is a weekly overview page. My current weekly overview has three components. At the top is my weekly top 3, which, like the daily top 3, is the list of three things that need to happen for the week to be a success.

Below that, I have a mini one-week calendar. I’m currently using this for meal planning, but I’ve used it for many different things. I’ve tracked workouts there, after-work commitments, or days in the office. It is an easy way to get a quick overview of the week.

Finally, I have a weekly To-do/Brain dump list. My To-do list is where everything I think I have to do this week or could potentially do gets written down. Everything goes on one list, whether it is for work or personal. This is the one list where I don’t worry about overcommitting. The goal of this list is to get all the things I need to do out of my head and on paper so I can chill out. Here’s an example before I filled in the To-Do section:

My weekly page is also where I do the migration part of the bullet journal method. At the start of a week (ideally), I go through the previous week’s pages and anything that did not get finished either gets marked as obsolete or gets added to this week’s to-do list. The migration process means that if I keep procrastinating on something, I have to write it again the next week and the week after that until I finally do it. This blog post, for example, has been migrated forward at least six times. Eventually, I get sick of writing the same thing every week, and I do the task, delegate it, or mark it obsolete. Occasionally, I’ll have a task or idea that I know I won’t get to at any time soon, but I’m not ready to give up altogether. Those tasks get copied to my “Someday List” elsewhere in my notebook. Every time I move to a new notebook, I migrate the “Someday List,” removing things that are no longer engaging.

The Reality

When I lay it out like this, it sounds like a lot, and I sound like some sort of organizational master. I want to be clear: I am not. My notebook has plenty of half-filled-out or blank pages. I still overplan my days at least once a week (today was particularly bad). I still occasionally miss deadlines. Perfection isn’t my goal. My goal is to find a system that I actually use more often than not that reduces my stress. And my particular spin on bullet journaling has filled that role for me.

Manage the person in front of you

For me, “manage the person in front of you” means several things. First, it reminds me that everyone needs a different management approach or style. It also reminds me that what worked last month, last week, or even three hours ago won’t always work right now. I need to be able to adapt quickly to what an individual needs at this moment.

There are some simple ways you can adapt your management style, like being extra patient and respectful of someone struggling with things in their personal life. But you may need to change other aspects of your management approach to meet your team members where they are at. Perhaps they usually prefer a collaborative coaching approach, but right now, they just want someone to tell them what to do. Maybe they typically want to be left alone to do their thing, but this week, they need regular check-ins and reassurance. I rely a lot on people reading, asking questions, and listening to what is not being said to figure out what to try next. And I’m still learning.

Managing the person in front of you is also a reminder that my direct report is not me. Our personalities, backgrounds, strengths, and weaknesses are different. I have to remember that the techniques, methods, or tricks that have worked for me may not work for them. When that happens, I need to suggest different ways to get things done or set them up with a mentor or coach who better understands their perspective.

Finally, managing the person in front of you reminds me to be realistic in my expectations. Everyone is capable of learning and growing, but that doesn’t happen for anyone overnight. I need to recognize and respect where someone is today and work from there, even if where they are today is backward from where they were last week. Sometimes, managing the person in front of me means I recognize that they can’t reach their goals on my team, and it is time for me to help find them a new place where they can be successful.

Manage the team in front of you

The next step from “manage the person in front of you” is “manage the team in front of you.” Unless I have a genuinely stellar manager, my preferred management and team structure is “loosely structured benevolent neglect.” Basically, leave me alone to figure out how to solve my problems and how to work with others to do so. I don’t like meetings for the sake of meetings. I’m not a fan of the many forms of team togetherness that companies do. I hate paperwork for the sake of paperwork.

But my team is not me. My team gave me feedback that they wanted more team connection and identity, so we have regular team meetings and a mascot. We also spent a day hacking on some code together earlier this year. I admit that doing all that was uncomfortable for me. But it was what they needed and wanted.

About 4 years ago, my team asked if we could have team OKRs like the software engineering teams, so I wrote team OKRs. At the time, few of my peers had team OKRs, but that’s okay. Their teams didn’t need them. My team did. So, I managed the team I had.

Managing the team you have also means being realistic about what the team can actually get done in a day, a week, or a quarter. Stretch goals are fine, but overestimating or underestimating your team’s bandwidth because it is easier or because you wish it were true doesn’t work. You need to manage the team you actually have, not the team you wish you had.

We manage adults

This seems trite, but especially as I’ve moved up, this phrase has helped me a lot. I manage adults. Adults take care of themselves. I don’t need to remind them that conference centers can be chilly. Adults can get their work done with minimal oversight. I don’t need to hover over them to ensure they complete their work. Adults are capable of asking for help. I don’t need to force my help on them before they ask.

Also, adults get to make bad decisions, and adults get to have unpleasant feelings. I do. My team does. It is part of being human. We turn down opportunities that would be great for our careers for various good and bad reasons. We misspeak. We show up for work grumpy. We get disappointed or sad or angry sometimes. As long as we treat others respectfully and do our best to fix things we break, it is okay if we’re disappointed that our project slipped again. I’m content to sit with a report while they’re frustrated and just hear them out. I’ll empathize or commiserate with a report who’s mad about something. I’ll share stories of my mistakes with someone upset or embarrassed by something they did incorrectly. Adults (and children) get to feel the full range of human emotions. Trying to protect them from disappointment, discomfort, frustration, or anger is disrespectful. I’ll be clear: I don’t purposefully upset my team, but I also don’t protect them from taking risks on things that may not work out. And when things don’t go to plan, I’ll be there to help them figure out what’s next.

What’s on your mind?

I’ll start with the obvious. I did start Every. Single. One-on-one. with “What’s on your mind?” for years. I’m sure I found it in some “OMG, you’re a tech manager now” book or blog post, but I kept it up because I like the kind of conversations it leads to.

I don’t like to make one-on-ones all about status. There’s a time for that, but I prefer we use one-on-one time for problem-solving and strategy. To set the tone as something other than status, an open-ended question that can be about more than work is effective. With this question, I learn about what is happening in people’s lives outside of work. I sometimes hear gossip going around the company that I may have missed. I’ll also hear about any recent product or leadership decisions that are top of mind for folks. Their answer gives me valuable context for the rest of the 1:1. If someone is upset, I can shift the meeting to be about reassurance and support. If someone is excited, I can amp my energy up to match. If there’s something they’re proud of, we can celebrate together.

When I finally got bored of “What’s on your mind?” I switched it up. Recently, I’ve used “What’s on your list?”, “What do you want to talk about?” and “What’s important to you today?” to achieve the same goals. Even then, I still use “What’s on your mind?” several times a week.

How will we know when you are done? What does success look like?

In my experience, one of the most challenging skills to master, especially for intelligent folks, is defining success or even completion. Many folks I know, including myself at times, are prone to getting stuck in a revision loop. They’ll spend hours, days, or weeks trying to make the code 1% better or the slides slightly better visually, even if they’re past the point of diminishing returns. Additionally, in DevRel, it can be tricky to define success. Many DevRel teams don’t use metrics effectively, and even if they do, it can be hard to see how one person’s work contributes to the team’s success.

I help with this skill using two questions. First, we must establish when a project is complete. My go-to question to define completion is “How will you know when you are done?” or the slightly more intimidating “How will you, me, and my boss know when you are done?” Folks usually have an answer for this, although sometimes it is challenging, particularly with learning tasks or open-ended investigations.

Once we’ve established what “done” looks like, I follow up with “What does success look like?” Sometimes, just finishing a task is success. For example, closing bugs can be success. But there’s usually a bigger story if you look deeper. Maybe the end goal of closing the bugs is to see error rates fall.

If we’re struggling to define success, my next question is, “Why are we doing this work?” If that doesn’t help us find a good definition of success, I’ll say, “Imagine we time travel 6 months into the future, and we’re deciding if we should continue/repeat this work or invest more. How would you want us to make that decision?” Usually, one of those will break loose some ideas for evaluating the success of our work.

Many folks use the SMART(ER) goals framework to define success. For me, that’s the end state. In my experience, if you start with that framework, some people will pick any random metric for the measurable part, even if it doesn’t make sense or is out of their control. By starting with open-ended questions like “What does success look like?” we can define completion and success without the strict guidelines of a framework. After we’ve deciphered what our instincts say about success and brainstormed possible ways to evaluate our work, we can use a framework to formalize the goals.

If you had a magic wand, what would you change about the product/team?

Sometimes, in the sprint to a deadline, or the frustration of yak-shaving on an issue, we can get “blinders” and lose sight of the bigger picture. Losing context like this makes it difficult to make good decisions because you can’t anticipate how your work will impact others. I’ve also seen it contribute to a lack of developer empathy.

When I notice myself or others overfocusing on something in a problematic way, I’ll use a thought experiment to try to break out of our current myopia. Usually, I say something like, “If you had a magic wand, what would you change about this product?” or “If you had a magic wand, what would you change about the team?” The magic wand bit is essential. If I leave it off, people often limit themselves to what seems doable or easy. If the goal is to get beyond the current problem, we must stop worrying about what’s possible or easy for just a few minutes. Bringing in some whimsy, like the idea of a magic wand, makes it clear that reality and practicality aren’t crucial in this moment.

What are you trying to do?

This has been a favorite question of mine for more than a decade. Before I was a manager, I would use it when helping folks at meetups who were learning to code or learning Ruby. The answer to this question helps you quickly understand the level that someone is currently thinking at. Often, it also enables you to gauge their understanding of their tools and their problem. An answer of “I’m trying to make it not throw a null pointer exception” is different from “I’m trying to remove all the keys from this HashMap that have a value of false” and is different from “I’m trying to create the order summary page for my site.”

Once you’ve established the altitude someone is working at, you can tailor your response to their current level of thinking. Someone struggling with syntax doesn’t need a lecture on architecture.

Additionally, suppose their answer is highly technical but doesn’t match their code or doesn’t make conceptual sense in their language or framework. That is a sign that they misunderstood something several steps ago. When this happens, you need to work backward to find and resolve their misunderstanding before you can help them move forward.

I now use this question to help folks debug people situations about as often as I use it for code. When solving interpersonal conflicts, I usually need to ask it multiple times to truly understand the situation. “I’m trying to get so-and-so to stop being a jerk.” will morph into something more actionable like, “I’m trying to get them to respect my opinion and accept my proposed solution.” when you repeat, “What are you trying to do?” a few times.

Tell me more about that…

This last phrase I use frequently isn’t a question. It is the simple phrase, “Tell me more about that.” My team is diverse. They have different backgrounds, interests, and specialties and work on different parts of Google Cloud. They’ll often tell me about a problem they’re facing or something technical they’ve learned that I don’t know well. When that happens, I usually respond with, “Oooh, tell me more about it!” This isn’t limited to technical topics either. Recently a direct mentioned the game The Finals. I had never heard of it, so I asked them to tell me more and learned some fascinating things about current game trends.

I’ve tried some variants of this phrase. Years ago, a friend recommended “Teach me about that” to swap roles and shift power dynamics with your team. I’ve tried it, but it doesn’t work as well for me. I feel that the word teach makes the interaction too formal and puts expectations on folks to explain things the “right” way. When I’m really struggling to follow something and need more info, I sometimes fall back on “Can you give me more words about that?” or “Can you use different words to explain that?” or even “Can you give me the explain like I’m five version of that?” If I use one of these phrases, I am careful to repeat back what I heard so they can verify that I understand. That’s good active listening, but it also lets the folks on my team correct me and ensures we’re continuing the conversation from a place of mutual understanding.

Do you have go-to phrases?

I’m curious if others have phrases and questions they use repeatedly or if I’m unique in my choice to use the same questions repeatedly. Also, if folks have great questions or phrases, I’d love to add them to my managerial toolbox.

When I was little, my parents talked to us about dyslexia and other learning differences using phrases like “everyone’s brain works differently” and “different people need different tools to be successful.” For example, my sister might listen to a novel instead of reading it on paper. Someone with ADHD might use a watch alarm to remind them to feed the cat. A tool people with dyslexia sometimes use is scribing (from the word scribe), where they dictate something and another person writes what they say. My sister would use this technique to speed up the creation of rough drafts for essays and term papers.

When I became a manager and began supporting people on their career journeys, I started reaching back to the things I saw and learned growing up to help peole with “different brains.” I’ve tried many of the tools and techniques I learned, but the one I turn to the most is scribing. The word scribing can be awkward, so instead, I refer to it as “you talk, I type.”

I mostly use this technique in one-on-one brainstorming meetings, especially around strategy. I’ve also used it to help when someone has many projects in flight and wants my help prioritizing them. In general, I reach for this tool if I get the impression that a situation is complex enough or stressful enough that it is difficult to simultaneously come up with ideas and record and organize them. And while I have neurodiverse folks on my team, I’ve used this technique with everyone at some point. It is another example of the curb cut effect, something I learned in the context of accommodating disability can benefit everyone.

I’ve developed some general guidelines about how I run a “you talk, I type” session based on both scribing and pair programming best practices. For example, I offer to write while someone else talks, but I don’t insist. I also offer to share my screen if it makes them more comfortable seeing what I’m typing in real time. Everyone needs to be comfortable for this technique to be successful.

I break from traditional scribing in that I don’t write down things verbatim. I also don’t expect complete sentences and paragraphs from the talker. When I type, I use a lot of bulleted lists, and I paraphrase or summarize to make ideas more concise. For me, the goal isn’t a word-for-word record of what is said. The goal is to get all the ideas out of their head and into a form where we can organize thoughts, address concerns, and generally make sense of a situation.

If the talker stalls or says they have nothing more to say, I’ll ask a few questions to ensure we didn’t miss anything important. My favorite question to keep things going is, “What else?” I’ll repeat that over and over as long as it keeps the ideas flowing. I’ll also bring up edge cases like “What do we need to think about for folks outside the US?” or “What events do we have coming up?”

Once we’re satisfied all the ideas are on “paper,” we can start grouping related ideas, organizing by priority, rephrasing things, and whatever else we need to do to accomplish our original task. I’ll often highlight related items in the same color to avoid moving them around the page. Sometimes, I’ll ask the person I’ve been collaborating with to review what’s written and identify any patterns or groups they see. I try to let whoever did the talking lead this part of the process so that they have ownership of the final result.

I’ve been asked, “How is this different than just taking notes?” There are a lot of similarities, but for me, it feels much more like driver-navigator pair programming. I’m acting as an input device. I’m largely silent, only speaking up when I need the other person to clarify something, change their current level of abstraction, etc. In a normal meeting, there’s much more back-and-forth discussion, and ideally, one person isn’t doing all the direction setting and supplying all the ideas. When I do “you talk, I type,” nearly all of the ideas and information come from someone else, and I’m just writing them down for us to work with later.

I’m curious if others do something similar with their direct reports or peers. I don’t think we recognize how hard it can be to simultaneously express new ideas and write them down. I’m frequently amazed that by eliminating the need for someone to write or remember things they can suddenly come up with many creative solutions to problems and awesome ideas for future projects.

One of my main work projects this year is Jump Start Solutions. These are a fast way to get hands-on and learn with Google Cloud. Each solution is a package of Terraform, application code, and interactive tutorials. The solutions can be deployed with one click in the UI, so you don’t have to install anything on your dev environment to build applications and learn new things.

It turns out, Jump Start Solutions is another optimization problem. Early on, we decided that the solutions should deploy quickly. We also wanted them to be free or cheap to try. We felt it was important that the solutions be more realistic than existing “Hello, World” type demos. And we wanted to show folks how to properly set up things like secrets, networking, and IAM. Lastly, since the program aims to accelerate learning, everything in a solution must be easy to explain and understand.

Those constraints gave us another optimization problem, “Fast, cheap, realistic, and understandable. Pick any N.” Except we didn’t know if N was 1 or 3. We just knew it wasn’t going to be 4.

You can think of these things as a four-team tug-of-war. Each time you start to optimize one constraint, you move further away from the goal with another.

For example, realistic applications with properly designed security, CI/CD, networking, etc, are complex. They have a lot of components connected in complicated ways. This makes them hard to understand, especially for those new to the cloud.

Likewise, realistic CI/CD and deployment pipelines are often slow, especially on an initial release that requires complete provisioning and a new build. Including a realistic CI/CD setup in our solutions slowed down deployments.

In the end, we had to make some compromises. One of the significant compromises we made was prebuilding containers and deploying our applications directly from Terraform. Deploying directly from Terraform isn’t a best practice. Instead, a better approach is to use Terraform for provisioning and use a CI/CD pipeline to deploy your application and run any post-deploy setup steps. In most cases, you also wouldn’t deploy your application from a container image someone else built months ago. You’d build with the latest dependencies and code changes, run tests or security scans, and then deploy. But again, all that takes time. Using prebuilt artifacts and having a single step in our provisioning and deployment process allowed us to cut deployment time by half or more. Ultimately, the improved user experience was worth deviating from best practice.

We made another compromise around deployment time as well. Initially, we had hoped to get deployment time under 5 minutes. We figured folks would only be willing to wait that long. But 5 minutes to provision infrastructure, especially infrastructure, set up networks, deploy an application, and do any post-deploy steps like loading a test database was only achievable for some of the solutions we wanted to build. We could have also made things fast by reducing the complexity, but we felt strongly that basic and “Hello, World” applications weren’t an adequate learning environment. So, we relaxed our requirement from 5 to 10 minute deployment times as the goal. We also created a review process for any solutions that took longer so we could be confident the learning goals were worth the added wait time.

Finally, there was the issue of cost. We want to keep costs down and prioritize using free tier and trial credits as much as possible. However, real-life architectures don’t usually use the smallest possible database, a micro k8s cluster, or precisely one VM. If you want to help folks learn concepts like multi-zone architectures, load balancing across VM clusters, or sharing large amounts of data, you must provide enough cloud resources to teach those skills. Our original requirements for a Jump Start Solution set a target max price. But we also make a point to review each solution to ensure that any time we don’t use free resources, it is necessary to support the learning goals for that solution.

While I wish we could have pulled off a magic trick and managed “fast, cheap, realistic, and understandable, you get all 4!” the world doesn’t work that way. Ultimately, I found thinking through acceptable versus unacceptable tradeoffs in this optimization problem fascinating. I’m proud of what we managed to do even when we couldn’t do everything perfectly.

I’ve also gained new perspectives on many of the stories from my early career. I sometimes wish I could go back and share what I’ve learned with my younger self. Sadly, time machines still don’t exist, so I’ll share here and hope it helps someone else.

The impossible assignment

This story takes place early in my career when I was working as a tester. At the start of the planning cycle, I was pulled aside and told I was getting a special assignment working with just one developer.

Our assignment was to automate the processing of a particular ticket queue. Most of the tickets needed no action. Some got sent to an automated process for further handling. And some required the support team to step in and intervene. Today, we’d probably use an LLM for this type of thing, but it was the mid-2000s, so we built a decision tree.

Once we had the architecture decided, I started preparing my test data. The tickets were all plain text. In my overeager youth, I decided this meant I needed to learn Perl. I bought the Camel Book and, two days later, knew enough Perl to parse the stream of tickets and build up a labeled set of test data and a test harness.

The dev and I started iterating and fine-tuning the decision tree. While we worked, I monitored the queue for any interesting edge cases I could add to our test data. After a few weeks, we had a system that could correctly process all the tickets coming into the queue. And we documented the rules used to make the decisions. I was so proud of myself.

A few days after we finished, I was in a 1:1 with my grandboss. He asked what I had been working on. I shared our recent success with our queue processing tool. He responded with something like, “Oh, that project. Yeah, we all thought that was impossible. Good on you for figuring it out.” He then shared that while I was working, they’d hired a vendor to build a solution. The vendor solution was less accurate than what we built. In the end, the company chose the vendor solution.

So, what did I learn from this experience?

Not all assignments are good assignments

The most important thing I learned from that situation was that not all assignments were good assignments.

I thought this was a good assignment. It was special. It was technologically interesting. I got to show off my tech skills and write code. I also knew that if we were successful, we would save the company a lot of money and help protect the brand’s reputation. All of that made it seem like doing this well would reflect well on me.

But in the end, it was a bad assignment. It didn’t help my career. The work wasn’t valuable to the people above me. They’d decided we’d fail before we even began. They struggled to believe us even when presented with evidence that our solution was cheaper and more accurate than the alternative. At review time, I wrote, “strong work on a canceled project.”

With more experience, I know to be wary of “special” assignments that are understaffed. I’m also cautious of work that doesn’t “look like” what the rest of the team was doing. While I was learning Perl and building test harnesses, my peers were doing manual software testing. That made it harder to compare my work to theirs and show how it contributed to shared team goals.

Bad assignments can have good parts

Even though this was a bad assignment, there were some good parts. Most importantly, it started my transition from test to dev. It also reignited my interest in automated software testing as a way to reduce feedback time and prevent regressions.

I also got to learn Perl on this project. Knowing Perl made it easy to pick up Ruby a few years later. And Ruby unlocked much of my current career.

Learning Perl in only a few days made me confident that I could keep up with new technology trends and learn whatever skills I needed for a project.

This project was also my first exposure to super-basic AI in the real world. Since then, I’ve built several other systems like this, most recently for the demo I worked on for I/O 2022.

Don’t assign impossible work to your team

My experience with an “impossible” project influences how I manage my team. Simply put, I won’t assign impossible work. That shouldn’t need to be said, and it shouldn’t be a thing that happens, but it does. So, I look out for tasks that seem extremely unlikely to succeed.

I also try not to assign work that will be thrown away or won’t be valued by those above me in our leadership chain. This can be hard. Sometimes, projects get canceled with no warning. But I do my best. That often means moving slowly to see how things evolve or doing some investigation myself before offering a project to a team member. When I’m unsure about the long-term prospects for a project, I offer it to the more senior members of my team, and I’m clear about the risks I see.

I strive to keep my management 1-liners phrased as positives. So instead of “don’t assign impossible work” and “don’t assign throw away work,” I say, “Don’t offer projects unless there’s a reasonable chance of success.”

Also, it is easier to say no to bad assignments if you are a bit overloaded or a bit understaffed. “I’ll get to that after I complete these higher priority items” is, after all, the business way of saying no. But even if you can’t blame your workload, there are ways to politely decline assignments that “aren’t aligned with our career goals and the direction we’re taking the team.”