Quality Control at High Velocity

Computer Science is ground zero for the impact of LLMs and their Agent Harnesses. All white collar jobs will likely see some impact from AI powered tools, but in programming, the revolution is not around the corner. It happened a couple years ago and adapting to the new state of play is no longer optional.

One of the most consequential changes in this new era is the rising pace of development. Features or services which used to take weeks and months are now being completed in hours and days. A recent and notorious examples of this rapid pace, illustrates the consequences of this unprecedented developer velocity: Bun's port from Zig to Rust

This article is worth the read, however, for those who are not familiar with the conversation around this work, it should be said that this was a highly controversial initiative. Even major programming figures who promote Rust, like Bryan Cantrill, found issue with this project. I don't have much to say on the commentary around the port, but it has become a massive and publicly documented case study for the Rust vs Zig debate and, important to my argument today, it demonstrates how quality is impacted by AI programming tools. This is an excellent breakdown of the change, the results before and after, and some of the controversy around the rewrite. Bun Rewrite Analysis

Bun's recent rewrite proves important aspects of the AI programing tools we now use on the daily: Agents can produce useful code artifacts at a massive scale with eye watering velocity. Quality is not an inherent characteristic of the code produced by Agents, and quality is impacted by developer tool choices. Finally, switching from Zig to Rust overnight with 1 million agent written lines of code did not have a dramatic impact on code quality.

So what does this mean for the general set of developers? I believe it is not possible to compete as a developer on the open market without the help of Agents because the productivity win is just too strong. However, this increase will come at the cost of quality. Even with $165k in token spend, an unreleased cutting edge model, one of the largest test suites in the world, the memory safety guarantees of Rust, and a working application to directly copy, the final artifact was not free of quality issues. The number of bug

I am not working on problems which require the complexity of Bun, however, I solve problems every day with a $100/mo subscription, some loose feature requirements from a ticket, 200 lines of Typescript which have no runtime guarantees, a cursor human review, and some regression tests for unrelated parts of the code base. If this description is representative of your job, I expect us to be drowning in maintenance, bleeding trust, and failing to deliver on our sales pitch.

As far as I can tell, if we try to increase speed without any changes, the decisions which are industry standard in web development will prove to be cataclysmic.

I have been working on this problem for some time by accident, because building systems reliably without expecting everyone on the team to be a phenomenal engineer is identical to building with Agents in mind. In some sense, that is the North Star of how to compete in the era of prolific AI development tools:

Build your projects with tools and techniques which encourage safe, performant, and simple implementations and discourage or block unsafe, slow, and complex implementations.

This is not new, and it is the opposite of rocket science. However, it is critical to decreasing the error rate of both humans and agents. Here are my preferred tactics and tools to implement this philosophy.

Tactic 1: Compile Time Verification

When working with Agents, providing low latency tool calls which grade some category of code quality is essential. We have excellent tools for this that already exist, and we need to consider these essential to modern projects.

• Static Types
• Unit Tests
• Assertions
• Compilation
• Garbage Collection

If you follow the industry standard for web development, or ask AI to build a project from scratch without directing it, you will use one of the most popular languages in the world, a dynamically typed interpreted language, unit tests will be sparse and assertions are absent entirely. However, leveraging all of these tools together creates an excellent recipe for improving basic correctness. Before your human review, if the agent can ensure that the code compiles, all functions are called with the correct types, there are no regressions against the expected behavior of the system, and no invariants are violated, it prevents a huge set of issues automatically. This baseline of correctness is non-negotiable. If your patterns or tools cannot give you the ability to verify these qualities at compile time, you are begging for the dumbest problems to surface at runtime.

A brief aside on Typescript.

Typescript is sometimes called progressively typed, however this is a poor term. While it does provide type information as an overlay on top of javascript, this does not have the same guarantees which a typed language provides. At run time, in production, when users actually use your service, these guarantees are not enforced strong enough to be helpful. The typescript compiler guarantees that the system is type safe, if and only if, what you have told it about the real world is valid. This creates circular logic: if the developer provides accurate type information to the Typescript compiler, the compiler will validate those types are handled properly. Also at any point in time the whole type checking engine can be turned off with the any type. This is the worst type of 'guarantee': correct, but only if the programmer is correct, and not always enforced. Runtime enforcement with Zod can improve the situation, however, it still relies on the developer properly gating all external entry points into the codebase with Zod objects which are parsed with the runtime enforcement mechanism. Developers and Agents remain ultimately responsible for enforcing the correctness themselves. If you work in Javascript, you should absolutely use Typescript, and I respect the engineering which has made it into an excellent developer tool. I feel the same about Zod, but for AI assisted development, I believe stronger guarantees can and should be enforced.

With these desired qualities in mind, I believe Go is the only reasonable choice. It is a bold claim to declare a single programming language the absolute winner, but this claim is domain specific. I believe for the set of programs which are related to web development, Go is the correct choice when possible. Compiling to a static binary, compile time static typing, and garbage collection form a robust basis for applications. Rust is the closest competitor on these dimensions, however its complexity alone disqualifies it, as complexity in language design encourages logical bugs despite the borrow checker being excellent for safety without the performance penalty of garbage collection. Rust makes sense for many problems, however, those problems have more performance or scale requirements than what I work on.

Unit tests help ensure the next step after compiler errors and type checking: logical correctness. With AI assistance, writing tests is essentially free, so there are very few downsides to writing a huge amount of tests. It can be very helpful to over test the codebase at the unit level; so long as the time to run all of the tests remains small. If you have large amount of super fast unit tests, an Agent can call the tests and ensure that any existing functionality remains intact after their changes.

Asserting the negative space of expectations is one of the most helpful things I have learned from Tiger Style and it pairs well with excessive unit testing. When writing a function, if you accept two buffers and expect them to be the same length, I recommend checking that condition is true and returning an error if not. Then, at the end of the function, check again and crash if the check fails. This pattern will seem excessive at first, but it greatly increases your ability to detect issues before production. If you let yourself or an AI program in a permissive style in which you 'correct' small issues all over the place, over time, the codebase will feel haunted. I was working in a system early this year which became haunted and I really don’t want you to suffer through that. My co-worker had a habit of ‘fixing’ the data being entered into the system rather than crashing or returning an error. If the user entered a string which was too long, the system would truncate it. If the database row was missing a column, add a null value to keep it type safe and act like nothing wrong happened. This created a foggy grey area between inputs being received from the user's form response and what was written to the database. Instead of a user support ticket saying: "my form is being rejected" instead we had to debug silently truncated strings after the bug was already causing issues in our testing group. The co-worker in question was GPT 5 Codex.

To mitigate this, in my instructions to agents and in my own code, I try to fail very loudly with assertion errors when expectations are not met instead of “fixing” the issue. With this in place, comprehensive unit tests can be used to search the space of codepaths for crashes. This produces a system which feels solid and is either working properly or is crashed immediately. Improper responses are worse than downtime in almost any application where you care about correct answers.

Tactic 2: Eliminate the Frontend

With some basics for correctness covered, let’s discuss more controversial patterns I am having success with. I work largely as a full stack web developer and I am now a strong advocate for getting rid of the separation between the frontend and backend then removing nearly all frontend JS.

I used to build web applications like this:

Postges -> ExpressJS or Flask -> React

Which when deployed would look like this on GCP:

Managed Cloud SQL -> Cloud Run Service -> Cloud Run Service

First and foremost, this architecture fails because JavaScript is unfit for production outside of limited usage in the browser. (See lesson #1.) In most applications there are three primary responsibilities: state persistence, access / mutation control, and presentation. The industry standard approach may seem like it houses these concerns in three separate locations with clean separation: database, api, and frontend. However, all three of these are leaky abstractions with all three locations deeply interconnected in a well built system.

Here are some downstream problems caused by that core issue:

• duplicates work, and all copies must be kept in sync.
• encourages poor patterns.
• turns a simple website into distributed systems problem.

Double the Work

When a system is built in this style, you duplicate nearly every problem and have to implement solutions twice, often in different languages. If you want to accomplish simple functionality like a multi step form, you need to setup the javascript in React to get state from the API, show the current state of the form, validate the state before submitting to the API, then display the response. In the API you need to fetch the state from the database, expose the state to the frontend, validate all responses from the frontend, record state in the db, then tell the frontend the request was processed. This is an inescapably pattern which duplicates fetching the current state, making that state available, validating input, and reporting status. If the logic underpinning those stages becomes out of sync, you will have bugs. If you happen to be unfortunate enough to have different teams working on the frontend and backend, there will be a huge amount of communication overhead to get this right.

Encourages Poor Patterns

You can escape the duplicate code problem, but the 'solutions' are usually poor trades. To eliminate duplicate state, something could be pushed entirely to the frontend. In a multi stage form, if it was implemented purely in the frontend, you save code by only writing a POST API route for final submission. However, the form cannot be resumed from another device and, if poorly implemented, the content will be lost on page refresh.

I have pushed the 'frontend only' style to the limit with an internal tool I built that has no API. It is only a React app and noSQL database. This can work, but it is wildly insecure even with perfect database permissions schemes because all business logic is running in an untrusted environment. This model also makes debugging nearly impossible because you cannot properly instrument all client devices without an insane Sentry logging system.

NextJS and similar innovative frameworks push really hard in this direction, confirming the duplication problem is industry wide. The quantity of security issues in these frameworks reveal the inherent security problems of this model. Not to mention the performance optimization nightmare which is invited by the lack of clarity around where operations are occurring.

Distributed Systems Problem

When you have a database, potentially many API instances, and every client running their own copy of the frontend, you have turned a website into a distributed systems problem. An enormous amount of problems are now on the table with this architecture. Distributed systems problems are notoriously very difficult, and most of the time, we avoid or crudely solve these problems in modern web development. We can avoid dealing with any of these problems with a better architecture.

All of these problems do not go away with AI programming, instead they are amplified. Fixing the architecture of modern web applications is a prerequisite for high quality AI assisted development.

Solution - API Only

Instead, I recommend everything be server rendered with DataStar filling any client side interactivity needs. DataStar is an excellent technology because it makes any interactivity you would like possible while retaining a server rendered design. After implementing this architecture, my 'frontend' is a folder of html templates in my API. This means only one repository and one runtime for all code. The simplicity of this solutions eliminates every complaint I have had with the frontend. No code duplication, no pressure to use bad patterns, no more wrangling distributed systems. This is the vision of modern frameworks like NextJS without the burden of JavaScript and the crippling security issues.

If you have never tried using a server driven pattern with data star, you will be blown away at how easy it is to build applications which traditionally are wildly difficult. Shared Game of Life However, the benefits do not stop there. In lesson 1, unit testing was important because it validates system behavior and helps to prevent regressions. When all of the frontend is html templates, testing is very easy. Just check for properties in the html output. To ensure a formatting system is working, make dummy data, pass it to the function which hydrates the html, and ensure the properly formatted string is returned. This also solves the instrumentation problem. Just capture the HTML being sent to the client device and you can replace / examine all meaningful state which is not already being tracked with the server's observability suite.

Because the browser works in declarative strings, using declarative strings to interact with it, rather than a complex JavaScript state machine, eliminates an uncountable set of problems before they can steal your time.

Not to belabor the point, but if you make architectural decisions to improve the development experience for humans, you do the same for Agents. Every choice which reduces complexity and improves testability will result in more accurate artifacts from the models and improve the speed at which they can be produced.

Tactic 3: Database Latency is Everything

For essentially all web applications, your main constraint on performance is database latency. Bryan Cantrill has an excellent talk about debugging performance issues. The Hurricane's Butterfly: Debugging Pathologically Performing Systems The most important part of the talk is the clarification that improving the performance of a system is debugging, not tuning. Highly recommend listening to his rationale behind that distinction, the key lesson being performance issues are a problem to be solved, not a situation where you haphazardly change random things until your numbers go down.

If you watch flame graphs of your web application running in the browser, or using my API only tactic, watch the call graph as Go builds the html template, database latency will typically dominate the picture. If this is not true, this is probably a bug you need to go track down. In these types of systems, almost all of latency between a user pressing a button and the green check of successful submission is network round trips and database latency.

To make progress on this, tactic #2 helps by preventing network round trips between API and Frontend, but through necessity it must still have the network round trip to the browser, so it generally turns out to have a net neutral change to latency. However, if you are hydrating templates in the API, why not co-locate the database in this API, so there is no database latency? For some applications this is not feasible, but for many more applications than you would think, Sqlite is a practical solution. It is shocking what is possible to build with a single Go binary that serves HTML fragments and has Sqlite for all non-blob persistence. Recently, I completed performance testing on a new planning tool which uses this exact stack. On a single instance of a Google Compute Engine, with 2 vCPU and 1GB RAM, I am able to serve and estimated 100 daily active users for $12. This planning tool has 3 year retention for plans, which have many subcomponents and storage hungry features like change logs and permission records. Despite these features making storage seem like a potential issue, using only a 10GB system volume for the GCE instance should be able to support +2k plans in a production setting. This performance is insane and only possible because of the latency reduction in the data path.

Tactic 4: Simplify DevOps

This is the most exotic of my recommended tactics, but it follows the prior tactic closely. If you want to run Sqlite or even Postgres on the same machine as your application for latency reasons, you cannot use server-less systems like Google Cloud Run. This initially seemed like a problem to me, however, I have started having tremendous success with Nanos as a replacement to Docker + Server-less container orchestration. This is way too much to cover in a couple paragraphs, but the results of making this choice dramatically simplify dev-ops and make co-locating the database actually a reasonable option.