Full Transcript
[Music] Welcome to the Solspace Podcast. Thanks for listening.
Welcome back to the Solspace Podcast.
This is Mitchell Kimbrough, founder of Solspace, your host. Today's going to be another solo cast. I've gotten to a point where I like to listen to quite a few solo casts myself and I get a lot of value out of them, so we've gotten pretty decent feedback on the ones I've produced.
So here's another one. Today's topic is why AI-generated websites are breaking in production and how to prevent it. So we're in an AI age now.
Web developers and the marketing directors who employ those people, they're using AI tools like Claude Code, v0, Bolt. These are AI web development tools and they let web developers move faster and test ideas more cheaply than they could before. There's real value in this, but the speed they're gaining in development doesn't automatically translate into reliability in production.
So on this topic of using AI web development tools, these are some of the things that as a marketing director you should be watching out for as the people that you work with use these tools and start to deploy this code into production. So the first topic, as always, is security. Just this past couple of weeks, Vercel, which is a platform as a service, kind of a web hosting provider that we like to use a lot, they sent out emails about a security update required for the open source platform Next.js that they maintain.
This is a popular JavaScript website and web application framework and it's really popular with AI web development tools because AI loves to build with Next.js and React, which is part of what underpins Next.js. So you're getting a lot of this kind of code generated by these AI tools. So a vulnerability was detected in that stack and all these specific use cases were problematic, as is the case with all kinds of security issues that happen with web software over time. Anyone with an affected website needed to promptly update their code in order to avoid a security exploit.
So let's imagine that you launched an AI-generated website using this code base, which is quite common. Then you start getting emails from the web host that you need to deal with the security vulnerability. But, I mean, aren't you busy handling your own marketing duties, your own campaigns, your own staff issues, and all the other stuff you need to deal with? Do you really have time to drop everything and handle the security issue? Well, you have to, because the website security is for real.
This is your brand. Having a problem here is a real problem in your role. This is something that becomes a priority.
We consider this a core reliability problem. And that's the thought that we're always thinking about with websites is this is reliability. How can we get these websites to be as reliable as possible? Security ends up being one of the first considerations.
So now you have to triage this. You have to determine, is your website affected by the security issue? You have to figure that out, because you need to know if you're going to drop everything to take care of this. Are you using the specific library that's referenced in the security bulletin? Are you on the version that has the problem? If you go and do the upgrade they recommend, will other stuff break? How long will it take you to cross-check everything after you've done the upgrade? Wouldn't it be great if you could just call your web developer and hand the problem off? Somebody you got on retainer who can deal with this? Well, it still is relevant, because that team could have potentially used an AI web development tool to build up part of the code base that's affected.
So the question is, do they understand it well enough to dive in and fix it? Do you have to get involved? Security matters here. There's another thing to think about when it comes to AI web development tools, and that's called the context window. So this is moving into some more technical territory here, but think about an AI's or an LLM's context window a lot like your computer's RAM.
So the amount of RAM that you have on your computer is determinant of the size of problem that your computer can handle. So with an LLM, there are upper limits on the context window, which means there's only so much of a website's code base that can be loaded into the LLM at a time, and for it to be effective with that. Without all the context for the website's code, though, the LLM is destined to create some problems that only surface in production use cases.
So how does this translate to my marketing director clients? This means your AI could miss the effect that a change in part of the website's code can have on another part of the website's code. These websites get pretty complicated, and the code base gets pretty large, and if you reach upper limits on what the context window for the LLM can handle, first of all, the work is going to move more slowly. Second of all, details could be missed.
The next issue to talk about with respect to web development tools in the AI space is the question of input and output. So unless yours is a pure content website, these websites and web apps that AI tools will help us build will do what all computers do. They'll receive input and produce an output.
In production, the permutations that are possible for inputs increase quite a bit compared to what they're like during development. A lot more people hitting the same application, doing slight variations on the same theme, that really changes the scope of bugs that can crop up. Now an experienced web developer is in a better position to test for production use cases.
We can often intuit where in the application things might break once the general public starts to use it. We've been there before, we've done it. So the AI opportunity increases the speed of development, but maintaining your relationship with the developer means you avoid production problems.
The next thing to be thinking about with these AI web development tools is the environment. Your site is great in the constrained environment of the AI website builder. But when you deploy it to production level servers, will the environment be the same? The environment here is the servers and the infrastructure that run the website, the types of the underlying operating systems and so forth.
You have to ask, will those things remain the same over time? Now with a lot of tools like Netlify and Vercel, which are platform as a service providers, deployments and environments are nicely streamlined. So you get a nice sort of coherent environment each time is deployed on each instance across the CDN or whatever the case may be. So you have some nice consistency there, but you still have to think about this problem.
But when you do run into problems with databases, config files, APIs and the like, the LLM you use to build the thing is not necessarily going to be capable of helping you fix the production level environment. You can't necessarily see it or get that context loaded and understand it. You need a developer like a human brain for this.
Next issue to talk about with these web development AI platforms, again, stuff like v0, which is a great platform, Bolt, Claude code is bloat. You got the issue of code bloat. So just today I was fixing an issue with an AI generated website that I had built a few months ago.
It had included some UI components that my site was not using and likely never would. When I went to make a small update, the site would, it just wouldn't build anymore. There was a problem with the dependency and one of the UI components that I was not using.
So the AI originally built this code base. It included some UI components that it thought I would want or need at some point, just kind of out of habit. And then one of those components had a dependency on something else that had fallen out of version.
It had fallen behind. Whoever was maintaining that code fell behind or abandoned the package or whatever the case may be. So I had to track down the chain of code that eventually ended in that unused component.
I would track all those pieces down and extract them surgically so that I could get my build to complete and get my code to push up to production. So as a developer, I would have to know when I was removing unneeded and extraneous components, I'd have to know that those would not affect the application and do any harm in production. And the LLM and AI is not necessarily going to know it, know the full extent of the code base the way that a human brain that has training in this and experience in this can.
So once I deleted that component, I had to deploy the whole thing again. This whole experience is what we mean by code bloat. You get more code in there that you need.
You don't know which pieces are required in which you can delete. You're not sure how they all hang together. And it just becomes a big bloated mess, which you definitely don't need as a marketing director trying to manage multiple web properties.
What this translates to for you is you're maintaining code, you're maintaining digital assets, that the security implications of this are that you are responsible for securing unnecessary code, you have to worry about something that shouldn't even exist in the first place. Another thing to consider when using these AI platforms are edge cases. So an LLM can build a pretty sweet app, and do so pretty quickly with minimal effort from you or your developer.
But they don't have an imagination. These LLMs cannot necessarily imagine the different variations that they might be presented with in user behavior out in the real world, the edge cases. An experienced human developer can imagine these weird edge cases, and they can code for those.
They can set up an architecture that's ready for that. They can adapt to that over time. And these are the sorts of things that you will only see in production.
You won't see in testing, and you won't see at the conceptual level necessarily. It takes some experience and some human creativity to see these things. So good LLM usage combined with a good experienced web developer can be the perfect combo actually, because you can proactively avoid those weird edge cases that will create future pain for you, while leveraging and taking advantage of the speed and ease with which you can get apps up and running and built using these AI platforms.
So how this translates for you as the marketing director is, there are weird scenarios that will pop up in production. It'll happen on your watch, someone in the C-suite is going to notice, and you're going to have to deal with it. And why create that problem in the first place? And maybe secondarily, why not have someone available, like an actual human who's been there and done that before, who can help resolve these things when they come up? So to wrap up, AI is great.
I've been really impressed with this set of tools that we've been able to use in our web development work. We think that there's a lot of great potential here for us to speed up our development, make things less expensive for our marketing director clients. There's a lot of opportunities for you guys to bring AI into your websites to do more interesting things for your customers, to reduce friction and create more reliability.
But remember, AI is just a tool. The person who knows which tool to use and how to use it matters just as much now as it ever did. So just imagine the possibilities of a team made up of you, a great web developer, and some excellent AI tools.
Fit those things together in the right way, and you can achieve some excellent results over time. Thanks for listening. Thanks for tuning in.
Check back again. I intend to continue talking about AI, talking about how these things overlap with the concerns that marketing directors have to deal with on a daily basis.
[Music] You've been listening to the Solspace Podcast.