A co-founder I advise asked me about hiring a high school student for a software development internship. They had an impressive portfolio, but it was also clear that most of it was AI-generated. The co-founder asked, “How should I think about this?” My answer was immediate: “I have no idea!!!“

After several weeks of some intense “vibe coding” [I do not like this term, but here we are], I have a more nuanced perspective.

To start, I’ve reactivated the ~30-year-old dormant neurons I used to manage developers. A good tech lead always tries to understand each developer’s strengths & weaknesses and adapt accordingly. Junior developers may need regular code and design reviews, where a more senior developer can propose an API/design and then go implement without much guidance.

In that framework, AI coding agents (today) are very strong on knowledge (e.g., instant access to Google) but weak on judgment and experience. They are simply amazing for relatively simple or “tedious” coding tasks: integration, basic UI, test cases, devops, validity checking, configuration, formatting, reports, basic refactoring, etc. Being able to “Program in English” feels like nothing short of magic.

But, the agents struggle when the problem moves into “real” computer science: algorithms, asynchronous designs, threads, race conditions, locking, security, consistency, optimization, scalability, etc. Unguided, the agent will happily write the code (eager to please!), but it’s fragile. Worse, the agent doesn’t know it’s fragile: it happily fixes bug #1 but then introduces new bugs. Fixing those seems to work (for now), but a new bug appears later, or new features trigger cascading new bugs and long debugging and bug-fixing cycles.

This is where experience matters: a seasoned developer will recognize the thrashing, pull back, pinpoint the fragility, and provide specific refactoring guidance to move the agent toward a more robust design & implementation. (I’ll write a follow-up post later on my experience here, with some suggested practices.)

Back to the co-founder’s question: how do we evaluate?

I think the answer (again, today) revolves around: what kind of software is being developed? If it’s relatively simple, tedious, integration-heavy, “forms on a database with reports,” work, junior developers with coding agents can get a lot done. But if it’s more sophisticated than that, you risk having a huge, fragile pile of “AI slop” that looks like it works but breaks when you need it most.