stress testing gpt 5.6

After following the Anthropic Fable/Mythos fiasco and OpenAI strategically announcing GPT 5.6 Sol with same-to-better benchmark outcomes, I decided to give it a spin. I last tried.. GPT 5.5? with goblin prompting which was fun, but I realized a lot of the commits did not do what I wnted it to do and I often ended up rewriting it with opus anyway. Testing out GPT 5.6 Sol, I made an ambitious but somewhat doable task of reading the existing lifelab written in Typescript/React embeded in tauri desktop app, and turning it into a native macos app just to see what happens. I set a "/goal" with some basic instructions and preferences (read the original app, make a list of features, iterate until you hit parity) and just unleashed it on my mac over 4 days.

I was blown away!

1. The model seems really smart in workflows. It would open the webapp in browser, and render the website and take a screenshot to compare the design. It could convincingly describe the screenshots (monospaced font, org styling) and turn it into action points

2. Over 3 days, it made 120 commits and ran 750 subagents, it wrote 180k lines of code and with a testing harness

3. Using computer control skill, it can actually test and play with the app!

4. I only ran out of weekly usage on the last day, after continuously churning tokens. I had like 4 usage resets given to me for unknown reasons and it feels it can run non stop on ultra reasoning. I remember telling my friend it is incredible value for the cost, because I was able to hit Claude rate limits on the $200 subscription.

So yeah, OpenAI can run basically nonstop and puke out code and subagents at a rate I have never experienced before and it definitely feels more tokens = more intelligence and more things done. Unfortunately, when I checked the resulting code, it was all slop of the absolute worst kind.

While I think highly of my vibecoded project in terms of thoughts and ideas I put in, 180k lines for a TODO app was a bit suspect. looking deeper, it wrote 110k lines of Objective-C code that contained 4 duplicate state machines for do-undo in text editing. The 70k tests code also contained 10 standalone binaries (I think it did not want to do XCTests in Xcode, and decided to test it by just making mini apps that only import a subset of modules).

When I launch the app, I would get a familar style to the web app superficially, but the code itself looked absolutely revolting. It ran dozens of GPUs for 3 days only to produce unshippable code. So I got way more tokens, but was it actually worth it? It seemed to waste both my and it's own time.

I was not pleased. I had a very reliable workflow going with Opus, and GPT gave me slop. I will definitely admit I might have been using it wrong and might have had too high expectations. For Opus, I am more hands on with separate sessions for planning (with mockups) to write a spec, and then I have a fresh session execute the final spec. I LGTM the resulting PR and I am mostly (about 80-90% of the time) satisfied with the result and basically never revert the code.

For Codex/GPT, I let it run loose, which arguably is the opposite of what I should have been doing. According to reddit vibe coders, opus is better for open ended planning, GPT is better for well defined task execution. I might have been using GPT completely wrong so I might not be right to critize it, but as a amateur coder I tried to "one shot" an app and definitely did not expect the poor quality of the outcome. Some might say it does not matter, but if this code gets published and ingested by the next LLM, we might se incomprehensible amounts of slop in GPT 7 when there is no good code left to be copied.

My humble recommendation would be to have a human-reviewable specs and implement it in milestones. I don't think it can solo-write a good macos app today even with an existing app as template. I suspect a lot of the "X one shot my idea" is just a result of the idea being fairly generic and contained in the training data.

Another argument is, if the app superficially does all you want, is that not enough? It should be impressive that GPT can make a look-a-like macos in pure objective-C (barely comprehensible to me) without ever directly launching XCode to check the UI, and after enough token churning it has the same functionality as the original web app. However, the code it generated definitely resembles individual agent traces where each agent only has the context of its task and doesn't look at the overall app architecture. Thus we end up with 5 duplicate, high quality state machines implementations and no agent just stops to think "hmm, this can be unified". I used to think the claude PR reviews or GPT PR reviews can catch this but I did not have much success with it running 100% without me in the loop.

In a post LLM era, the code written by GPT 5.6 is unmaintainable, incomprehensible and not worth learning from. Maybe future developers will never read again, but we must admit most of the current coding capability IS learned from previously painstakingly handwritten code (which in itself contains a lot of slop). I was convinced by my friend that the next LLM will overwrite and desloppify the code of the previous LLM, but my experience seems to be the opposite. The next LLM is more capable to hack through the requirements to create more sloppy code than before, with me trusting it more despite the outcomes getting worse.

I might not be able to do a leetcode medium without practice, but I can at least tell badly designed software. Maybe this won't be true next year.