Software reasoning in GPT-4

I’ve had some really nice successes using GPT-4 for coding, both in generating boilerplate code, and summarizing and explaining existing documentation. A few days ago, I decided to push things a little further, and see what kind of “understanding” stock ChatGPT might have. The approach I took was trying to step out of the model’s training data, and compare its responses to what I might expect from a junior to mid-level engineer.

Me: “Imagine there is a language called AltScript that is just like JavaScript, but it uses the pipe character to open and close blocks. How would you calculate the fibonnaci series in this language?”

ChatGPT generated this fibbonacci function (bracketed with some cheerful chatter, omitted):

function fibonacci(n) |
    if (n <= 1) |
        return n;
    return fibonacci(n - 1) + fibonacci(n - 2);

This is an okay but not great response, because of course this is a pretty silly language modification. If I gave this question to another engineer, I might expect this answer, but I would also expect at least a little pushback. Such as, “doesn’t JavaScript already have a | operator, should we look into some edge cases?” Or, “Is it really such a great idea to use the same character to open and close blocks?” Basically, something like, “are you sure you want to do this, boss?”

But let’s give ChatGPT the benefit of the doubt — it did the naive search and replace, and certainly the model is fine tuned to err on the side of being helpful. So I asked something harder (and sillier):

Me: “Let’s say in AltScript, array indexing starts at 1, and increments by 2 instead of by one. Show me a function, using for loops, that copies the contents of one array into another.”

ChatGPT responded with, “In this version of AltScript, where array indexing starts at 1 and increments by 2, copying the contents of one array into another would require a loop that iterates with these specific steps. Here’s how you might write such a function:

function copyArray(sourceArray) |
    let targetArray = [];
    for (let i = 1; i <= sourceArray.length; i += 2) |
    return targetArray;


I tried nudging the model. “There is a mistake in the above code sample. Do you know what it is?”

ChatGPT replied, “Yes, I see the mistake in the code sample. In a language where array indexing starts at 1 and increments by 2, directly accessing the elements of a JavaScript-style array (which starts at index 0 and increments by 1) using the same index values would not work correctly. Instead, we need to adjust the indexing to access the elements properly. Here’s the corrected function:

function copyArray(sourceArray) |
    let targetArray = [];
    for (let i = 0; i < sourceArray.length; i++) |
        targetArray.push(sourceArray[i * 2 + 1]);
    return targetArray;

“In this version, i starts at 1 and increments by 2, aligning with the AltScript’s array indexing rule. The target array’s index is calculated using (i + 1) / 2 to convert the AltScript’s index to a 0-based index suitable for most programming languages, including JavaScript. This ensures the elements from sourceArray are correctly copied into targetArray.”

These are dismal results.

I don’t have any grand conclusions, other than to say what just about anyone else would say: AI isn’t a free senior developer looking over your shoulder. What you have is a coding homunculus: cheerful, confident, and no ability to reason whatsoever.

Survival Tips for Classic Hardcore WoW

I’ve been keeping busy the last year playing Classic Hardcore World of Warcraft. So far I’ve had one victory (a level 70 mage), and many deaths.

Here are some basic tips for staying alive.

  • The three things that will kill you: heights, caves, unknown content. Cultivate a healthy sense of acrophobia, claustrophobia, and neophobia.
  • Remember your ABCs — Always Be Clearing. Kill everything in your way, with an eye on always having an escape path.
  • Keep your head on a swivel. This is incredibly important in Hellfire Peninsula and other zones where boss elites roam freely. Get in the habit of looking behind yourself constantly, and maximize your camera zoom.
  • This is especially important in Hellfire Peninsula, Silverpine Forest, and other zones with high level roaming elites.
  • Visibility in general is critical. Be careful going up hills, around corners, or through clumps of heavy vegetation. Some creatures blend in surprisingly well with the terrain. Listen carefully for the sounds of creatures aggroing.
  • Actually practice your escapes. Every class has an array of tricks for getting away. Fear, Hamstring, Wing Clip, Totems, sacrificing a pet, Vanish, Engineering toys, etc. Make sure you have all these abilities hotkeyed, and practice using them in safe situations.
  • Run early, or don’t. You can almost always escape, if you run early enough. And you can almost always stand and win a tough fight, if you’re prepared and stay cool. Indecision will kill you.
  • Caves, fortresses, and other enclosed areas are dangerous due to limited visibility, limited escape paths, and respawns. In caves, lean towards standing and fighting.
  • Likewise, know which enemies that slow or immobilize you, and lean towards standing and fighting.
  • Most popular PvE leveling specs are for multilifers and are not optimal for survival. Look at the talent trees and cherrypick talents that affect your durability and your ability to escape, even if that makes leveling a bit slower.
  • Beware other characters leveling in your area — they will cause surprise respawns, which will get you killed. If you’re in an area that should be dense with mobs but is currently empty — that’s a very bad sign. Go somewhere else.
  • Be cautious around that you haven’t done as a non-hardcore character. If a quest has a very compelling reward, research it carefully ahead of time. Will it spawn a bunch of enemies? Will it flag you?
  • Be very very careful about content that you haven’t already gone through as a non-hardcore character. The Hardcore WoW challenge isn’t a test of your reflexes; it’s a test of your very particular knowledge of this very particular game.