I’ve had some really nice successes using GPT-4 for coding, both in generating boilerplate code, and summarizing and explaining existing documentation. A few days ago, I decided to push things a little further, and see what kind of “understanding” stock ChatGPT might have. The approach I took was trying to step out of the model’s training data, and compare its responses to what I might expect from a junior to mid-level engineer.<\/p>\n\n\n\n
Me: “Imagine there is a language called AltScript that is just like JavaScript, but it uses the pipe character to open and close blocks. How would you calculate the fibonnaci series in this language?”<\/p>\n\n\n\n
ChatGPT generated this fibbonacci function (bracketed with some cheerful chatter, omitted):<\/p>\n\n\n\n
function fibonacci(n) |\n if (n <= 1) |\n return n;\n |\n return fibonacci(n - 1) + fibonacci(n - 2);\n|<\/code><\/pre>\n\n\n\nThis is an okay but not great<\/em> response, because of course this is a pretty silly language modification. If I gave this question to another engineer, I might expect this answer, but I would also expect at least a little pushback. Such as, “doesn’t JavaScript already have a |<\/code> operator, should we look into some edge cases?” Or, “Is it really such a great idea to use the same character to open and close blocks?” Basically, something like, “are you sure you want to do this, boss?”<\/p>\n\n\n\nBut let’s give ChatGPT the benefit of the doubt — it did the naive search and replace, and certainly the model is fine tuned to err on the side of being helpful. So I asked something harder (and sillier):<\/p>\n\n\n\n
Me: “Let\u2019s say in AltScript, array indexing starts at 1, and increments by 2 instead of by one. Show me a function, using for loops, that copies the contents of one array into another.”<\/p>\n\n\n\n
ChatGPT responded with, “In this version of AltScript, where array indexing starts at 1 and increments by 2, copying the contents of one array into another would require a loop that iterates with these specific steps. Here’s how you might write such a function:<\/em>“<\/p>\n\n\n\nfunction copyArray(sourceArray) |\n let targetArray = [];\n for (let i = 1; i <= sourceArray.length; i += 2) |\n targetArray.push(sourceArray[i]);\n |\n return targetArray;\n|<\/code><\/pre>\n\n\n\nYikes! <\/p>\n\n\n\n
I tried nudging the model. “There is a mistake in the above code sample. Do you know what it is?” <\/p>\n\n\n\n
ChatGPT replied, “Yes, I see the mistake in the code sample. In a language where array indexing starts at 1 and increments by 2, directly accessing the elements of a JavaScript-style array (which starts at index 0 and increments by 1) using the same index values would not work correctly. Instead, we need to adjust the indexing to access the elements properly. Here’s the corrected function:<\/em><\/p>\n\n\n\nfunction copyArray(sourceArray) |\n let targetArray = [];\n for (let i = 0; i < sourceArray.length; i++) |\n targetArray.push(sourceArray[i * 2 + 1]);\n |\n return targetArray;\n|<\/code><\/pre>\n\n\n\n“In this version, i<\/code> starts at 1 and increments by 2, aligning with the AltScript’s array indexing rule. The target array’s index is calculated using (i + 1) \/ 2<\/code> to convert the AltScript’s index to a 0-based index suitable for most programming languages, including JavaScript. This ensures the elements from sourceArray<\/code> are correctly copied into targetArray<\/code>.”<\/em><\/p>\n\n\n\nThese are dismal results. <\/p>\n\n\n\n
I don’t have any grand conclusions, other than to say what just about anyone else would say: AI isn’t a free senior developer looking over your shoulder. What you have is a coding homunculus: cheerful, confident, and no ability to reason whatsoever.<\/p>\n","protected":false},"excerpt":{"rendered":"
I’ve had some really nice successes using GPT-4 for coding, both in generating boilerplate code, and summarizing and explaining existing documentation. A few days ago, I decided to push things a little further, and see what kind of “understanding” stock ChatGPT might have. The approach I took was trying to step out of the model’s […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-994","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.goer.org\/Journal\/wp-json\/wp\/v2\/posts\/994","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.goer.org\/Journal\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.goer.org\/Journal\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.goer.org\/Journal\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.goer.org\/Journal\/wp-json\/wp\/v2\/comments?post=994"}],"version-history":[{"count":1,"href":"https:\/\/www.goer.org\/Journal\/wp-json\/wp\/v2\/posts\/994\/revisions"}],"predecessor-version":[{"id":995,"href":"https:\/\/www.goer.org\/Journal\/wp-json\/wp\/v2\/posts\/994\/revisions\/995"}],"wp:attachment":[{"href":"https:\/\/www.goer.org\/Journal\/wp-json\/wp\/v2\/media?parent=994"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.goer.org\/Journal\/wp-json\/wp\/v2\/categories?post=994"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.goer.org\/Journal\/wp-json\/wp\/v2\/tags?post=994"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}