ChatGPT tries to pull a ‘Kasparov’ on me
I have redone the historical experiment of man playing chess against the machine. There are some easy to notice differences between myself and Kasparov. However, it’s worth noting that neither my opponent has chess in their daily routine, so it was an evenly matched game between occasional amateurs.
ChatGPT only interprets text, so I played blind, letter by letter. For context, this linguistic model was trained to predict the next word* for a given context. This simple criterion can encompass a multitude of tasks, which can be formulated as a “completion” problem, in which the next most suitable word is chosen iteratively. In the case of my experiment, after the word string “1. e4”, which represents the movement of the pawn in front of the king, it is very likely that “e5” will come next, the mirror response of the other player’s pawn. And so on.
I expected the openings to proceed quite naturally, being standard moves and therefore well documented on the internet (which represents 60% of the model’s learning data). At the 6th move, I exchanged knights and at 9th, I checked, sacrificing a pawn just to see what it would do. It figured out that the king had to move and captured the pawn with it. I said “hmm ok, interesting”. And I would be lying if I didn’t admit that from there on, I started to play a bit more defensively, feeling a bit of the fear that maybe I had underestimated my opponent.
That was pretty much the end, because on the 11th move, it responded with an invalid move, probably because I had broken the chain of related moves. And that’s the conclusion I expected, although I’m still impressed that it only came after 10 correct responses. Unlike the program that beat Kasparov, GPT has no concrete model for the chessboard. It has no mechanism to model the constraints of the rules. It’s not that it lost the game because it made a bad move — it’s that it made an invalid move, because it doesn’t actually understand how to play.
But it’s very good at spotting patterns in word sequences. And one of these patterns might be that after “What’s the capital city of Romania?”, “Bucharest” should come (probably 20% uncertainty with “Budapest”) or that after “pawn e4” comes “pawn e5”. Or, for even this text right now, something from my own expression patterns. It doesn’t matter. It’s rote learning. And that’s not a problem, because we are just so realizing how much of the world we live in can be just mimicked.
So yes, it’s very good that schools are concerned with plagiarized essays. It’s very good that “content creators” are sweating. It’s time for us to “upgrade” ourselves beyond superficiality. I’m sorry to announce, but bullshit is already being made faster, cheaper and more convincing.
I won a chess game with a machine that wasn’t made to play chess. Gg. The truth is that the machine won, from the moment I sat down at the table.
If you want to try it yourself:
https://www.chess.com/analysis allows for easily translating the moves into Portable Game Notation (PGN)
and https://chat.openai.com/chat, which you probably already know
