Catching a Chess Cheater Using R and GridDB

Chess is an ancient game played by millions of people everyday. Chess, like many other games, has seen a boom with the enhancement of technology due various playing platforms giving players the chance to interact, learn, and exchange ideas with one another. Nonetheless, the most drastic improvement chess has seen over the last couple of decades is the introduction of the artificial intelligence based models which have become so good that even the best player in the world cannot compete with them.

However, there are two sides to every coin, and in this case computers have given impostors the ability to cheat while playing other humans causing issues of fair play in many online chess tournaments. So, is there anyway cheating can be detected and if so what are the methods to catch a cheater?

In the following sections, we will explore how to build a simple algorithm for detecting blatant cheating.

Before we start with the analysis, let’s firstly start by understanding some chess terminology. Firstly, a centipawn value is a numeric value that a chess engine uses to evaluate a chess position. The standard is that the simplest piece such as the pawn is considered to be worth 100 cp. Similarly, the queen which is the most valuable piece is worth 900 cp.

Secondly, the Elo rating is a metric used in chess to assign each chess player a strength score. For simplicity reasons, one can think of the Elo score as a relative score to the rest of the players in the field and that the higher the value, the better the player is.

Thirdly, a method that can be used to measure the accuracy of a person’s moves is the centipawn loss. This is a strategy to assess how bad a player’s moves were compared to the ones suggested by chess engines. Further, the most commonly used methodology is to compute the average centipawn loss which means how much the player deviated from the actual engine’s best recommendation.

Mathematically, it can be defined as follows:

$$acl =\frac{ \sum\limits_{i=1}^{n} |x – x_i|}{n}$$

where $x$ is the evaluation after following the engine’s move whereas $x_i$ is the actual evaluation after player made a move.

Lastly, we need to define some assumptions before starting the analysis. If a player has an average loss of $0$, then the player has followed the moves as suggested by the engine 100% of the time. Hence, the larger the loss the weaker the player is. For this example, we will consider an average value lower than $15$ as cheating.

Analysis

Now let’s actually get right into our comparison.

Prereqs

Before we get started, you will need to install:

• GridDB

If you install via the package manager, it will come installed with the JDBC connector. If you are installing through other means, please make sure you download the JDBC connector for GridDB: GridDB JDBC.

Setting Up your R Environment

To follow along all you need to do is to install R and Stockfish on your computer. I am using R version 4.2.2 and Stockfish version 14.1. However, other versions should be working fine. The complete code for this project can be found in this repository. Next, load the necessary R packages

library(dplyr)
library(magrittr)
library(stringr)
library(bigchess)
library(RJDBC)

R Analysis

Since the process of engine calculation takes some time, it is always better to store the results in the database so that we won’t need to rerun the algorithm all the time. Besides this, we can build our own chess game database where we can keep information about the games we played. We will be using GridDB because it is lightweight and fast. I am using version 5.0.0 for this tutorial. Moreover, we are using gridstore-jdbc-5.1.0 as the driver for RJDBC which can be downloaded by clicking here. To connect to the database, we use the following R code

driver <- JDBC(
  driverClass = "com.toshiba.mwcloud.gs.sql.Driver",
  classPath = "/usr/share/java/gridstore-jdbc-5.1.0.jar"
)
conn <- dbConnect(driver, "jdbc:gs://127.0.0.1:20001/myCluster/public", "admin", "admin")

Chess games are usually stored in a .pgn format where they store all the moves each player makes with extra information such as the rating of the players, piece colors, and more. We will be using only 4 pgn games in this project, but you can download large chess datasets for active and inactive players in this website. Besides, it only requires that you place the data in the Data folder and by running the cheat_detector.R script all of the statistics and the rest of the pgn information for your large chess database will be calculated. Let’s simplify the tutorial by only taking one match called normal_game.pgn. To read the data we can utilize the package called bigchess.

game <- read.pgn(con = file("normal_game.pgn"), stat.moves = F, extract.moves = 0)
> str(game)
'data.frame':   1 obs. of  9 variables:
 $ Event   : chr "Rated Blitz game"
 $ Site    : chr "https://lichess.org/kOMi8fle"
 $ Date    : chr "2022.10.21"
 $ Round   : chr NA
 $ White   : chr "Krye_Kuzhinieri"
 $ Black   : chr "D1stknightmaster"
 $ Result  : Ord.factor w/ 1 level "1-0": 1
 $ Movetext: chr "1. e4 e5 2. g3 Nc6 3. Bg2 Bc5 4. Ne2 Nf6 5. O-O d6 6. c3 Bg4 7. d3 h6 8. Qc2 O-O 9. Nd2 Ne7 10. Nf3 c6 11. d4 B"| __truncated__
 $ NMoves  : num 51

Moreover, chess games use various notations to write the moves. But Stockfish accepts only the Long Algebraic Notation (LAN) so we need to convert the string.

game_lan <- san2lan(game$Movetext[1])
> game_lan
[1] "e2e4 e7e5 g2g3 b8c6 f1g2 f8c5 g1e2 g8f6 e1g1 d7d6 c2c3 c8g4 d2d3 h7h6 d1c2 e8g8 b1d2 c6e7 d2f3 c7c6 
d3d4 g4f3 g2f3 c5b6 c1e3 e7g6 a1d1 d8c8 a2a4 a7a5 c3c4 f8e8 d4d5 b6e3 f2e3 c6d5 e4d5 e5e4 f3g2 f6g4 c2c3 
c8c5 e2d4 a8c8 b2b3 c5b4 c3b4 a5b4 d1e1 e8e5 d4b5 c8d8 h2h3 g4f6 b5d4 g6e7 g3g4 h6h5 d4f5 e7f5 f1f5 h5g4 
f5e5 d6e5 h3g4 f6g4 g2e4 g7g6 g1g2 f7f5 e4f3 g4f6 e1h1 g8g7 g2f2 e5e4 f3e2 b7b6 h1d1 g6g5 d5d6 f5f4 e3f4 
g5f4 d1d4 f4f3 e2f1 g7f7 f1h3 f7g6 a4a5 b6a5 c4c5 f6e8 d6d7 e8c7 c5c6 g6g5 d4e4 c7d5 e4d4"

Next, let’s calculate the average centipawn loss by getting the score of each value after a move. For instance, after white makes a move, we get the evaluation of the engine and we store the value. In essence, different engine depths return different evaluations; that is, the higher the engine depth the more accurate is the evaluation provided by Stockfish. Here we are using engine_depth of 20.

all_moves <- str_split(lan, " ")[[1]]
for (i in seq_along(all_moves)) {
    notation <- trimws(paste(notation, all_moves[i]), which = "left")
    score <- uci_engine(engine_path) %>%
      uci_position(moves = notation) %>%
      uci_go(depth = engine_depth) %>%
      uci_quit() %>%
      uci_parse(filter = "score")

    score <- score * (-1)
    if (i %% 2 == 0) {
      black_scores <- c(black_scores, score)
    } else {
      white_scores <- c(white_scores, score)
    }
  }

After calculating the scores, we need to take the difference of each move. For example, we get the difference between the evaluation before and after white made a move. This will give us the loss.

black_difference = white_scores + black_scores
white_difference = white_scores + dplyr::lag(black_scores)

Although very rarely, we can still get positive values due to engine not taking all possibilities into account. We will convert values larger than zero, meaning the values where the person outperformed the engine, as zero for simplification.

white_difference = ifelse(white_difference > 0 | is.na(white_difference), 0, white_difference),
black_difference = ifelse(black_difference > 0 | is.na(black_difference), 0, black_difference)

Lastly, we want to get the percentage of best moves for each player. To do that, we can split the data into cuts by specifying the desired ranges. For this tutorial, we will call a move which is in range 0-25 as precise, 25-75 as average, and more than 75 as a bad move.

limit_ranges = c(0, 25, 75, Inf)
limit_names = c("Precise Move", "Average Move", "Bad Move")
white_cuts = cut(white_difference * -1, breaks = limit_ranges, include.lowest = T, labels = limit_names)
black_cuts = cut(black_difference * -1, breaks = limit_ranges, include.lowest = T, labels = limit_names)

Then, the percentages and the loss can be calculated as follows:

# Percentages
white_percentages <- prop.table(table(game_results$white_cuts))
black_percentages <- prop.table(table(game_results$black_cuts))

# Loss
white_average_cp <- abs(mean(game_results$white_difference))
black_average_cp <- abs(mean(game_results$black_difference))

Results

The analysis above has completed successfully and now we have a chess database in our hands. By running SELECT * FROM chess_table_depth_20 you can get various information for each of the games played. Among others, the database stores information such as: event name, location where games were played, date, round, player names, and the centipawn statistics for each player. Here is sample of our database

Now, let’s crunch the data and test the performance of our model. The following is a game I played online on Lichess. I know that I and my opponent haven’t cheated and we can use this to test our model.

We can filter the data score by fetching the game from GridDB by running SELECT * FROM chess_table_depth_20 WHERE Black == 'D1stknightmaster'. This yields the following results:

The second game is a game played between two of the strongest chess players in the world in the Championship Chess Tour Opera Euro Rapid tournament. We know that nobody cheated in this game but we want to know what is the average centipawn loss of the chess elite.

We can filter the data by running the following query SELECT * FROM chess_table_depth_20 WHERE Black == 'Nakamura, Hikaru' from where we get the following scores:

The next match is a game where I played against the computer. In this case we want to detect that black cheated. So, we expect a very low acl.

We filter the data by running SELECT * FROM chess_table_depth_20 WHERE Black == 'lichess AI level 8'. The results are:

Here we can clearly observe that the player using black pieces has cheated since the value of the loss is smaller than the one we set to be acceptable.

The last game I want to analyze together is the infamous one between world champion Mangus Carlsen and the young American prodigy Hans Niemann. This game is in particular significant because Carlsen accused Niemann of cheating. Although a lot has been said on the topic, this report by Chess.com does not believe Hans cheated in this very game. However, since we have build our model, let’s test it. We filter the data by running SELECT * FROM chess_table_depth_20 WHERE Black == 'Niemann, Hans Moke'.

Hence, it can be argued that Carlsen did not have a good day overall and that Niemann’s performance was not extraordinary. However, this technique does not work well with selective cheating. To improve this model, you can find some useful ideas in the concluding part of this article.

Lastly, I want to display how all players compare to one another. Below you can see that the engine clearly outperforms the world champion. However, the biggest difference is not in the fact that it plays always the best moves, but how rarely a simple online engine makes a mistake. Moreover, an amateur player like myself does on average play a bad move 40% of the time.

Conclusion

In this article we focused on detecting blatant cheating in chess. Although the statistical model we created is simple, it achieves good results in cases where players follow most of the engine moves. However, this model does not work well for selective or smart cheating because a good player does not need to copy all the engine moves but a couple of them in critical moments. The following can be done to improve the model:

• Create groups of acl scores for different buckets, i.e: 2000-2100, 2100-2200,…, and compare similar players.
• Create player’s acl scores over multiple games and compare to observe anomalies.
• Optimize acl threshold levels. We used 15 in this experiment, but the values can change. A smaller value catches a cheater with higher confidence, whereas a larger value could cause false positives (accusing a person for cheating when in fact they did not)

To conclude, enjoy competing with others using your own strength and avoid cheating because you cannot run away from statistics :).

If you have any questions about the blog, please create a Stack Overflow post here https://stackoverflow.com/questions/ask?tags=griddb .
Make sure that you use the “griddb” tag so our engineers can quickly reply to your questions.