I'm fluent in R but a total beginner with deep learning. I'm trying to splice together two bits of code from the Chollet et al. Deep Learning with R book (2d ed), namely one for a CNN model for the MNIST data set (pp. 221ff) and the other for data augmentation on images (pp. 241ff). The latter example involves the dogs-vs-cats database from Kaggle that I don't want to have to bother signing up to get, which is why I'm trying to figure out how to do data augmentation with MNIST instead.

The problem is that the format for the MNIST images doesn't match that for the dogs and cats, and I don't know enough about image formats, tensors, or maybe even how arrays and rasters work in R to fix the problem. The MNIST images are grayscale while the dogs and cats are in color, but I'm not sure how that can be a fatal problem.

If I can get my code to run at all, the images are modified the same incorrect ways. Some garbled images involve heights vs. widths while others involve mixing up one image with another, suggesting that the fundamental problem is a mismatch in how different functions interpret the dimensions. The same is suggested by error messages complaining about dimensions. But I still can't make it work.

Here is what I've tried:

library(keras)

mnist <- dataset_mnist()
images = mnist$train$x
mnist$train$y[1:2] # 5 0  - what the first two images are supposed to be

data_augmentation <- keras_model_sequential() %>%  # Insert/delete # to try each transformation separately
 # layer_random_flip("horizontal")    # Flips vertically instead of horizontally ("5")
 # layer_random_flip("vertical")      # Shows wrong image without any change (some "5" & some "0")
 layer_random_rotation(0.1)         # Garbles all images completely (maybe trying to merge "5" & "0"?)
 # layer_random_zoom(0.2)             # Only expands vertically
 # layer_random_brightness(0.2)       # Works OK?
 # layer_random_contrast(0.2)         # Works OK?
 # layer_random_height(0.2)           # Overlaps images
 # layer_random_width(0.2)            # Adjusts height instead of width
 # layer_random_translation(0.2,0.2)  # Adjusts vertically but not horizontally, while overlapping images ("5" & "0")

Try 4D tensors

images = mnist$train$x %>%
     array_reshape(c(60000, 28, 28, 1)) # Changing order parameter from "C" to "F" doesn't help

par(mfrow = c(3, 3), mar = rep(.5, 4))
image <- images[1, , , ]
plot(as.raster(as.array(image), max = 255))
for (i in 2:9) {
 augmented_images <- data_augmentation(images[1:2, , , ]) # Focus just on "5" & "0"
 # augmented_image <- augmented_images[1, , , ]  # Wrong number of dimensions
 augmented_image <- augmented_images[1, , ] # Must be 3D, not 4D - why?
 plot(as.raster(as.array(augmented_image), max = 255))
}

Try 3D tensors

images = mnist$train$x %>%
     array_reshape(c(60000, 28, 28)) # Changing order parameter from "C" to "F" doesn't help

par(mfrow = c(3, 3), mar = rep(.5, 4))
image <- images[1, , ]
plot(as.raster(as.array(image), max = 255))
for (i in 2:9) {
 augmented_images <- data_augmentation(images[1:2, , ]) # Focus just on "5" & "0"
 augmented_image <- augmented_images[1, , ] # Still must be 3D, not 3D - why?
 plot(as.raster(as.array(augmented_image), max = 255))
}

When the code doesn't crash due to having the wrong number of dimensions, the image modifications are mostly wrong, as with this attempt to rotate a "5":

these are not rotated 5s