Learning Image to Image Translation with Generative Adversarial Networks

import numpy as np
import cv2

SPEED = 25
cap = cv2.VideoCapture('./source/beach/beachvideo.mp4')

capture_run = np.random.randint(1000)
totalFrames = 0

def capture_square(frame, square_size, capture_run, totalFrames):
     startX = np.random.randint( int((w-square_size)) )
     startY = np.random.randint( int((h-square_size)) )
     capture = frame[startY:startY+square_size, startX:startX+square_size]
     cv2.imwrite("./captures/cap{0}_f{1}.jpg".format(capture_run, totalFrames), 
     capture, [int(cv2.IMWRITE_JPEG_QUALITY), 90])
     totalFrames = totalFrames+1

while(cap.isOpened()):
     ret, frame = cap.read()
     h, w, c = frame.shape
     cv2.imshow('frame', frame)

     if (h < 600) or (w < 600):
          print("width or height of video must be at least 600px")
          break

     key = cv2.waitKey(SPEED)

     # CAPTURE FRAMES WHEN SPACE IS PRESSED
     if key == 32:
          square_size = 600
          capture_square(frame, square_size, capture_run, totalFrames)
          totalFrames = totalFrames + 1

     # QUIT ON 'Q'
     if key == ord('q'):
          break

cap.release()
cv2.destroyAllWindows()

For our task, this approach didn’t deliver a better quality result than the superior images we had access to from Getty Images. However, it provides an interesting avenue for additional CycleGAN projects and we may return to it as a training data source in the future.

Once we had a stable model and good set of training data image sets, we began running various training sessions overnight leveraging both local machines and Azure VM instances to compare various runs concurrently.

During the run, we would periodically check with TensorBoard to see how well our model was doing. TensorBoard proved invaluable for monitoring our model. However, it seemed to struggle with graphing the model itself, so CycleGANs might be a bit much for it:

We set up TensorBoard variables that allowed us to monitor not only our loss functions, learning rate decay, and other important parameters but also provided us with glimpses into the images being generated by our model. During training these varied from bizarre to strangely beautiful before settling down to results that looked closer to our goal. Here’s an example of an exotic sunset from one of our runs:

Even if we could never generate photorealistic results, we could at least generate science fiction book covers! Here is a cherry-picked sample of our results at the end of the hackfest, showing that in many cases we approached our goal.

While some of the results were very impressive, there are further optimizations we would like to explore in the future. Additional GANs can be supplemented to help clean up various artifacts created in the process, as well as to allow us to have better resolution while improving the output size. There are also traditional post-processing techniques that can be applied which don’t require deep learning and can improve the aesthetic quality and color balance of the images, which would be valuable if this approach eventually moves toward production quality.