clgen example fails on GPU with NOT NULL constraint failed: training_telemetry.training_loss

[wldhx]

Hi,

Trying to run the clgen demo from master (65643f) gives me intermittent failures of this kind:

source activate tensorflow_p36 # anaconda
git clone https://github.com/ChrisCummins/phd
cd phd
bazel build -c opt //deeplearning/clgen
~/phd/bazel-bin/deeplearning/clgen/clgen --config ~/phd/deeplearning/clgen/tests/data/tiny/config.pbtxt

<snip>
[gpu_scheduler]	INFO	Acquired GPU 0 (Tesla T4)
I0716 08:43:36 gpu_scheduler.py:102] Acquired GPU 0 (Tesla T4)
<snip>
[prof]	INFO	GetTrainingData() in 2ms
I0716 08:47:03 prof.py:181] GetTrainingData() in 2ms
[data_generators]	INFO	Encoded corpus of 757,760 tokens (clipped last 2,542 tokens) in 3 ms.
I0716 08:47:03 data_generators.py:166] Encoded corpus of 757,760 tokens (clipped last 2,542 tokens) in 3 ms.
[data_generators]	INFO	Step shape: X: (64, 64), y: (64, 64).
I0716 08:47:03 data_generators.py:252] Step shape: X: (64, 64), y: (64, 64).
[data_generators]	INFO	Memory: 32 KiB per batch, 6 MiB per epoch, 185 MiB total.
I0716 08:47:03 data_generators.py:261] Memory: 32 KiB per batch, 6 MiB per epoch, 185 MiB total.
[tensorflow_backend]	INFO	Instantiated TensorFlow graph with 286,812 trainable parameters in 488 ms.
I0716 08:47:03 tensorflow_backend.py:235] Instantiated TensorFlow graph with 286,812 trainable parameters in 488 ms.
[tensorflow_backend]	INFO	Restoring checkpoint /tmp/phd/deeplearning/clgen/tiny/model/e549e3374ed531acc1f9c397e7006bda59906d6a/checkpoints/checkpoint-13
I0716 08:47:03 tensorflow_backend.py:383] Restoring checkpoint /tmp/phd/deeplearning/clgen/tiny/model/e549e3374ed531acc1f9c397e7006bda59906d6a/checkpoints/checkpoint-13
[saver]	INFO	Restoring parameters from /tmp/phd/deeplearning/clgen/tiny/model/e549e3374ed531acc1f9c397e7006bda59906d6a/checkpoints/checkpoint-13
I0716 08:47:03 saver.py:1290] Restoring parameters from /tmp/phd/deeplearning/clgen/tiny/model/e549e3374ed531acc1f9c397e7006bda59906d6a/checkpoints/checkpoint-13
[data_generators]	INFO	Encoded corpus of 757,760 tokens (clipped last 2,542 tokens) in 3 ms.
I0716 08:47:03 data_generators.py:166] Encoded corpus of 757,760 tokens (clipped last 2,542 tokens) in 3 ms.
[tensorflow_backend]	INFO	Epoch 14/32:
I0716 08:47:03 tensorflow_backend.py:408] Epoch 14/32:
 54% (100 of 185) |##############################################################################################                                                                                | Elapsed Time: 0:00:05 ETA:   0:00:04[clgen]	ERROR	(sqlite3.IntegrityError) NOT NULL constraint failed: training_telemetry.training_loss
[SQL: INSERT INTO training_telemetry (model_id, timestamp, epoch, step, training_loss, learning_rate, ns_per_batch, pending) VALUES (?, ?, ?, ?, ?, ?, ?, ?)]
[parameters: (1, '2020-07-16 08:47:09.809000', 14, 2505, nan, 0.001987038928589296, 56796.04, 1)]
(Background on this error at: http://sqlalche.me/e/gkpj) (IntegrityError)

  stacktrace:
      #1  /home/ubuntu/phd/bazel-bin/deeplearning/clgen/clgen.runfiles/phd/deeplearning/clgen/clgen.py:181 Train()
      #2  /home/ubuntu/phd/bazel-bin/deeplearning/clgen/clgen.runfiles/phd/deeplearning/clgen/clgen.py:184 Sample()
      #3  /home/ubuntu/phd/bazel-bin/deeplearning/clgen/clgen.runfiles/phd/deeplearning/clgen/clgen.py:406 DoFlagsAction()
      #4  /home/ubuntu/phd/bazel-bin/deeplearning/clgen/clgen.runfiles/phd/deeplearning/clgen/clgen.py:415 main()
      #5  /home/ubuntu/phd/bazel-bin/deeplearning/clgen/clgen.runfiles/phd/deeplearning/clgen/clgen.py:293 RunContext()

Please report bugs at <https://github.com/ChrisCummins/phd/issues>
E0716 08:47:09 clgen.py:260] (sqlite3.IntegrityError) NOT NULL constraint failed: training_telemetry.training_loss
[SQL: INSERT INTO training_telemetry (model_id, timestamp, epoch, step, training_loss, learning_rate, ns_per_batch, pending) VALUES (?, ?, ?, ?, ?, ?, ?, ?)]
[parameters: (1, '2020-07-16 08:47:09.809000', 14, 2505, nan, 0.001987038928589296, 56796.04, 1)]
(Background on this error at: http://sqlalche.me/e/gkpj) (IntegrityError)

  stacktrace:
      #1  /home/ubuntu/phd/bazel-bin/deeplearning/clgen/clgen.runfiles/phd/deeplearning/clgen/clgen.py:181 Train()
      #2  /home/ubuntu/phd/bazel-bin/deeplearning/clgen/clgen.runfiles/phd/deeplearning/clgen/clgen.py:184 Sample()
      #3  /home/ubuntu/phd/bazel-bin/deeplearning/clgen/clgen.runfiles/phd/deeplearning/clgen/clgen.py:406 DoFlagsAction()
      #4  /home/ubuntu/phd/bazel-bin/deeplearning/clgen/clgen.runfiles/phd/deeplearning/clgen/clgen.py:415 main()
      #5  /home/ubuntu/phd/bazel-bin/deeplearning/clgen/clgen.runfiles/phd/deeplearning/clgen/clgen.py:293 RunContext()

Please report bugs at <https://github.com/ChrisCummins/phd/issues>

Here's the whoami, host OS is AWS ami-0abcbc65f89fb220e.

I could not reproduce in chriscummins/phd_build (=> on CPU).