Problem with TF2 version of Deep CFR using Keras 3

[lanctot]

Run on Ubuntu 24.04 using Python 3.12 and Keras 3.1.1:

(venv) lanctot@nitro-exp:~/open_spiel/open_spiel/python/algorithms$ python deep_cfr_tf2_test.py 
2024-04-13 08:03:52.363130: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-13 08:03:52.393130: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-13 08:03:52.943457: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-04-13 08:03:53.659446: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-04-13 08:03:53.659691: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Running tests under Python 3.12.2: /home/lanctot/venv/bin/python
[ RUN      ] DeepCFRTest.test_deep_cfr_runs0 ('leduc_poker')
/home/lanctot/venv/lib/python3.12/site-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead.
  warnings.warn(
/home/lanctot/venv/lib/python3.12/site-packages/keras/src/layers/layer.py:1248: UserWarning: Layer 'advantage_network_2' looks like it has unbuilt state, but Keras is not able to trace the layer `call()` in order to build it automatically. Possible causes:
1. The `call()` method of your layer may be crashing. Try to `__call__()` the layer eagerly on some test input first to see if it works. E.g. `x = np.random.random((3, 4)); y = layer(x)`
2. If the `call()` method is correct, then you may need to implement the `def build(self, input_shape)` method on your layer. It should create all variables used by the layer (e.g. by calling `layer.build()` on all its children layers).
Exception encountered: ''tf.function only supports singleton tf.Variables created on the first call. Make sure the tf.Variable is only created once or created outside tf.function. See https://www.tensorflow.org/guide/function#creating_tfvariables for more information.''
  warnings.warn(
[  FAILED  ] DeepCFRTest.test_deep_cfr_runs0 ('leduc_poker')
[ RUN      ] DeepCFRTest.test_deep_cfr_runs1 ('kuhn_poker')
/home/lanctot/venv/lib/python3.12/site-packages/keras/src/layers/layer.py:1248: UserWarning: Layer 'advantage_network_7' looks like it has unbuilt state, but Keras is not able to trace the layer `call()` in order to build it automatically. Possible causes:
1. The `call()` method of your layer may be crashing. Try to `__call__()` the layer eagerly on some test input first to see if it works. E.g. `x = np.random.random((3, 4)); y = layer(x)`
2. If the `call()` method is correct, then you may need to implement the `def build(self, input_shape)` method on your layer. It should create all variables used by the layer (e.g. by calling `layer.build()` on all its children layers).
Exception encountered: ''tf.function only supports singleton tf.Variables created on the first call. Make sure the tf.Variable is only created once or created outside tf.function. See https://www.tensorflow.org/guide/function#creating_tfvariables for more information.''
  warnings.warn(
[  FAILED  ] DeepCFRTest.test_deep_cfr_runs1 ('kuhn_poker')
[ RUN      ] DeepCFRTest.test_deep_cfr_runs2 ('liars_dice')
/home/lanctot/venv/lib/python3.12/site-packages/keras/src/layers/layer.py:1248: UserWarning: Layer 'advantage_network_12' looks like it has unbuilt state, but Keras is not able to trace the layer `call()` in order to build it automatically. Possible causes:
1. The `call()` method of your layer may be crashing. Try to `__call__()` the layer eagerly on some test input first to see if it works. E.g. `x = np.random.random((3, 4)); y = layer(x)`
2. If the `call()` method is correct, then you may need to implement the `def build(self, input_shape)` method on your layer. It should create all variables used by the layer (e.g. by calling `layer.build()` on all its children layers).
Exception encountered: ''tf.function only supports singleton tf.Variables created on the first call. Make sure the tf.Variable is only created once or created outside tf.function. See https://www.tensorflow.org/guide/function#creating_tfvariables for more information.''
  warnings.warn(
WARNING:tensorflow:5 out of the last 32 calls to <function DeepCFRSolver._get_matched_regrets at 0x7c11808a9a80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
W0413 08:03:54.968961 136417104351360 polymorphic_function.py:157] 5 out of the last 32 calls to <function DeepCFRSolver._get_matched_regrets at 0x7c11808a9a80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[  FAILED  ] DeepCFRTest.test_deep_cfr_runs2 ('liars_dice')
[ RUN      ] DeepCFRTest.test_matching_pennies_3p
/home/lanctot/venv/lib/python3.12/site-packages/keras/src/layers/layer.py:1248: UserWarning: Layer 'advantage_network_17' looks like it has unbuilt state, but Keras is not able to trace the layer `call()` in order to build it automatically. Possible causes:
1. The `call()` method of your layer may be crashing. Try to `__call__()` the layer eagerly on some test input first to see if it works. E.g. `x = np.random.random((3, 4)); y = layer(x)`
2. If the `call()` method is correct, then you may need to implement the `def build(self, input_shape)` method on your layer. It should create all variables used by the layer (e.g. by calling `layer.build()` on all its children layers).
Exception encountered: ''tf.function only supports singleton tf.Variables created on the first call. Make sure the tf.Variable is only created once or created outside tf.function. See https://www.tensorflow.org/guide/function#creating_tfvariables for more information.''
  warnings.warn(
/home/lanctot/venv/lib/python3.12/site-packages/keras/src/layers/layer.py:1248: UserWarning: Layer 'advantage_network_19' looks like it has unbuilt state, but Keras is not able to trace the layer `call()` in order to build it automatically. Possible causes:
1. The `call()` method of your layer may be crashing. Try to `__call__()` the layer eagerly on some test input first to see if it works. E.g. `x = np.random.random((3, 4)); y = layer(x)`
2. If the `call()` method is correct, then you may need to implement the `def build(self, input_shape)` method on your layer. It should create all variables used by the layer (e.g. by calling `layer.build()` on all its children layers).
Exception encountered: ''tf.function only supports singleton tf.Variables created on the first call. Make sure the tf.Variable is only created once or created outside tf.function. See https://www.tensorflow.org/guide/function#creating_tfvariables for more information.''
  warnings.warn(
[  FAILED  ] DeepCFRTest.test_matching_pennies_3p
======================================================================
ERROR: test_deep_cfr_runs0 ('leduc_poker') (__main__.DeepCFRTest)
DeepCFRTest.test_deep_cfr_runs0 ('leduc_poker')
test_deep_cfr_runs('leduc_poker')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lanctot/venv/lib/python3.12/site-packages/absl/testing/parameterized.py", line 322, in bound_param_test
    return test_method(self, testcase_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2_test.py", line 41, in test_deep_cfr_runs
    deep_cfr_solver.solve()
  File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2.py", line 439, in solve
    advantage_losses[p].append(self._learn_advantage_network(p))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2.py", line 686, in _learn_advantage_network
    main_loss = self._advantage_train_step[player](*d, tfit)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lanctot/venv/lib/python3.12/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/tmp/__autograph_generated_filezdqatnm5.py", line 18, in tf__train_step
    ag__.converted_call(ag__.ld(self)._optimizer_advantages[ag__.ld(player)].apply_gradients, (ag__.converted_call(ag__.ld(zip), (ag__.ld(gradients), ag__.ld(model).trainable_variables), None, fscope),), None, fscope)
  File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 269, in apply_gradients
    self.apply(grads, trainable_variables)
  File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 314, in apply
    grads, trainable_variables = self._filter_empty_gradients(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 571, in _filter_empty_gradients
    raise ValueError("No gradients provided for any variable.")
ValueError: in user code:

    File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2.py", line 663, in train_step  *
        self._optimizer_advantages[player].apply_gradients(
    File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 269, in apply_gradients  **
        self.apply(grads, trainable_variables)
    File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 314, in apply
        grads, trainable_variables = self._filter_empty_gradients(
    File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 571, in _filter_empty_gradients
        raise ValueError("No gradients provided for any variable.")

    ValueError: No gradients provided for any variable.


======================================================================
ERROR: test_deep_cfr_runs1 ('kuhn_poker') (__main__.DeepCFRTest)
DeepCFRTest.test_deep_cfr_runs1 ('kuhn_poker')
test_deep_cfr_runs('kuhn_poker')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lanctot/venv/lib/python3.12/site-packages/absl/testing/parameterized.py", line 322, in bound_param_test
    return test_method(self, testcase_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2_test.py", line 41, in test_deep_cfr_runs
    deep_cfr_solver.solve()
  File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2.py", line 439, in solve
    advantage_losses[p].append(self._learn_advantage_network(p))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2.py", line 686, in _learn_advantage_network
    main_loss = self._advantage_train_step[player](*d, tfit)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lanctot/venv/lib/python3.12/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/tmp/__autograph_generated_filezdqatnm5.py", line 18, in tf__train_step
    ag__.converted_call(ag__.ld(self)._optimizer_advantages[ag__.ld(player)].apply_gradients, (ag__.converted_call(ag__.ld(zip), (ag__.ld(gradients), ag__.ld(model).trainable_variables), None, fscope),), None, fscope)
  File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 269, in apply_gradients
    self.apply(grads, trainable_variables)
  File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 314, in apply
    grads, trainable_variables = self._filter_empty_gradients(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 571, in _filter_empty_gradients
    raise ValueError("No gradients provided for any variable.")
ValueError: in user code:

    File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2.py", line 663, in train_step  *
        self._optimizer_advantages[player].apply_gradients(
    File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 269, in apply_gradients  **
        self.apply(grads, trainable_variables)
    File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 314, in apply
        grads, trainable_variables = self._filter_empty_gradients(
    File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 571, in _filter_empty_gradients
        raise ValueError("No gradients provided for any variable.")

    ValueError: No gradients provided for any variable.


======================================================================
ERROR: test_deep_cfr_runs2 ('liars_dice') (__main__.DeepCFRTest)
DeepCFRTest.test_deep_cfr_runs2 ('liars_dice')
test_deep_cfr_runs('liars_dice')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lanctot/venv/lib/python3.12/site-packages/absl/testing/parameterized.py", line 322, in bound_param_test
    return test_method(self, testcase_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2_test.py", line 41, in test_deep_cfr_runs
    deep_cfr_solver.solve()
  File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2.py", line 439, in solve
    advantage_losses[p].append(self._learn_advantage_network(p))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2.py", line 686, in _learn_advantage_network
    main_loss = self._advantage_train_step[player](*d, tfit)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lanctot/venv/lib/python3.12/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/tmp/__autograph_generated_filezdqatnm5.py", line 18, in tf__train_step
    ag__.converted_call(ag__.ld(self)._optimizer_advantages[ag__.ld(player)].apply_gradients, (ag__.converted_call(ag__.ld(zip), (ag__.ld(gradients), ag__.ld(model).trainable_variables), None, fscope),), None, fscope)
  File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 269, in apply_gradients
    self.apply(grads, trainable_variables)
  File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 314, in apply
    grads, trainable_variables = self._filter_empty_gradients(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 571, in _filter_empty_gradients
    raise ValueError("No gradients provided for any variable.")
ValueError: in user code:

    File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2.py", line 663, in train_step  *
        self._optimizer_advantages[player].apply_gradients(
    File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 269, in apply_gradients  **
        self.apply(grads, trainable_variables)
    File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 314, in apply
        grads, trainable_variables = self._filter_empty_gradients(
    File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 571, in _filter_empty_gradients
        raise ValueError("No gradients provided for any variable.")

    ValueError: No gradients provided for any variable.


======================================================================
ERROR: test_matching_pennies_3p (__main__.DeepCFRTest)
DeepCFRTest.test_matching_pennies_3p
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2_test.py", line 57, in test_matching_pennies_3p
    deep_cfr_solver.solve()
  File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2.py", line 439, in solve
    advantage_losses[p].append(self._learn_advantage_network(p))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2.py", line 686, in _learn_advantage_network
    main_loss = self._advantage_train_step[player](*d, tfit)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lanctot/venv/lib/python3.12/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/tmp/__autograph_generated_filezdqatnm5.py", line 18, in tf__train_step
    ag__.converted_call(ag__.ld(self)._optimizer_advantages[ag__.ld(player)].apply_gradients, (ag__.converted_call(ag__.ld(zip), (ag__.ld(gradients), ag__.ld(model).trainable_variables), None, fscope),), None, fscope)
  File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 269, in apply_gradients
    self.apply(grads, trainable_variables)
  File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 314, in apply
    grads, trainable_variables = self._filter_empty_gradients(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 571, in _filter_empty_gradients
    raise ValueError("No gradients provided for any variable.")
ValueError: in user code:

    File "/home/lanctot/open_spiel/open_spiel/python/algorithms/deep_cfr_tf2.py", line 663, in train_step  *
        self._optimizer_advantages[player].apply_gradients(
    File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 269, in apply_gradients  **
        self.apply(grads, trainable_variables)
    File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 314, in apply
        grads, trainable_variables = self._filter_empty_gradients(
    File "/home/lanctot/venv/lib/python3.12/site-packages/keras/src/optimizers/base_optimizer.py", line 571, in _filter_empty_gradients
        raise ValueError("No gradients provided for any variable.")

    ValueError: No gradients provided for any variable.


----------------------------------------------------------------------
Ran 4 tests in 2.216s

FAILED (errors=4)
(venv) lanctot@nitro-exp:~/open_spiel/open_spiel/python/algorithms$ 

[lanctot]

Tagging original author @StochasticEntropy , but we could use some community help to port this algorithm to Keras 3.