Thank you for your great job. I want to train something in gpu , but got a assert

Okay, I understand. You also need to set use_tpu=False</code

training in gpu about ddsp HOT 3 CLOSED

magenta commented on August 16, 2024

training in gpu

from ddsp.

Comments (3)

jesseengel commented on August 16, 2024 1

Okay, I understand. You also need to set use_tpu=False in get_train_op.

train_op = ddsp.training.train_util.get_train_op(loss, 
                                                 learning_rate=learning_rate,
                                                 use_tpu=False)

I'll put in a PR to update the notebook to make this clear and set this to default as that's the more common option.

from ddsp.

jesseengel commented on August 16, 2024

Hi! Can you clarify, is this in colab, on cloud or locally? What command are you using to train?

from ddsp.

jiemojiemo commented on August 16, 2024

Hi! Can you clarify, is this in colab, on cloud or locally? What command are you using to train?

Hi！I wan to train ddsp in locally. To be more precise, I want to run 3_training.ipynb in my gpu server.

I have tried modifying the code

# Setup the session.
import os
assert "COLAB_TPU_ADDR" in os.environ, "ERROR: Not connected to a TPU runtime; please set the runtime type to 'TPU'."
TPU_ADDRESS = "grpc://" + os.environ["COLAB_TPU_ADDR"]
sess = tf.Session(TPU_ADDRESS)

to this

sess = tf.Session()

But I got some warnings and errors

WARNING:tensorflow:cross_replica_sum should be used within a tpu_shard_context, but got unset number_of_shards. Assuming 1.

InvalidArgumentError: No OpKernel was registered to support Op 'CrossReplicaSum' used by node training/CrossReplicaSum (defined at /root/anaconda3/envs/hw/lib/python3.6/site-packages/ddsp/training/train_util.py:101) with these attrs: [T=DT_FLOAT, _class=["loc:@train.../transpose"]]
Registered devices: [CPU, GPU, XLA_CPU, XLA_GPU]
Registered kernels:
  <no registered kernels>

	 [[training/CrossReplicaSum]]

Errors may have originated from an input operation.
Input Source operations connected to node training/CrossReplicaSum:
 training/clip_by_global_norm/training/clip_by_global_norm/_0 (defined at /root/anaconda3/envs/hw/lib/python3.6/site-packages/ddsp/training/train_util.py:35)