Segmentation fault in `tf.config.experimental_connect_to_cluster` with `@tf.function` about tensorflow HOT 2 OPEN

Hi @Zoeeeeey ,

Thanks for reporting. As mentioned I have tested the code snippet in tf-nightly without tf.function decorator and it executes fine. With tf.function decorator it seems going through infinite loop as I waited for more than 5 mins and still it continuous to execute but couldn't get segmentation fault error.

from tensorflow.

Hi @SuryanarayanaY ,

1.When I tried to replicate this reported, I initially encountered a situation where there was 'no response' and 'no termination' during the first run.

When I used ctrl+c to terminate the program and ran it again, I encountered a Segmentation fault (core dumped).

My envs which occurs segfault：

Python 3.9
tensorflow 2.16.1
gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Nov_30_19:08:53_PST_2020
Cuda compilation tools, release 11.2, V11.2.67
Build cuda_11.2.r11.2/compiler.29373293_0

2.Nevertheless, there are situations where the behavior is inconsistent, whether adding @tf.function or not.

For example, the part that contains pass can be replaced with a print statement.

When @tf.function is added, the program fails to terminate normally.

Why is this the case?

3.Additionally, I have a question about the tf.config.experimental_connect_to_cluster function.

I'm unclear about the exact content that should be passed into the first parameter cluster_spec_or_resolver.

For example, in this case, a cluster is created with tf.train.ClusterSpec, but the specified workers (such as "worker0.example.com:2222") do not actually exist.

What should be the correct content to set for this parameter?

4.Additionally, to facilitate your replication, I have attempted to create a gist in Colab. Please consult it.(https://colab.research.google.com/gist/Zoeeeeey/1782cc57b0a63d8c6a022eeae0977235/62522_tf-nightly-v.ipynb)

Thank you for any response. ;->

from tensorflow.