Ran out of memory error for 12GB GPU memory for 2MB training data? about chatbot-rnn HOT 4 CLOSED

On Tue, Mar 14, 2017 at 9:57 PM Hanfei Sun ***@***.***> wrote: I run the training for scotus.bz2 data by typing in python train.py. The program is run on AWS p2.8xlarge instance (12 GB GPU memory per GPU). And it throws the following exception Limit: 11330676327 InUse: 11035230976 MaxInUse: 11038110976 NumAllocs: 10139 MaxAllocSize: 71999744 W tensorflow/core/common_runtime/bfc_allocator.cc:274] **************************************************************************************************** W tensorflow/core/common_runtime/bfc_allocator.cc:275] Ran out of memory trying to allocate 34.33MiB. See logs for memory state. W tensorflow/core/framework/op_kernel.cc:975] Resource exhausted: OOM when allocating tensor with shape[3000,3000] — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#9 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAT7sR8UnLXJPImpsyjJ0VTP5wUL6jp7ks5rlv84gaJpZM4MdAzS> .