Does this work on a jetson nano? about gpt2bot HOT 1 CLOSED

I can’t think of any issues running it on jetson nano, the main package you will have to install are transformers; you can find dependencies in their setup.py file. Cuda is available in got2bot and enabled by default, but the bot can be deployed on CPU devices as well. It takes a one-two seconds to generate a response without GPU; with GPU a fraction of second; this of course depends on how far is the stop token in generated response.

I’d definitely rethink setting turns_memory to unlimited since it will slow down generation process with time; values between 2 and 5 are optimal for multi turn-based chat. Number of samples of 5 will further increase processing time (up to 2-3x), and you will have to implement your own logic for selecting the best candidate sample (by default it’s done by random selection)

from gpt2bot.