Do you have any code from your discord bot you're willing to share? I'd be happy to share back any updates I made to it. I've been wanting to play with this idea for a bit.

I don't understand, the Ampere they refer to in their free tier are cpu's not gpu's. How did you manage to do that

Custom PyTorch with on-chip acceleration: https://cloudmarketplace.oracle.com/marketplace/en_US/listin...

Not as fast as a GPU, but less than 5 seconds for a 250 token response is good enough for a Discord bot.

This is the most interesting thing I've read in this thread. How have I never heard of this accelerator?!