Thank you for your response. $600/month for Llama 3.1 8B in g6.xlarge would be reasonable cost, but as you graciously pointed out, the API cost would be cheaper. Thus, we’ll likely go with the OpenAI and other API costs. What are the privacy concerns?
For the purpose of experimentation with HuggingFace TGI, is there anything cheaper than $600/month that we could use for testing? For example, can the intern turn off the GPU instance when they are not working? I’m trying to figure out what to recommend to them. I am somewhat confused as to the costs for the GPU-enabled containers and I don’t want to put the burden of the cost recommendation on the intern. If they make a mistake with the purchase of a container, they may feel bad.
What I’d like to do is buy them the resources, then instruct them to test out HuggingFace TGI in the resource that I purchased for them. They can then report back on any performance or result optimization differences.