Post

Using a lot of gen AI in your application? Here are a few strategies to optimize costs: #buildinpublic #webdev 🧵 👇

1/ Prompt caching If you're running the same prompt, OpenAI's API offers automatic context caching for generations over 1k tokens. Put the variable part of the prompt at the end of it - otherwise caching won't take effect. This can reduce costs by 50%. platform.openai.com/docs/guides...

2/ Batching Just like caching, you can reduce the costs by 50%, except you can do that by batching multiple generations at once. This API is asynchronous and can take up to 24 hours to finish running. This is a trade-off for complexity in your app. platform.openai.com/docs/api-re...

4/ Openrouter It has a lot of models you can try out with a single account and payment. Some of the cheapest ones are: OpenAI o1-mini, Gemini Flash, Llama 3.2 Instruct Experiment and find which model suits your needs best.

4-2/ If you're using an abstraction layer like Vercel's AI SDK, the model and API you use becomes an implementation detail and can be changed very easily. Just make sure the model can run tools you need, like JSON generation.