LLMs are getting smarter and quantization algorithms are making those LLMs to be trained on smaller and smaller resources. The architecture and pipelines for building new multimodal LLMs and ensemble of collaborative LLMs are becoming more and more complex. Tree of Thoughts is an example of such custom architectures, which shows big potential to improve LLM accuracy significantly. Building Data Infrastructure of such customizable LLM architecture is still an open yet extremely important problem. We will discuss how such data architectures can be built on scale.