We used frameworks in the past, tried langchain, langgraph, and Openai's agents SDK pretty extensively. Now we roll our own, generally a much better and cleaner experience. We essentially built our own internal framework for our own use-case, we liked the graph approach of langgraph - so we took elements of that. We write everything async, and added nice handling for streaming.
You can see our framework [here](https://github.com/aurelio-labs/graphai). I don't necessarily recommend it as it's built for our use-cases and I make no guarantees for others, but it might be interesting to see what rolling your own might look like
Sharing my walkthrough on fine-tuning LLMs with LoRA using NVIDIA's NeMo microservices. The result is a llama-3.2-1b-instruct model fine-tuned to be really good at function-calling, making it ideal for agent-use.
It was a ton of fun to figure it out and it brought back some nostalgia from the days of training ML models, tweaking learning rates, dropout, and watching loss charts in W&B.
Final performance was way better than any 1-3B parameter LLM I tried with agentic workflows in the past.
Releasing this walkthrough on fine-tuning LLMs with LoRA using NVIDIA's NeMo Microservices (they sponsored the video, but with no reqs on what I do or say). We cover a ton on building prod AI applications, including:
- LoRA fine-tuning of Llama 3.2 1B
- Deploy of private AI agent systems
- Using NVIDIA NIMs to host our fine-tuned LLM
- Interacting with our LLM + streamed output
It was a ton of fun to figure this out, and it brought back some nostalgia from the good old days of training ML models, tweaking learning rates and dropout, and watching loss charts in W&B.
The result is a llama-3.2-1b-instruct fine-tuned to provide pretty good function-calling abilities (better than any other out-of-the-box 1-3B models I tried).
I made a course covering everything you'd need to know to start building with LangChain. It assumes no level of expertise, you can be a complete beginner and by the end of the course be building AI agents with chat memory, streaming, async, etc. There's no cost, it's all free!
I and my team have been building a python library for improving the steerability of AI agents, the library also allows us to add an essentially unlimited number of tools to agents, add safety guardrails, etc. We have a lot more coming and I'm just excited to share our first public release. Let us know your thoughts!
Not weighing on whether it is AGI-like or not, but my view on the search component is that it's like when we google info, we have a search term and we don't know whether that will return good results or not. So I suppose in order for RAG to become better the LLM needs to be able to review the information given, and decide whether to search more (which they can do with agents to be fair), and we also need to be building retrieval systems that do very well at returning the right information even with sub-optimal search queries.
I think LLMs are fairly good at deciding what information is relevant to a query or not though, it is rare that I have found an LLM to be distracted by RAG results (even when they're not relevant)
Yeah RAG has been around for some time, paper [2] being where I first stumbled onto it — I remember a 2-3 years ago building a RAG pipeline using either the model they trained in paper 2, or that was trained based on the same idea, with Deepset AI's Haystack library. It has been a little surreal to see that terminology become so popular again
I may be misunderstanding, but I'll try to answer — quantization typically means retrieval will be slower (if referring to techniques like product quantization), but that is the case whether you're at 10K vectors or 1B vectors, afaik it doesn't really make a difference because you're only quantizing the query vector at query time (it has been awhile since I read anything on quantization, so I could be mistaken).
Maybe your question is referring to the need to have quantization at larger index sizes? In which case, yes would typically be true because you're wanting to either (1) minimize the index size when quantizing it, or (2) optimizing the query space (to search through less). Whether you want (1) or (2) will impact on the type of quantization being performed (basically 1 == product quantization, and 2 == inverted index)
So once you get to the 1M+ size, you need to consider quantization in some form - or you can go with graph-based retrieval, if you don't mind using a lot of disk space.
You can see our framework [here](https://github.com/aurelio-labs/graphai). I don't necessarily recommend it as it's built for our use-cases and I make no guarantees for others, but it might be interesting to see what rolling your own might look like