Hello Everyone,
This article is going to be a quick introduction to Open-Source AI developer tools.
On this Newsletter we have quite a few software developers who are building with LLM open-source models and tools.
With the arrival of so many impressive Open-source models of late including open-weight models finally closing the gap with GPT-4 on LMSYS’ Chabot Arena, such as Cohere’s Command-R, it’s time to take a look with the research and analyses of specifically on AI developer tools.
The Strategy Deck is known for its visuals, infographics and stand-out market research. Highly recommend her services as a consultant and researchers as well, contact her here.
My Approach to Native Sponsors
As my Newsletter evolves I’m trying to choose consciously the sort of builders I want to associate with. This includes giving early stage startups and solo builders steep discounts. My latest Native Sponsor is a small new builder that I’m especially bullish about, it’s an AI copilot that autocompletes text EVERYWHERE on MacOS. In every app!
I haven’t seen many copilots for MacOS.
📶 From our sponsor: 📶
Bring your AI to every Mac app, using the app’s context
Omnipilot brings AI to every Mac app, using the app’s context to provide intelligent assistance. You can invoke it with a shortcut to supercharge writing, email, and getting answers.
Thanks to Alex for this article. You can read her bio at the end of the article. Or contact her on LinkedIn.
Articles to Revisit
Market Map: Gen AI Companies with Foundational Models
How BigTech Invested in AI Companies in 2023
An Overview of Google’s AI Product Strategy
By of
This is a collection of some of the most popular open source AI and ML developer tools, ranked by the number of stars they have on GitHub, for projects active in 2023 and 2024. It focuses on developer applications used to train and deploy ML models and AI agents and its purpose is to highlight the breadth and diversity of tools and frameworks that are being built by the open source AI community and the vast potential in the space.
The collection contains:
Model training and inference frameworks:
General Purpose ones – TensorFlow, Hugging Face Transformers, Diffusers,
Optimized to run on local clients – GPT4ALL, Ollama, MLC LLM and
AI agent development kits – Autogen, Semantic Kernel, Unity ML Agents Toolkit
Vector databases – Milvus, Faiss
Orchestration tools – LangChain, LlamaIndex, Flowise, Mindsdb
Compute Optimization libraries – Colossal AI, DeepSpeed, Ray, Vllm
Editor’s Note: Context: Read Chip Huyen’s long blog.
20 Popular Open Source AI Developer Tools
TensorFlow is a widely used library for training and inference of ML models that offers significant versatility and scalability across platforms, from consumer hardware to clusters of servers. Open sourced in November 2015 by Google, it has since collected over 182,000 stars on GitHub.
Hugging Face Transformers is a library that offers access and tools to fine-tune and deploy a vast collection of the open source models that are hosted on Hugging Face. It is compatible with TensorFlor, Pytorch and JAX and includes support for text, image and audio models. Launched in 2018, Transformers has gathered 124,000 stars on GitHub.
LangChain is a framework for developing and deploying applications powered by large language models developed by the company with the same name. The tool provides modular building blocks and components for building custom chains, it allows inspection, monitoring and evaluation of applications and can turn any chain into a REST API. Launched in October 2022, LangChain has achieved over 81,000 stars on GitHub.
PyTorch is a very popular General Purpose framework and library developed by Facebook’s AI Research lab and released in October 2016, with over 77,000 stars on GitHub.
GPT4ALL is a client that can install and run AI models on consumer-grade and edge hardware provided by Nomic AI. Optimized for CPU-only, no-internet environments, GPT4All runs on Windows, OSX and Ubuntu and can run a series of models, including Alpaca, Llama, Pythia, Mosaic, Falcon, StablLM and custom GPT4All ones. Launched in August 2023, it gained over 63,000 stars on GitHub.
Ollama is a tool which enables users to use open source LLMs locally on their Windows, macOS or Linux machine. With support for a variety of models, including Gemma, Llama, Mistral, Mixtral, Command-R and Llava, the tool gathered over 52,000 stars on GitHub since its launch in February 2023. Besides the GitHub repository, resources for Ollama are available on the website and Discord channel.
ColossalAI provides a collection of parallelism components for distributed training and inference of models with a few lines of code. Their tools support data, pipeline, tensor and sequence parallelism, as well as a zero redundancy optimizer and a method for automatic management of parallelization. Since its creation in 2021, the Colossal AI repository has gained over 37,000 stars on GitHub.
DeepSpeed, provided by Microsoft, is a deep learning optimization library for distributed training and inference. It is developed to support large to very large models that need to be trained at scale on hundreds and thousands of GPUs in resource constrained systems, and yet still deliver low latency and high throughput for inference. Since its launch in May 2020, it has gathered over 32,000 stars on GitHub.
LlamaIndex is a data framework for LLM-based applications with Retrieval Augmented Generation. The tool provides the abstractions necessary to more easily ingest, structure, and access private or domain-specific data in order to inject them into LLMs. Its components include Data connectors to get existing data from their native source and format, data indexes to structure it in intermediate representations and engines to provide natural language access to the data. Since it launched in Nov 2022, Llamaindex collected over 30,000 stars on GitHub.
Ray is an unified compute framework for scaling AI and Python workloads — from reinforcement learning to deep learning to tuning, and model serving. It consists of a core distributed runtime and a set of AI libraries for simplifying ML compute, including ones that deal with datasets, distributed training, hyperparameter tuning and inference. Developed by Anyscale, Ray has over 30,000 stars on GitHub.
Milvus is a vector database built to power embedding similarity search and AI applications. It is used to make unstructured data search more accessible regardless of the deployment environment. Developed by The Linux Foundation AI & Data organization, Milvus has gathered over 36,000 stars on GitHub.
Faiss (Facebook AI Similarity Search) is a library for efficient similarity search and clustering of dense vectors developed by Meta AI. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and parameter tuning. Since its launch in March 2017, it collected over 27,000 stars on GitHub.
Autogen is a multi-agent conversation framework that works as a high-level abstraction for building workflows using multiple LLMs. Developed by Microsoft, it is meant to support the development of modular and complex agents to perform sophisticated tasks that use AI models, as well as a variety of other tools and components. Since its launch in September 2023, Autogen collected over 24,000 stars on GitHub.
Hugging Face Diffusers is a library for fine-tuning and deployment of pretrained diffusion models for generating images, audio, and 3D objects. It includes diffusion pipelines for inference, interchangeable noise schedulers for different diffusion speeds and output quality and access to pretrained models from the Hugging Face platform. Diffusers has over 22,000 stars on GitHub.
Flowise is a low-code, drag and drop tool to develop customized LLM orchestration flows and AI agents. Built around customization and modularity, the framework supports integrations with other frameworks, the creation of autonomous agents for various tasks and integration or open source, locally run LLMs. Since its launch in February 2023, Flowise has collected over 21,000 stars on GitHub.
Mindsdb is a platform which automates pipelines that connect real-time enterprise data to AI systems. It is used to train and customize models, automate tasks, define and execute trigger events and provide observability. Since it launched in 2017, it has gathered over 21,000 stars on GitHub.
Semantic Kernel from Microsoft is an SDK that lets developers build AI agents that can call on existing code. It lets them mix conventional programming languages, like C# and Python, with LLMs using prompt templating, chaining, and planning capabilities in order to build AI experiences into existing applications. Released in March 2023, Semantic Kernel has gathered over 17,000 stars on GitHub.
Vllm is a high-throughput and memory-efficient inference engine for LLMs that uses PagedAttention, an algorithm for the management of attention keys and values. Developed at UC Berkeley, it was launched in September 2023 with the paper “Efficient Memory Management for Large Language Model Serving with PagedAttention” and it has since collected over 17,000 stars on GitHub.
Machine Learning Compilation for Large Language Models (MLC LLM) is a universal deployment solution that allows native deployment of any large language models with native APIs with compiler acceleration. The mission of the project is to enable everyone to develop, optimize and deploy AI models natively on everyone’s devices with ML compilation techniques. Developed by researchers at Carnegie Mellon University, MLC LM has over 16,000 stars on GitHub.
The Unity Machine Learning Agents Toolkit enables games and simulations to serve as environments for training intelligent agents. It provides PyTorch-based algorithms to enable game developers to train intelligent agents for 2D, 3D and VR/AR games. The agents can be used for multiple purposes, including controlling NPC behavior (in a variety of settings such as multi-agent and adversarial), automated testing of game builds and evaluating different game design decisions pre-release. The toolkit gathered over 16,000 stars on GitHub.
Author’s Bio
Alex Sandu, the author of this guest post, a popular contributor to AI Supremacy, and the writer of The Strategy Deck, a newsletter focused on AI market analysis, is seeking a new role.
With expertise in building global consumer, developer and open source tech products, Alex brings 15 years of experience in cross-functional management roles, including Technical Product Management and Corporate Strategy.
Alex is a seasoned expert in building features from concept to launch for hundreds of millions of users worldwide, driving product strategy through data and insights, and managing strategic operations and annual planning for large organizations.
An experienced Product and Strategy Manager, Alex excels in aligning product vision with customer and market requirements, building competitive differentiation for your company and driving product development and impact in market.
To talk to Alex about how she can be a valuable addition to your team, reach out at alex [at] TheStrategyDeck [dot] com, or on LinkedIn.
In March, 2024 Chip Huyen wrote a blog post titled: “What I learned from looking at 900 most popular open source AI tools”.
I find her work really compelling. In a Twitter post she speculated on:
Major Problems in AI that Startups Could Address
1. Data synthesis: AI has become really good both at generating and annotating data. The challenge now is to make sure that the generated data is safe and legal, e.g. not violating any IP.
2. Evaluation: evaluation has gotten so much harder with LLMs, both because many people treat models as blackboxes (we deploy models someone else developed for us) and because outputs can be open-ended. At the same time, investment in evaluation is nowhere close to investment in model or application development. I’d be interested in arena-style evaluation, embedding evaluation, human-in-the-loop evaluation, as well as small, specialized scorers (instead of using large models like GPT-4 as judges).
3. Energy: the bottleneck to scaling AI is no longer compute but electricity. I’m interested in all energy-related problems, including both new energy sources and energy trading.
4. Any application that allows you to collect unique data that nobody has. I’ve heard concerns about building applications that seem to be “wrappers” around popular APIs. If you can get to the market early and gather sufficient data to continually improve your product, data is your moat.
5. GPU-native everything: many data science toolings, including scikit-learn, pandas, and Spark, aren’t built to run natively on GPUs. There have been efforts to make these tools more efficiently leverage GPUs, but I think there’s still a lot of room for the software layer for GPUs (and not just NVIDIA GPUs).
6. Curated Internet: bots are already ruining dating apps, search (bots are incredibly good at SEOs), and social media. I’d like to be able to set a boundary for my Internet, e.g. to limit the search results to those written by people I trust, or sources verified to be human.
From the blog article I cited above, here are some of the infographics that are related to our topic of today.
The AI Stack Has Exploded in the Early 2020s
In 2023, the layers that saw the highest increases were the applications and application development layers.
Coding
Bots
Info Aggregation
Image Production
Workflow Automation
2023 was the Year AI Grew Up
Is the Era of the one-person billion-dollar company feasible?
According to Chip’s massive survey: “845 repos are hosted on 594 unique GitHub accounts. There are 20 accounts with at least 4 repos. These top 20 accounts host 195 of the repos, or 23% of all the repos on the list. These 195 repos have gained a total of 1,650,000 stars.”
If you are a developer, builder or even a tinkerer these are interesting times.
Is the Democratization of AI heralding the Era of the Solo Builder?
Chip noticed that applications started by individuals, on average, have gained more stars than applications started by organizations.
Live fast, die young – as Github Repos so with Consumer AI Apps
With all of this building we also see an incredible among of trends, churn and temporary spikes in usage and stars (i.e. developer discovery). In 2022 to 2025 we are still in the ‘hype curve’ of Generative AI’s nascent first years.
I’ve noticed this trend too with Consumer AI apps of course, where an app can become popular suddenly, only to quickly decline as rising competitors become more impressive. What this means is trends in Generative AI are at this time, and may continue to show very temporary growth curves given that the open-source LLM acceleration is so dynamic during this period.
AI developer tools are the backbone of this movement. Unique solo builders are also what makes the space so exciting to watch.
If you find Chip Huyen’s work inspirational and want to learn more, please consider purchasing her book.
If you are a software developer thinking of dabbling in Generative AI, don’t sleep on it.
Don’t Sleep on It
Image: Check out Chip Huyen’s IG.
Some Helpful Newsletters
AIEdge
Ahead of AI
AI Tidbits
Supervised
AI on a Budget
Latent Space
You can discover more AI Newsletters on this curated list.
Read More in AI Supremacy
