Crawlr is an open-source CLI AI agent built to streamline the process of ingesting web-based information into structured knowledge bases. Utilizing OpenAI's GPT-3.5/4 models, it traverses specified URLs, cleans and chunks raw HTML into meaningful text segments, generates concise summaries, and creates vector embeddings for efficient semantic search. The tool supports configuration of crawl depth, domain filters, and chunk sizes, allowing users to tailor ingestion pipelines to project needs. By automating link discovery and content processing, Crawlr reduces manual data collection efforts, accelerates creation of FAQ systems, chatbots, and research archives, and seamlessly integrates with vector databases like Pinecone, Weaviate, or local SQLite setups. Its modular design enables easy extension for custom parsers and embedding providers.

AI_RAG delivers a modular retrieval-augmented generation solution that combines document indexing, vector search, embedding generation, and LLM-driven response composition. Users prepare corpora of text documents, connect a vector store like FAISS or Pinecone, configure embedding and LLM endpoints, and run the indexing process. When a query arrives, AI_RAG retrieves the most relevant passages, feeds them alongside the prompt into the chosen language model, and returns a contextually grounded answer. Its extensible design allows custom connectors, multi-model support, and fine-grained control over retrieval and generation parameters, ideal for knowledge bases and advanced conversational agents.