Crawlr is an open-source CLI AI agent built to streamline the process of ingesting web-based information into structured knowledge bases. Utilizing OpenAI's GPT-3.5/4 models, it traverses specified URLs, cleans and chunks raw HTML into meaningful text segments, generates concise summaries, and creates vector embeddings for efficient semantic search. The tool supports configuration of crawl depth, domain filters, and chunk sizes, allowing users to tailor ingestion pipelines to project needs. By automating link discovery and content processing, Crawlr reduces manual data collection efforts, accelerates creation of FAQ systems, chatbots, and research archives, and seamlessly integrates with vector databases like Pinecone, Weaviate, or local SQLite setups. Its modular design enables easy extension for custom parsers and embedding providers.

Agentic-RAG-DeepSeek combines agentic orchestration with RAG techniques to enable advanced conversational and research applications. It first processes document corpora, generating embeddings using LLMs and storing them in DeepSeek's vector database. At runtime, an AI agent retrieves relevant passages, constructs context-aware prompts, and leverages LLMs to synthesize accurate, concise responses. The framework supports iterative, multi-step reasoning workflows, tool-based operations, and customizable policies for flexible agent behavior. Developers can extend components, integrate additional APIs or tools, and monitor agent performance. Whether building dynamic Q&A systems, automated research assistants, or domain-specific chatbots, Agentic-RAG-DeepSeek provides a scalable, modular platform for retrieval-driven AI solutions.