Crawlr is an open-source CLI AI agent built to streamline the process of ingesting web-based information into structured knowledge bases. Utilizing OpenAI's GPT-3.5/4 models, it traverses specified URLs, cleans and chunks raw HTML into meaningful text segments, generates concise summaries, and creates vector embeddings for efficient semantic search. The tool supports configuration of crawl depth, domain filters, and chunk sizes, allowing users to tailor ingestion pipelines to project needs. By automating link discovery and content processing, Crawlr reduces manual data collection efforts, accelerates creation of FAQ systems, chatbots, and research archives, and seamlessly integrates with vector databases like Pinecone, Weaviate, or local SQLite setups. Its modular design enables easy extension for custom parsers and embedding providers.

DocGPT is designed to simplify information extraction and Q&A from documents by providing a seamless conversational interface. Users can upload documents in PDF, Word, or PowerPoint formats, which are then processed using text parsers. The content is chunked and embedded with OpenAI's embedding models, stored in a vector database like FAISS or Pinecone. When a user submits a query, DocGPT retrieves the most relevant text chunks via similarity search and leverages ChatGPT to generate accurate, context-aware responses. It features interactive chat, document summarization, customizable prompts for domain-specific needs, and is built on Python with a Streamlit UI for easy deployment and extensibility.