Crawlr is an open-source CLI AI agent built to streamline the process of ingesting web-based information into structured knowledge bases. Utilizing OpenAI's GPT-3.5/4 models, it traverses specified URLs, cleans and chunks raw HTML into meaningful text segments, generates concise summaries, and creates vector embeddings for efficient semantic search. The tool supports configuration of crawl depth, domain filters, and chunk sizes, allowing users to tailor ingestion pipelines to project needs. By automating link discovery and content processing, Crawlr reduces manual data collection efforts, accelerates creation of FAQ systems, chatbots, and research archives, and seamlessly integrates with vector databases like Pinecone, Weaviate, or local SQLite setups. Its modular design enables easy extension for custom parsers and embedding providers.

Agentic Shared Memory provides a robust solution for managing contextual data in AI-driven multi-agent environments. Leveraging vector embeddings and efficient data structures, it stores agent observations, decisions, and state transitions, enabling seamless context retrieval and update. Agents can query the shared memory to access past interactions or global knowledge, fostering coherent behavior and collaborative problem-solving. The library supports plug-and-play integration with popular AI frameworks like LangChain or custom agent orchestrators, offering customizable retention strategies, context windowing, and search functions. By abstracting memory management, developers can focus on agent logic while ensuring scalable, consistent memory handling across distributed or centralized deployments. This improves overall system performance, reduces redundant computations, and enhances agent intelligence over time.