Overview
AI-Steve is a personal research and memory engine designed for fast recall, context-rich answers, and long-horizon insight. It merges a Flask web app, PostgreSQL with embeddings, and Claude-powered reasoning into a single system that can search across daily life in seconds.
Search Content, sentiment intelligence, and health analytics feed a daily and weekly messaging cadence: morning health + priority briefs, multi-year reflections (1, 3, 5, 7, 10 years ago), and end-of-week recaps with a week-ahead projection.
Dynamic ingest dashboard showing emails, Q&A pairs, photos, attachments, calendar events, to-dos, iMessages, and curated imports.
Desktop UI grounded by embeddings and transcripts, with quick access to images, stats, and history.
Daily multi-year reflections (1, 3, 5, 7, and 10 years ago) to surface patterns and memory anchors.
Mobile-friendly view for long-form recall and storytelling (same engine, responsive layout).
Image Explorer integration: similarity search, clustering, and annotation flow.
Apple Health integration: correlations, lag effects, and sleep drivers.
Code Directives: Domain-Specific Auto-Coding
Added into my AI-Steve infrastructure is the ability to auto-code projects in a domain-specific way using simple natural language project descriptions on top of Droid, Claude, and/or Codex.
What brought it to the next level is using the AI-Steve RAG system to wrap a project direction in my voice - imagine enabling all your coders to code given their own past projects and insights. It is akin to saying: build this new OS in the voice of Linus Torvalds.
Code directives pipeline: auto-ingest, curation, AI review, iterative coding loop, and report delivery.
Search Content Experience
A Google-like interface for your personal data. Hybrid search blends semantic similarity with keyword precision, deduplicates chunked documents, and surfaces media inline so the result feels like a visual archive, not a database.
Hybrid Search, Human-Scale Results
Search Content spans emails, attachments, photos, calendar events, and sentiment summaries with tunable semantic/keyword weighting, date filters, and content-type pills. Results expand into full documents, keeping the context intact.
Search Content interface with Google-like ranking and visual recall.
Sentiment + Health Intelligence
AI-Steve turns personal data into narrative insight across time. Monthly reflections roll into quarterly, yearly, and all-time sentiment summaries, while Apple Health analytics surface correlations and sleep drivers that are immediately searchable.
Time-Scale Storytelling
Sentiment narratives stitch together emails, messages, calendar events, health metrics, and life milestones. The result is a living record of emotional tone and well-being across months, quarters, years, and full history.
Morning motivational reflections highlight what happened 1, 3, 5, 7, and 10 years ago—turning memory into momentum.
End-of-week messages summarize sentiment + health trends, then project the coming week with focus areas, reminders, and context grounded in Search Content.
Combined health and sentiment visuals for longitudinal insight.
Core Components
Flask Web App
Lightweight web layer handling chat, search, and dashboards with secure sessions.
Chat + LLM
LLMHandler abstracts Anthropic API; ChatHistoryManagerV2 stores transcripts in Postgres for recall and longitudinal context.
Vector Store
PgVectorDB wraps PostgreSQL pool with pgvector, embeddings, and health checks. Shared by search and chat to avoid duplicate model loads.
Monitoring
StatusChecker and notifier hooks send operational alerts (email) and ensure upstream services (DB, queues) are healthy.
Agentic Build Loop
Small automation and coding tasks can be spun up through Droid, Claude Code, and Codex when the system needs to ship a quick script or workflow.
RAG & Retrieval Flow
PgVectorSearch) → context assembly (ContextManager + TemporalSearch) → Claude → response + transcript persisted.
Temporal and URL de-duplication modules keep responses grounded, while ContextualLearning stores corrections for future runs.
Data Pipelines
Email + Files
Email, attachments, and file drops are ingested and embedded; processed IDs are tracked to avoid duplicates.
Embedding
Text and vision embeddings flow into documents and image_embeddings tables for retrieval.
Nightly Maintenance
Nightly refresh cleans ingestion queues, keeps clusters healthy, and verifies database connections.
Corrections
User corrections captured in UserCorrections dir and database; fed back into context assembly to reduce repeated mistakes.
Daily + Weekly Messaging
Scheduled emails deliver morning health summaries, multi-year reflections (1, 3, 5, 7, 10 years ago), and end-of-week recaps with week-ahead projections.
Apple Health Exports
Daily health CSVs and face captures feed correlation analysis, sleep scoring, and longitudinal wellness tracking.
Online Presence
Monitors public mentions, profiles, and updates for people and projects, with periodic refresh.
News Alerts + RSS
RSS feeds are dynamically assigned and updated based on evolving search history and relevance.
Content sources captured: emails and attachments, calendar events, to-do items, iMessages, Mac Photos, additional curated/imported photos, chat sessions, Q&A, and Socratic pair extractions from emails.
Visual ingest is critical: photos (including sub-images/video frames) feed CLIP embeddings that power the Image Explorer, turning similarity and clustering into fast annotation and richer RAG grounding.
Built with natural-language coding: scripted via spoken English using Wispr Flow driving agentic CLI tools (Droid, Claude Code, Codex, Gemini CLI) to assemble pipelines end-to-end.
Safety, Security, Ops
- Rate limiting and IP blocking/whitelisting guard login brute force.
- Session cookies hardened in HTTPS mode; proxy headers handled via
ProxyFix. - PG password required via env; pool health checks plus vector extension validation.
- Logging to
logs/(web, similarity, ingestion) with rotation handled externally. - Monitoring hooks page operators when ingestion or DB checks fail.
Image Explorer Integration
AI-Steve embeds a visual memory system. Image Explorer is designed to rapidly find, cluster, and annotate related images so the RAG layer gains richer visual context.
Similarity Search
CLIP embeddings stored in image_embeddings; search always returns the query image and filters by threshold/face counts.
Clustering
Nightly k-means auto-detects optimal K, preserves user-named clusters, and replaces system kmeans_* sets.
Annotation
Inline annotation of clusters and similarity groups writes to DB and per-image JSON sidecars while keeping user labels durable.
Chat Entry Points
Desktop and mobile chat include an “Images” launcher that opens explorer in a new tab.
Faces & Tagging
Automatic face extraction creates face embeddings that cluster naturally across Mac Photos and video frames. Tagging and exclusions make face-aware search and annotations fast and consistent.
Face clustering and tagging across decades of photos; bulk selection to label similar faces in one pass.
Apple Health Integration
Apple Health exports feed a nightly analytics pipeline that computes correlations, lag effects, and sleep-concentration models. The results are stored as visual artifacts and surfaced inside AI-Steve for review, coaching, and longitudinal insights.
Face → Health treats the face as a longitudinal wellness signal. Daily images support both retrospective (last night) and predictive (tonight) sleep models with strict temporal alignment.
Data Ingest
Daily CSV exports are normalized into a unified table with sleep, activity, cardiovascular, and environmental metrics.
Analytics + ML
Correlation engines surface top drivers for deep and REM sleep, plus a Random Forest model for sleep concentration.
Visual Memory
Key plots are embedded and searchable alongside other visual artifacts, enabling fast recall and trend review.
Full correlation heatmap across Apple Health metrics, sorted by sleep concentration.
Top 20 features driving sleep concentration prediction.
.png)
Top 25 correlates with deep sleep (positive and negative drivers).
.png)
Top 25 correlates with REM sleep to isolate recovery signals.
Sleep-focused correlation matrix highlighting the strongest drivers across daily metrics.