Challenge
Bring heterogeneous public sources into a resilient pipeline, remove duplicates, extract useful entities from unstructured text and images, connect nearby episodes and preserve both data provenance and the limits of confidence.
OSINT / DATA ENGINEERING / AI
A system from public source to traceable report
OSINT Pipeline automatically collects public publications, normalizes them, extracts entities, correlates events across time and geography and produces structured reports. Sources, confidence, uncertainty and processing history remain part of every result.
- Python
- PostgreSQL
- PostGIS
- Redis
- Docker Compose
- scikit-learn
- Tesseract OCR
- Telegram Bot API
Why OSINT Pipeline was created
Why OSINT Pipeline was created
Analysts otherwise review many open channels, feeds and publications by hand, separate repetitions from new information and reconstruct relationships between events. The system turns that continuous stream into one database, scheduled reports and conversational access to previously collected data.
Who the product serves
The product serves research and analysis teams that need continuous monitoring of public sources, a reproducible path from publication to conclusion and convenient delivery through reports and a restricted Telegram interface.
Solution
We built a containerized system of collectors, queues, processors and publishers. PostgreSQL with PostGIS stores messages, events, geography and relationships; NER, OCR and DBSCAN enrich the data; AI produces a cautious structured review, while Telegram and scheduled publishers deliver reports and database-grounded answers.
Capabilities
Product capabilities
The capabilities support one operating scenario and share common state, control and analytics.
Public-source collection
Collectors retrieve open Telegram publications, RSS and Atom feeds, social posts and additional public datasets.
Normalization and deduplication
Every record is linked to its source and external identifier, while media hashes and database constraints prevent repeated ingestion.
Entity extraction
Dictionary-based NER, fuzzy matching and regular expressions identify named objects and locations, while OCR reads text from images.
Spatiotemporal analysis
PostGIS stores geography and DBSCAN groups events that are close in time and space into episodes for further review.
Source-aware AI reports
The generator separates facts, source claims, assessment and uncertainty and includes confidence plus links to original publications.
Conversation and delivery
Users receive scheduled reports, can request a digest on demand or ask the bot questions grounded in data already stored by the system.
Outcome
Outcome
OSINT Pipeline brings collection, cleaning, geospatial analysis, AI review and publication into one reproducible process. Instead of manually scanning disconnected channels, the team gets a continuously updated database, reports with data provenance and controlled conversational access to accumulated context.
Python
PostgreSQL
PostGIS
Redis
Docker Compose
scikit-learn
Tesseract OCR
Telegram Bot API
New project
Working on a similar task?
Describe the current situation and the outcome you need. We will understand the context and propose a practical first stage.