Flow-correlation traffic analysis
relationship-gtpc-user derives a single per-subscriber correlation key as Hashkey = md5(IMEI + IMSI + phone_number + vsys_id) from mobile-core GTP-C signaling, and a separate per-tunnel key gtp_uuid = md5(uplink_teid + "|" + downlink_teid). On every update it writes not just the primary user<->TEID row (tsg_galaxy:relation_user_teid) but up to four additional secondary-index rows into tsg_galaxy:gtpc_knowledge_base, one per identifier (IMEI, IMSI, phone number, APN) with a type-digit prefix ("0"/"1"/"2"/"3") and — for phone number and APN — the value's characters reversed before use as the row key, a range-scan-friendly key design that lets any of the four identifiers be used to look up the associated tunnel/session independently.
radius-relationship-hbase (galaxy/tsg_olap) builds a bidirectional, MD5-keyed RADIUS accounting index in HBase — a framed-IP-to-account table and its account-to-IP mirror — so any subscriber account or assigned IP can be looked up in O(1) from either direction, per multi-tenant vsys_id. Each direction is fronted by a Guava write-through cache (10M-entry cap, 24h TTL) so lookups do not hit HBase on the hot path.
radius-account-knowledge's MapCompletedFunction explicitly prefers the RADIUS acct_multi_session_id field over acct_session_id when both are present in an accounting record, specifically to keep a single logical session identifier stable across events that would otherwise fragment it (e.g. re-authentication, mobility/roaming, NAS handoff) — a deliberate design choice for session-identity continuity rather than a default library behavior.
The VoIP-fusion correlation key linking a SIP signaling record to its RTP media record is not timing/heuristic-based -- it is an exact match between the negotiated media endpoint fields parsed out of the SIP record (from_rtp_ip/from_rtp_port/to_rtp_ip/to_rtp_port, i.e. the SDP-negotiated RTP endpoints carried in the SIP signaling) and the actual observed RTP flow's four-tuple (common_client_ip/port, common_server_ip/port). A separate normalization rule sorts every four-tuple by port then IP (smaller first) before use as a join key, and common_vsys_id is the default grouping/partition key.
VoipCalibrationWindowFunction only auto-merges a candidate group when it is exactly one-to-many (1 RTP matched to N SIP records, or 1 SIP matched to N RTP records); any other cardinality within a correlation-key bucket (e.g. 2 SIP + 2 RTP) is explicitly logged as anomalous ("Not logical") and every record in that bucket is emitted individually as an unmerged "violation" log rather than being forced into a merge or silently dropped.
VoIP correlation runs in two sequential time windows, not one -- an initial tumbling window (voip.calibration.window.time, configured at 30s) attempts to fuse SIP/RTP records sharing a key, and any record still unmatched after that window is held in a second in-memory cache and retried against a longer secondary window (sec.combine.sr.cache.secs, configured at 60s) before finally being emitted unmerged -- giving the pipeline up to ~90s of tolerance for SIP and RTP legs of the same call arriving out of sync.
Both VoIP-correlation implementations are boundary-aware rather than treating every SIP/RTP flow uniformly. The legacy pipeline (log-stream-voip-relation) has a configurable check.inner.network toggle and an isInnerIp() check that classifies purely-internal-network SIP flows into a distinct "sip-in" type which is passed straight to output unmerged (never fed into the correlation cache); the newer Easy-Stream rewrite (sip-rtp-correlation) exposes the same internal/external distinction as first-class Flink Table UDFs (IS_INTERNAL_IP_ADDRESS, HAS_EXTERNAL_IP_ADDRESS).
TSG's VoIP-fusion pipeline (galaxy/tsg_olap/log-stream-voip-relation) does not just correlate SIP/RTP metadata into call records -- when both correlated RTP legs carry a stored packet-capture file path (rtp_pcap_path), the pipeline concatenates the two paths and writes the combined path into the final merged VoIP record, i.e. the correlated call record carries a pointer to the actual retained audio capture(s) for that call, not just flow statistics.
A MESA Lab task tracker records the assignment "TSG: determine how many clients are behind an IP address," and a companion technical design document details the method: identify distinct TLS clients sharing one public IP using a <JA3 fingerprint, server domain, server IP> 3-tuple (JA3 alone collides across different apps), then use TLS Session Ticket reuse/lifetime sequences per identified client to detect multiple concurrent devices (i.e. NAT) behind that IP.
The same MESA-affiliated researcher (name redacted) gave a "cyberspace mapping" presentation that describes an operational dark-web ("暗网") mapping capability built on controlled/malicious Tor relay, HSDir, and guard nodes: 4,254 discovered node-to-hidden-node bindings and 63 identified hidden router families via guard-node correlation; a "parallel undetectable" hidden-service traceback method that embeds a watermark in the client's packet-direction sequence to recover a hidden service's real IP and guard-node IP; and a hidden-service "isolation" attack that floods a target's HSDir nodes with malicious descriptors, claimed effective against "a well-known foreign dark-web site."
A published MESA Lab / IIE-CAS paper ("ExitSniffer", CCIS-2021) and two filed patents describe a tool that actively builds 2-hop Tor circuits through every exit relay to decoy websites and diffs the decoy's observed source IP against the relay's public consensus IP to expose non-public "hidden node" infrastructure behind Tor exits (96% coverage of all exits in about 50 seconds), plus a complementary passive method -- a government-funded ("国家信息安全项目") controlled Tor relay that logs real predecessor-hop IPs -- that found hidden nodes behind 71 of 6,850 tracked routing nodes; the same research program's dataset separately logs 10,412,582 real anonymous-user access records against 1,627,920 distinct clearnet domains visited through Tor.
A MESA thesis proposal ("移动互联网信令数据关联分析方法研究") builds a SAPP platform plugin that parses LTE GTP-C control-plane signaling (S11 interface) to extract session/TEID identifiers and correlate them with user-plane IP sessions, explicitly to resolve subscriber identity that is otherwise unavailable because "the data plane and signaling plane are separated" in mobile networks. A later revision of the same proposal confirms the parser and the ID-to-IP correlation logic are both implemented as SAPP plugins.
zhanghongqing/knowledge-log is a Flink ETL pipeline that builds relationship graphs from DNS and connection logs (Ip2IpGraphProcessFunction, DnsGraphProcessFunction/DnsRelationProcessFunction) and persists them to ArangoDB (a graph database) plus ClickHouse, indicating infrastructure for graph-based flow-correlation analysis (e.g. clustering related proxy/VPN infrastructure by IP-to-IP or DNS relationships) separate from the per-packet DPI engine.
A MESA Lab researcher (name redacted) maintains a personal repo that archives a named toolset for Tor/dark-web deanonymization: an "IP protection software" folder (TorIM2, create_IM_ID_info), a "Tor clearnet domain collection" archive recording exit-node destination traffic, an "accelerated anonymous channel" tool, a "dark web service traceback" archive labeled as an old deanonymization attack, and a "dark web hidden node detection" collection of harvested hidden relays.
Internal engineering doc describes "整形平台" (internally versioned as "Nirvana" — GitLab repos reshape/nirvana_client and reshape/nirvana_platform), a Kafka/Redis-backed, fully-async C/S platform sitting downstream of sapp that performs cross-session, cross-link correlation to reconstruct complete file content, VoIP audio, and session metadata. Its explicit "单向流对准" (single-direction flow alignment) feature reunites the client-to-server and server-to-client halves of one session when they were captured separately (e.g. asymmetric routing, or a passive mirror tap seeing only one direction).
A 3-month E21 (Ethiopia) investigation into unreliable YouTube blocking found two independent, structural detection gaps: (1) at IGW (international gateway) sites, load-balanced multi-path routing causes TSG to see only single-direction (C2S-only or S2C-only) traffic for a meaningful fraction of sessions (~5-10% baseline, worse under load), breaking stateful app-ID/deny; (2) under peak load, NPB boards' CPU-triggered self-protection "bypass" (fail-open passthrough) hit ~50% of new sessions at the busiest site (Bole-IGW, ~86k new sessions/sec at 7.8Gbps/NPB), and TSG explicitly deprioritizes bypass protection for non-HTTP/SSL traffic to preserve web blocking under load.
A Pakistan deployment (internally 'WMS-UTR'/'P19', sites including Karachi and Islamabad) ingests full RADIUS AAA and NAT-log traffic from four named mobile carriers -- Jazz, Zong, Ufone, and Telenor -- to correlate every monitored network session with the subscriber's phone number (RADIUS 'Calling Station ID'), in near-real-time (target latency <1s, degrading to ~1hr under 8x traffic growth). The correlation engine, internally called 'CyberNarrator' (网络叙事者), writes resolved identity objects directly into security-policy objects every ~10 seconds so blocking policy can reference specific subscribers by phone number. Base-station location data is also parsed from Radius traffic.
A dedicated, actively-maintained TSG_OLAP Flink streaming job (galaxy/tsg_olap/sip-rtp-correlation, 200+ commits through late 2024) fuses one-way SIP signaling records with their corresponding RTP media-stream records into unified bidirectional VoIP call records in real time, keyed by address/session state with configurable expiry -- a production system for reconstructing complete VoIP call metadata (parties, duration/state) from separately-captured signaling and media flows.
A CAS Institute of Information Engineering (MESA Lab, advisor and PhD student names redacted) mid-term report describes three original, experimentally-validated Tor hidden-service deanonymization techniques under the name 'AlterCell': a watermarking attack exploiting a newly-discovered Tor cell-integrity-validation logic flaw that is 18.8x faster than the prior published signal-cookie attack and evades the Vanguard defense (100% TPR / 0% FPR across 341,721 monitored real Tor links over 5 days); a remote / rendezvous-point-side website-fingerprinting method for server-side hidden-service correlation (up to ~90% accuracy using a quadruple-loss metric-learning model); and an attack on Tor's SBWS bandwidth-measurement tool that inflates the researchers' controlled nodes' consensus bandwidth to raise their odds of being selected as a target's guard node. The framework was run against the live Tor network and deanonymized 113,670 hidden services between 2024-01-11 and 2024-03-11.
A MESA Lab experiment re-identifies individual users across time solely from which domains they visit, using one-hot/frequency "domain access vectors" (Jaccard, overlap-coefficient, cosine, Euclidean) computed against real production HTTP logs pulled from a ClickHouse table named tsg_galaxy_v3.connection_record_log_http_domain. The best-performing configuration (Alexa top 100-10000 domain list, overlap coefficient) distinguished the same user's adjacent-hour sessions from other users' sessions with high separation (~0.19 vs ~0.88 average difference).
The galaxy/tsg_olap/log-stream-voip-relation repo is a Flink/Kafka streaming job (VoIpRelationTopology) that correlates SIP signaling flows with their corresponding RTP media flows across VSYS (virtual-system) boundaries, reconstructing complete VoIP call sessions from separately-observed signaling and media traffic.