EgoNight
Towards Egocentric Vision Understanding at Night with a Challenging Benchmark
1INSAIT, Sofia University 2East China Normal University 3HKUST(GZ) 4Nankai University 5Fudan University · *equal contribution †corresponding author
Alexander Nevsky Cathedral, Sofia: the same walk by day and by night. Drag to compare.
→ / Space: next ← back F: fullscreen O: contents
Overview
Existing egocentric benchmarks focus on daytime. EgoNight asks how today's multimodal LLMs cope when the lights go out, with day–night aligned videos and question answering on both.
Why day–night alignment
At night, people and GPT alike struggle to write good question–answer pairs. So each Synthetic and Sofia night video has a daytime twin, and answers are written with the day video in view.
- Same route, same actions, aligned in time. Drag the divider to compare.
- EgoNight-Synthetic: one Infinigen scene rendered in Blender twice, with the lighting edited for night.
- EgoNight-Sofia: each route is recorded by day, re-recorded at night with the day video as reference, then trimmed to match.
- EgoNight-Oxford: night-only clips selected from Oxford Day-and-Night.
Annotation Pipeline
Nighttime Captioning
GPT-4.1 generates scene descriptions from nighttime clips.
Question Generation
Diverse candidate QA pairs created per task type.
Day-Augmented Answer Synthesis
The daytime reference is used to improve answer accuracy.
Human Refinement
All 3,658 pairs manually verified and refined (300+ hours).
Paired QA types (8)the same question is asked on the day and the night video
Unpaired QA types (4)night-specific, or impractical to pair
Dataset Info
EgoNight Dataset
160 videos, 70 day–night aligned pairs, combining synthetic and real-world sources.
EgoNight-VQA
3,658 human-verified QA pairs across 12 diverse question types.
Empirical Evaluation
Benchmarking of 10+ MLLMs reveals significant day–night performance gaps.
Dataset at a glance
| Subset | Videos | Day–night | Source |
|---|---|---|---|
 Synthetic | 100 | 50 pairs | Blender + Infinigen |
 Sofia | 40 | 20 pairs | Real world, Sofia |
 Oxford | 20 | night only | Oxford Day-and-Night |
Dataset Info: 12 Question Types
Perception types (object, text, scene sequence) and reasoning types (navigation, counting, non-common-sense). Lighting recognition, lighting dynamic and non-common-sense are new question types.
VQA Benchmark
Open-ended QA accuracy (%) on night videos, for each subset and difficulty level.
Overall scores
GPT-4.1 leads overall, with Gemini 2.5 Pro at 30.6: both closed-source models are ahead of every open model overall.
InternVL3-8B is the best open-source model, above the 72B Qwen2.5-VL (19.0).
EgoGPT, built for egocentric video, scores 14.3 overall, well below the best open model.
VQA Benchmark: Day vs Night
Accuracy (%) on the paired question types, asked on the day video and on its aligned night video.
Values as printed in the paper's day–night gap figure. Object: object recognition, Text: text recognition, Spatial: spatial reasoning, Scene: scene sequence, Navi: navigation, Count: counting of static, NonC: non-common-sense reasoning, Action: action recognition.
points overall, from 25.5 by day to 17.4 at night.
Perception suffers most. Object recognition drops the most on both subsets (−19.6 and −15.5 points); navigation and counting, which lean on reasoning, drop only 2–5 points.
The new types are the hardest: lighting recognition, lighting dynamic and non-common-sense score below the other question types.
Depth Estimation
Egocentric depth on EgoNight-Synthetic, where Blender gives ground-truth depth. Most methods get worse at night.
Day–Night Correspondence Retrieval
Given a night clip, find its daytime counterpart (N→D): the same place (spatial) or the same moment (temporal).
Fine-tuning
Qwen2.5-VL-7B fine-tuned on EgoNight, accuracy (%) on night videos.
Which part to tuneEgoNight-Sofia + Oxford, 7:3 train / validation
Synthetic to realtrain on Synthetic, test on Sofia + Oxford
Full fine-tuning is best
+9.21 points over zero-shot, the best overall, though accuracy stays low.
Vision encoder tuning
Mainly helps perception-heavy tasks, such as object and text recognition in low light.
LLM tuning
Improves perception and reasoning: language priors matter for nighttime understanding.
Synthetic-to-real transfer
Synthetic-only training transfers to real night video (+5.74 points), so synthetic data is useful.
Take-away Message
EgoNight is the first comprehensive benchmark for nighttime egocentric vision, addressing a critical gap between laboratory evaluations and real-world deployment conditions.
The illumination gap is real and large. State-of-the-art MLLMs degrade by 25–33% when moving from day to night, revealing a fundamental limitation.
Aligned synthetic + real data is a viable strategy. The strong correlation (r = 0.9359) validates using synthetic data for nighttime adaptation and evaluation.
Fine-tuning helps, but the problem is far from solved. Even after fine-tuning, models achieve only ~20% accuracy, leaving substantial room for improvement.
Future directions
- Illumination-aware pretraining for egocentric models
- Nighttime-specific data augmentation strategies
- Cross-modal (3D + audio + vision) approaches for low-light understanding
- Larger-scale real-world nighttime egocentric datasets
- Further adversarial cases
Acknowledgement
Funders & collaborators
Open-source tools
Contact
