LLaVA

Best of 2024 in Vision [LS Live @ NeurIPS]

Dec 22, 2024 · 55:46

Isaac Robinson and Peter Robicheaux from Roboflow and Vik Korrapati from Moondream recap the biggest shifts in computer vision in 2024, naming Sora as the year's most significant paper despite being a blog post, and highlighting its replication efforts (OpenSora) that use MAGViT V2 and diffusion transformers. They detail how SAM 2 extends the SAM approach to video with a memory bank and real-time performance, and how DETR-based detectors (RT-DETR, LW-DETR, DeFine) now surpass YOLOs on COCO by 4.6 AP at the same latency. Peter explains the CLIP-blind phenomenon: MMVP shows leading VLMs (ChatGPT, Gemini, LLaVA) fail at fine-grained tasks like reading clock hands, but models like PaliGemma 2 and AimV2 improve by incorporating pixel-level understanding and autoregressive image reconstruction, with AimV2 approaching 60.2 mAP on COCO. Vik introduces Moondream's 0.5B parameter pruned VLM and demonstrates a grounded chain-of-thought approach that dramatically improves sample efficiency for analog gauge reading, reducing the need for millions of synthetic examples.

[Paper Club] Molmo + Pixmo + Whisper 3 Turbo - with Vibhu Sapra, Nathan Lambert, Amgadoz

Oct 13, 2024 · 1:12:59

This episode covers two papers: AI2's Molmo open-source vision-language models and OpenAI's Whisper Large V3 Turbo. For Molmo, the key claim is that high-quality, audio-annotated data (Pixmo) enables models as small as 1B to match GPT-4V on academic benchmarks, with the 72B variant outperforming GPT-4o, Gemini 1.5, and Claude 3.5 Sonnet. The team avoided distilling proprietary models by having annotators describe images in speech for 60-90 seconds, then transcribing and augmenting the captions. For Whisper Turbo, OpenAI pruned the decoder from 32 to 4 layers and continued pre-training on 10 million hours of multilingual transcription data, making it 1.78x smaller than Large V3 with minimal word-error-rate increase. The model is faster and supports real-time chunk-based decoding, unlike English-only DistilWhisper. Nathan Lambert and Amgadoz provide commentary on data strategies and benchmarking nuances.

Personal AI Meetup - Bee, BasedHardware, LangChain LangFriend, Deepgram EmilyAI

Apr 6, 2024 · 58:54

This episode features Damien Murphy of Deepgram, Ethan of Owl/Bee, and Harrison of LangChain demonstrating how to build personal AI with real-time voice bots, wearable life-recording devices, and memory-enhanced journaling apps. Damien shows building a voice bot with subsecond latency using Deepgram, OpenAI, and open source code, costing about 6.5 cents per five-minute call. Ethan presents his Owl wearable that continuously records audio, triggers actions via hot word 'Scarlett,' and discusses challenges in adding vision and open source adoption. Harrison introduces LangFriend, a journaling app using conversational, semantic, and knowledge graph memory, referencing the Generative Agents paper for recency and importance weighting. The episode also highlights open source projects Whomane, Friend, and ADeus, arguing that hardware, voice, and memory are all necessary components for personal AI.