Generative AI at the Intersection of Vision and Language: Advances in Synthesis, Editing, and Multimodal Understanding

Generative AI is reshaping how we synthesize and edit visual content while bridging the gap between vision and language. In this talk, I will present our recent contributions that explore this intersection through advanced models and benchmarks. These includes a method that incorporates text-conditioned adapter layers into pretrained GAN inversion networks, enabling precise and intuitive text-driven editing of real images. Building on this, our work on text-based neural video manipulation disentangles content and motion, allowing for coherent and semantically meaningful video edits. Expanding beyond text, our SonicDiffusion framework introduces audio-driven image generation and editing by transforming audio features into representations compatible with diffusion models.To complement these efforts, we have also created robust benchmarks for evaluating multimodal systems, assessing their linguistic and temporal grounding, or compositional generalization and reasoning abilities. Collectively, these efforts advance the capabilities of generative models, fostering more intuitive and flexible multimodal AI systems.