Are there potential downsides of using audio-only content like podcasts as the main source of free-flow immersion?
There's really no need to confine yourself to audio-only content, such as podcasts, for free-flow immersion, as audio-visual content like movies and TV shows will be far more effective.
If you primarily learn through audio and visual materials, you will develop strong listening skills. This proficiency will be of great value in the long term. If you acquire your target language through listening before starting to read, the way your brain will process the language will be more in line with that of native speakers. However, native speakers don't limit themselves to listening alone. By excluding visual media you are lowering your comprehension for no reason at all.
Audio-visual content gives you all the benefits of audio-only content, and your comprehension will be much higher thanks to the visual cues. On the other hand, podcasts can be difficult to understand, particularly for beginner learners. They lack transcripts, visual context and anything else that could help you understand what is being said. In my experience, podcasts are virtually indistinguishable from white noise until you reach a solid intermediate level. So, there are huge downsides to relying solely on audio-only content for free-flow immersion, as it will drastically slow down vocabulary acquisition. I would recommend audio-visual content instead.
I deem audio-only free-flow immersion to be comparable to passive immersion. It is unlikely that during listening one can maintain the same level of focus and attention as when watching. While it is acceptable to use podcasts for passive immersion, as long as you understand the language used, it is advisable to choose a medium that you have previously watched actively.
The rate of acquisition also depends on what you do during intensive immersion. If during intensive immersion you read novels, comic books, or watch movies and TV shows, while looking up words and mining sentences, then you will be learning vocabulary faster, and the downsides of using audio-only content during free-flow immersion will be less significant.