The answer is obvious. Do active immersion.
Even if you have headphones on all day, without active immersion you can't expect any improvement. The whole point of passive immersion is to get additional exposure to the sentences you already have learned to some extent.
Especially in early stages during passive immersion you generally consume exactly the same content that you learn from during active immersion. During passive immersion your brain should be exposed to what you've been working on actively. If you're doing only passive listening, your brain is going to filter the information out as background noise.
During active immersion you need to give your full attention to the immersion and try to understand as much as possible. Make flashcards for the unknown bits. Then, when you're going to re-listen to the same content passively, listening is going to help you solidify the knowledge.
Passive immersion only helps to the extent that you are also doing active immersion. If you skip active immersion, you won't understand what's going on. But if you listen to familiar content, it is easier to follow the story. Even if your attention moves to something else, when you move it back, you can still understand what's going on because you already know the plot.
It is possible to learn a language without passive immersion, but it is almost impossible to do without active immersion. You need to actively immerse and look up as many words as you can while doing so.
If you do no active immersion, passive immersion does little to nothing. If you are doing lots of active immersion each day, then passive immersion starts to become very beneficial.
So what I recommend everyone to do is
watch anime, movies and dramas actively
and then put each episode through impd to get condensed audio.
Listen to the condensed audio in the background until
impd archives it.
The process should not require much time,
you only need time to watch a few episodes every day.