Watching with subs is a reading activity. If you have subtitles on all the time, you have less, maybe even no time to practice listening to the target language. If you practice reading more than listening, and you're not fluent yet, your listening ability starts lagging behind your reading ability. In return, having target language subtitles on is going to help you learn a lot more quickly.
If developing good accent and listening ability is important for you, always watch raw and turn the subs on only when you hear an unknown word. If getting a native-like accent is a priority, you should also cut down on any reading practice and learn entirely through listening. If you're not that concerned about it, you could keep two types of TV shows, one for watching raw and one for watching with subs so that you still have some dedicated time to learn through listening. And if you don't care about accent at all, maybe if you're learning your TL only to be able to read TL books or watch TV shows with subs, then it's okay to have them on all the time.
The amount of passive immersion you're doing also plays a role. You can allow yourself to watch more content with subs if you listen a lot in the background.
If your listening ability is underdeveloped by the time you reach fluency, catching up will take more effort than if you listen a lot from the beginning. But the process is pretty intuitive, to develop your listening ability all you have to do is listen a lot over a period of time.
In my experience neglecting listening as a beginner can be crippling for your speaking. Before I started learning English through immersion, I had spent several years not listening to any native English content and being forced to prematurely output at school. I was able to improve my listening relatively quickly after starting to immerse. Today I can understand everything I hear in English, maybe except some strong accents. However, when I speak, I don't know how to pronounce words correctly, and often I'm lost trying to recall or guess the right pronunciation. I don't have the same experience in Japanese because I took a listening-heavy approach from the start.
Lastly, I notice that it's hard to let go of TL subtitles if you already have them downloaded. A solution is to decide what shows you watch raw and what shows you watch with subs. If you watch something raw, don't download subtitles at all. I also recommend watching ongoings. When you watch an ongoing anime, you download a new episode every week. Because the episode is still new, there are no subtitles for it yet. Your only option is to watch raw.