If you watch with English and Japanese subtitles on the screen at the same time,
consider having both subtitle tracks hidden most of the time.
mpv you can press v to toggle subtitle visibility.
When you feel like you didn't understand something,
turn the subs on temporarily.
Mpvacious has key bindings
that let you toggle secondary subtitles as well,
or you can have them appear on mouse hover.
You can approach this question at 2 levels.
Training your listening ability.
Displaying both Japanese and English subtitles simultaneously would occupy a significant portion of the screen, making the subtitles difficult to ignore. If you have them visible all the time, it is likely that you will end up reading them. It may happen unconsciously, automatically, accidentally. Even if you catch an outline of a sentence in the corner of your eye, it could give you a hint and disrupt your listening practice. If you're genuinely committed to improving your listening skills, consider hiding all subtitles. This is the only way to ensure that you're not cheating.
Increasing your overall comprehension. To ensure effective learning, it's crucial that you don't read English subtitles. If you do, and if you don't pay attention to the Japanese subtitles, your progress will be zero. Try to understand the Japanese subs first.
If you must use English subtitles, try to make them less accessible, such as requiring a button press or mouse movement to reveal them.
Then consider whether English subtitles can help you, or if it is better to get rid of them completely. Prioritize understanding the Japanese subtitles first, and only opt for English subtitles if necessary. As I previously discussed in Should I put English translations on Bilingual TSCs, relying on translations doesn't help in language acquisition. While they may help you understand a rough meaning of a sentence, they don't aid in understanding how ideas are expressed in Japanese, and how grammar works.
The most harmful aspect of reading English subtitles is that one may feel like they comprehend the original text when, in reality, they understand only the translated meaning. Furthermore, using English subtitles carries the risk of creating false associations between English words and Japanese words with similar meanings. This can lead to biases in your Japanese comprehension.
In conclusion, it is OK to keep Japanese subtitles visible, but English subtitles are better kept hidden. Hiding both English and Japanese subtitles is good for improving your listening abilities. And if you hate going forth between turning subs on and off, consider watching some shows fully raw and other shows with Japanese subs always on.