As I mentioned here, there's no way to control the volume of individual Metro/Modern UI apps (not unless the app has its own volume control, which is unlikely). Metro apps are all bundled under System Sounds in the Volume Mixer, so it's an all-or-nothing proposition unfortunately.
Edit: Perhaps you can try keeping the main volume high, the System Sounds volume low/muted (which will affect all Metro apps including games), and open YouTube in Desktop IE or other browser.