I know exactly what you are talking about. My best guess is that VLC is trying to debayer the RGB8 output of the camera. Since the mono is encoded as RGB8, the software assumes a bayered image. Other manufacturers encode as Y800 which is easier to deal with. I am still trying to figure out how SharpCap is using the directshow output and getting mono. Ffmpeg does the same thing. In ffmpeg, I use something similar to the code listed below. This may have some un-needed commands as well as no inclusion of exposure controls. It's the best I can read from some notes I have scribbled. I have not tested it in the telescope yet and I don't know if the "gray" option is correctly ignoring the bayer or applying post debayered graying (which would be wrong). I'm not currently using the ASI120MM for streaming, but hope to in the future when I have more time to work out the kinks.
I wonder if the output format can be changed with the SDK? I have glanced through and it seems the *.dll probes the camera for its settings, so I am guessing it is coded in firmware. Maybe
---
eo="ASI120MM Camera (ZWO Design)"="Microphone (High Definition Aud" -vcodec h264 -pix_fmt yuv420p -vf format=gray -b 3000k -bufsize 3000k -acodec aac -strict experimental -f flv -max_interleave_delta 0 "rtmp://a.rtmp.youtube.com/live2/xxxx-xxxx-xxxx-xxxx"
