Actually, found another thread, it describes exactly the same issues. So posting my fix here - for anyone who will face similar issues in the future
How to enable ZWO ASI Camera drivers on Windows 11 (assuming it can see the camera in devices)
- Remove any ASI Camera drivers from your system that you have installed previously
- Download the driver installer from this message: https://bbs.astronomy-imaging-camera.com/d/13554-asi-drivers-for-windows-on-arm-ie-mac-m1-silicon/55
- Run the installer - it will extract driver files in a folder (Program Files by default for me)
- Run your Windows 11 in the Recovery Mode and disable driver signature enforcement
- While in the Recovery Mode, connect your camera to the computer, go to the Device Manager
- In the Device Manager, find your camera, select it and click Update Driver -> Browse my computer for drivers -> select arm folder under the folder from step 3
- Windows will show you a warning. As soon as you confirm installing unsigned drivers, the drivers will be set
Hopefully ZWO issues an official separate driver package for ARM