Operating 3 ASI6200MM cameras simultaneously in 16bit

ldjb

I just want to check to see if anyone can shed some light on my problem before I buy any new hardware.

What I want to do: Run 3+ ASI cameras simultaneously in 16-bit mode.

How I do this: I use the zwoasi python package to cause a camera to take a series of 10-second exposures. I then run multiple versions of this script simultaneously so that all 3 cameras collect images in parallel.

The problem: In 8-bit mode, everything works exactly as it should do. The 3 cameras each collect images at ₁₀ second intervals.

When set to 16-bit, the rate of collection is completely broken. A camera may not collect a single for minutes.

I am trying to understand why this is the case and I would very much appreciate any insight or advice any of you could provide troubleshooting this issue.

The fact that this is fine with 8-bit but broken with 16-bit suggests to me that this is a bandwidth issue. Additionally, there is no problem with running a single camera at 16-bit. The problem only appears when using multiple cameras.

Is this therefore maybe something to do with the total datarate available between the motherboard and the three USB3.0 ports that the cameras are connected to? Or perhaps some other kind of bottlenecking issue?

Currently, my best idea is to buy some USB3 PCI expansion cards on the hope that this will reduce bottlenecking (Assuming this is even a bottlenecking problem!)

Any help/suggestions/discussion on this issue would be greatly appreciated.

Many thanks,

ldjb

I forgot to mention, the fact that the images are very large (62MP) is another reason why I suspect there may be a bottleneck.

Kring

Its most likely either bandwidth as you suggested. Or you are initiating the next picture before the camera is in a ready state (which could also be associated with bandwidth and the file not transferred yet). Can you stagger the images by 3.3 seconds? Rotating them so after each pic, it has the full usb bandwidth for 3 seconds? Or can you add a pause or a camera ready check to your code to confirm image has transferred and camera can shoot before snapping the next?

i would suggest doing all three atleast for now to help troubleshoot, and get it working. then you can start the process of elimination till you hit failures.

gotta ask out of curiosity, what are you doing with these?

ldjb

Thanks for the response.

It seems that we have had a similar train of thought. For testing, my objective is to get 2 cameras to run in parallel. I will enumerate the things I have tested so far:

Run the two control processes with a 5s offset: No improvement. Still results in a highly disrupted observation cadence.

Write the images to different drives: The PC I am using has an NVME drive and a SATA drive. I tried running the code such that each of the 2 cameras saved to each of these locations. No improvement. This suggests to me that it isnt a write issue, but my understanding is limited in this area. Perhaps saving to different drives does not avoid some writing bottleneck.

Run 1 camera in 8bit mode and the other in 16bit mode: Yes this solves the problem, although im not too sure what this tells me.

I will provide further updates as I find out more. Any comments/suggestions are welcomed!

(To answer your question, the system is for very large FoV detection of optical transients)

Kring

the drive split typically won't matter because it's running on the same BUS. however, an NVMe drive (assuming M.2) and a name-brand model - you should be plenty sufficient. we're talking 3x116mb for a total of 360mb's of bandwidth, when the bus in PCI 2.0 X4 mode can handle 4000 mb/sec and the drive should be able to write atleast 1000mb/s... so performance wise you are well within spec

Checking in your motherboard BIOS to ensure that PCI is 3.0 X4. by default some motherboards limit to X2 because you have to disable one of the channels in order to combine for full speed X4.

Also run a quick benchmark like this: https://www.techspot.com/downloads/6014-as-ssd-benchmark.html so you can eliminate the PC, BUS, BIOS, DRIVE as a bottleneck.

Lastly, try to bake in 10 second pauses between subs, just to see if it's reset/ready timing

ldjb

Thanks again for the response.

That is an extremely helpful answer, thanks a lot. I haven't had any time to work on it today but I will respond with an update as soon as I do.

ldjb

Here is the latest update.

I am running on Linux, so I cant use the benchmarking tool linked above. Using the built-in Ubuntu "Disks" tool, i benchmarked the drive. The read speed was 3200 mb/s and write speed was 2700 mb/s - so no problems there.

For testing purposes, I tried running the exact same code on a different machine. In this case it was a laptop that has 2 USB3.0 ports. The code ran successfully and 2 cameras were ran in parallel in 16bit mode. Because there are only 2 ports, I cannot try running the third camera to see if this causes issues.

So the question is, what is different between the laptop and the desktop?

Running the command "lsusb" on the desktop shows (amongst other things):

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Running the same command on the laptop shows (amongst other things):

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

This suggests to me that the three USB3.0 ports on the desktop are being controlled by a single "Linux Foundation 3.0 root hub", whereas on the laptop the two USB3.0 ports each have their own "3.0 root hub". Unfortunately the laptop doesnt have a third USB3.0 port, so I cannot test if a third camera would cause problems.

The next step in my testing is to install a PCIe card with four USB3.0 ports on it, which should arrive in the next few days. Hopefully running the cameras from a PCIe slot will avoid using the single available "3.0 root hub" and allow the cameras to run in parallel.

I will post another update once I have tried this.

ldjb