libASICamera2 for aarch64?

Vector

@tech@zwo#46720

Hello again. I have performed the requested tests using the new library file you supplied on raspberry pi 4, aarch64 / armv8. Tests were performed with binaries compiled against libASICamera2 headers at v1.18, as it is the nearest available public header for libASICamera2.so.

Summary

Exposures at 8288 x 5644 @ 16bit all failed, both for test_gui2_video and test_gui2_snap.

Exposures at certain lower ROIs succeeded in test_gui2_snap (e.g., 8000 x 4000 @ 16bit).

Test Results

Test results can be seen here:
https://gist.github.com/vector-kerr/3d0e0ccc7b403f4921fb16cb3de5cb19

The files in the GitHub Gist are:

info.log - a standard dump of uname, lsb_release, etc
test-gui2-snap-8288x5644x16.log - A log for the run of test_gui2_snap at 8288 x 5644 pixels, 16-bit
test-gui2-snap-8000x4000x16.log - A log for the run of test_gui2_snap at 8000 x 4000 pixels, 16-bit
test-gui2-video-50ms.log - A log for the run of test_gui2_video at 8288 x 5644 pixls, 16-bit, with 50ms exposures
test-gui2-video-1000ms.log - A log for the run of test_gui2_video at 8288 x 5644 pixels, 16-bit, with 1000ms exposures
sequence-snaps.log - A log for the run of a version of the demo program I wrote ages ago (shared in one of my earlier posts), which takes a list of width and height combinations, and attempts to capture one picture in each resolution. This allows me to create a table of resolutions that do / don't expose successfully.

The table below shows the summary results of sequence-snaps.log. It seems that as the total number of pixels increases, the exposure failures also increase.

Note on Possible 'Hardware Limitation' Comment

I thought I would re-iterate this because it is important. This problem cannot be due to a hardware limitation, because when I use the 32-bit version of the raspberry pi operating system (armv7) and the corresponding libASICamera2.so file, the full-size full-bit-depth capture at 8288x5644@16bit succeeded. That is to say, on armv7 using the exact same hardware, I can successfully take full-resolution full-bit-depth images.

Image capture only fails when running aarch64 with the armv8 libASICamera2.so file. I mention this in my post on 1/Apr/2021, and 9/Apr/2021. The pi, camera, and cable used is the same in all tests.

Vector

Further test results at different resolutions with the sequence snap program (scroll right in each batch box to see all results):

Batch 1

7400 x 2800 = SUCCESS    7400 x 3200 = SUCCESS    7400 x 3600 = SUCCESS    7400 x 4000 = SUCCESS    7400 x 4400 = SUCCESS    7400 x 4800 = FAILURE    7400 x 5200 = FAILURE    7400 x 5600 = FAILURE    7400 x 5644 = FAILURE    
7600 x 2800 = SUCCESS    7600 x 3200 = SUCCESS    7600 x 3600 = SUCCESS    7600 x 4000 = SUCCESS    7600 x 4400 = SUCCESS    7600 x 4800 = FAILURE    7600 x 5200 = FAILURE    7600 x 5600 = FAILURE    7600 x 5644 = FAILURE    
7800 x 2800 = SUCCESS    7800 x 3200 = SUCCESS    7800 x 3600 = SUCCESS    7800 x 4000 = SUCCESS    7800 x 4400 = FAILURE    7800 x 4800 = FAILURE    7800 x 5200 = FAILURE    7800 x 5600 = FAILURE    7800 x 5644 = FAILURE    
8000 x 2800 = SUCCESS    8000 x 3200 = SUCCESS    8000 x 3600 = SUCCESS    8000 x 4000 = SUCCESS    8000 x 4400 = FAILURE    8000 x 4800 = FAILURE    8000 x 5200 = FAILURE    8000 x 5600 = FAILURE    8000 x 5644 = FAILURE    
8200 x 2800 = SUCCESS    8200 x 3200 = SUCCESS    8200 x 3600 = SUCCESS    8200 x 4000 = SUCCESS    8200 x 4400 = FAILURE    8200 x 4800 = FAILURE    8200 x 5200 = FAILURE    8200 x 5600 = FAILURE    8200 x 5644 = FAILURE    
8288 x 2800 = SUCCESS    8288 x 3200 = SUCCESS    8288 x 3600 = SUCCESS    8288 x 4000 = SUCCESS    8288 x 4400 = FAILURE    8288 x 4800 = FAILURE    8288 x 5200 = FAILURE    8288 x 5600 = FAILURE    8288 x 5644 = FAILURE

Batch 2

7400 x 1000 = SUCCESS    7400 x 2000 = SUCCESS    7400 x 3000 = SUCCESS    7400 x 4000 = SUCCESS    7400 x 5000 = FAILURE    7400 x 5644 = FAILURE    
7600 x 1000 = SUCCESS    7600 x 2000 = SUCCESS    7600 x 3000 = SUCCESS    7600 x 4000 = SUCCESS    7600 x 5000 = FAILURE    7600 x 5644 = FAILURE    
7800 x 1000 = SUCCESS    7800 x 2000 = SUCCESS    7800 x 3000 = SUCCESS    7800 x 4000 = SUCCESS    7800 x 5000 = FAILURE    7800 x 5644 = FAILURE    
8000 x 1000 = SUCCESS    8000 x 2000 = SUCCESS    8000 x 3000 = SUCCESS    8000 x 4000 = SUCCESS    8000 x 5000 = FAILURE    8000 x 5644 = FAILURE    
8200 x 1000 = SUCCESS    8200 x 2000 = SUCCESS    8200 x 3000 = SUCCESS    8200 x 4000 = SUCCESS    8200 x 5000 = FAILURE    8200 x 5644 = FAILURE    
8288 x 1000 = SUCCESS    8288 x 2000 = SUCCESS    8288 x 3000 = SUCCESS    8288 x 4000 = SUCCESS    8288 x 5000 = FAILURE    8288 x 5644 = FAILURE

Tech@ZWO

Vector

demo.zip

278kB

Attached a file include our test procedure
And could you check the CMD：cat /sys/module/usbcore/parameters/usbfs_memory_mb
recommend to set 200M

Vector

@tech@zwo#46749

Test Procedure

@tech@zwo#46749 Attached a file include our test procedure

Here is a GitHub Gist with the following log files:

The output of the commands in test-procedure.log which match the commands that you run in your demo_log_8288x5644_16bit.txt.
- The run of the demo program at 8288x5644@16bit is separated into its own file at test-run-at-full-resolution-16-bit-depth.log as the log output is large. This is the same as the last command that you run in your demo log file.
- I should note that this test procedure is very similar to tests that I have already performed. I hope that you have been reading the previous logs and test runs that I have been performing, as they take some time to perpare and run.
Additionally, I have included the output of a separate test run at 8288x5644@8bit in the file test-run-at-full-resolution-8-bit-depth.log
Additionally, I have included the output of a separate test run at 4144x2822@16bit in the file test-run-at-half-resolution-16-bit-depth.log

https://gist.github.com/vector-kerr/9140e1cbe15f8990c3dcde61cd371065

USBFS Memory

@tech@zwo#46749 And could you check the CMD：cat /sys/module/usbcore/parameters/usbfs_memory_mb
recommend to set 200M

Yep. I did that 2 weeks ago:

Vector And I have also verified that the USBFS Memory Allocation is set as required:

pi@raspberrypi:~ $ cat /sys/module/usbcore/parameters/usbfs_memory_mb
200

And I've also done that today:

pi@raspberrypi:~ $ date
Sat 29 May 14:11:21 AEST 2021

pi@raspberrypi:~ $ cat /sys/module/usbcore/parameters/usbfs_memory_mb
200

New Raspberry Pi 64 Bit Release

There is a new release of Raspberry Pi 64 Bit.

It is named raspios_arm64-2021-05-28 and is located here:
https://downloads.raspberrypi.org/raspios_arm64/images/

I will be downloading and installing it and trying some tests on that new version while I am waiting to hear back from you.

Where We Are At

I believe that I have very fairly demonstrated in detail that:

The hardware (raspberry pi, camera, cable) all work independently
Your own demo applications demonstrate that I can take 8288x5644@16 images on raspberry pi 32 bit successfully
Your own demo applications demonstrate that exposures at 8288x5644@16 images on pi 64 bit always fail

I have tried performing tests on 4 x 64-bit versions of raspberry pi, a 32-bit version of raspberry pi, and a 64-bit linux machine to determine the common factor between the exposure failed message.

I have used your snap and video demo programs, as well as my own test programs against your API, with versions 1.16, 1.17, and 1.18 of your SDK, as well as custom libASICamera2.so files which you have recently supplied.

I have provided an abundance of log files with all requested output, as well as extra information which could be useful for debugging.

After a brief search, it appears that I am not the first person to experience exposure failures with ZWO cameras.

I purchased the set of equipment that I currently own (64-bit raspberry Pi, ASI294MM-Pro) because:

The pi is low-power and portable, which is ideal for performing astrophotography in remote locations
The ASI camera has a high resolution, high bit depth, and very sensitive sensor
ZWO supply 64-bit ARMV8 linux drivers for ASI Cameras, which (should) allow me to use the camera on a 64-bit operating system which allows observatory applications to operate with more than 4GB of memory when required
- this was identified as important because under a 32-bit operating system I was receiving out of memory exceptions.

As it currently stands, I have spent countless hours (days?) performing test after test, submitting logs, writing operating systems to SD cards, installing updates, compiling source files, writing test applications, and all the rest, in order to use the product that I purchased in the way that I intended to use it when I purchased it.

Ultimately, determining the cause of this problem and resolving it will benefit me because I will be able to use the camera for its main purpose - to take photos - but it will also benefit you, because you will have a more stable and reliable product to offer consumers. Resolving this issue will also demonstrate to your customers that you are interested in supporting your own products.

If somebody purchased a Canon camera and it failed to take photos at its full resolution, it would be considered a significant issue for their product. I am almost baffled by the fact that I seem to have to push to come to a resolution for this issue, because not only is this camera failing to perform its primary job (taking photos!), there is no other method to control this camera or retrieve images from it other than through USB. If it was a point-and-shoot DSLR at least the push-button could be used to take photos, and the SD card could be removed to get the photos off it. With this camera, that is not an option.

As it currently stands, this camera with the supplied drivers does not perform the advertised function of taking photos at full resolution (8288 x 5644 @ 16-bit resolution) on an operating system with supported drivers (armv8) using compatible hardware (one device connected directly to a USB 3 port using a USB 3 cable) using supplied testing software (your snap and video demo programs).

Next Steps

I am happy to continue to perform tests and work with you to come to a resolution to this problem, however there must be some kind of substance to these tests. Continued test runs of the snap and video demo programs must have some kind of demonstrable purpose. The tests that I have performed over the last 2 months have very clearly shown that the issue lies at a minimum specifically with the ARMV8 / AARCH64 system.

There is exhaustive evidence above suggesting that there may either be a problem with the the camera driver, as the hardware is entirely capable of working correctly under specific circumstances. This may extend to the camera firmware, but I cannot make any claim either way here as I do not know how the firmware has been programmed or operates.

Under a 64-bit armv8 operating system, the failures experienced appear to directly correlate to larger file sizes that need to be transferred. If the bit depth is reduced or the image dimensions are reduced, then exposures seem to succeed more frequently.

Please let me know how you intend to proceed from here in order to resolve this exposure failure issue.

Vector

Tagging @tech@zwo, in case you missed my response above. (link: Vector)

Tech@ZWO

Vector
Thanks for the detailed and professional test
We have ruled out the usbfs possibility

I have tried performing tests on 4 x 64-bit versions of raspberry pi
=> Did you mean tried on 4 version OS? if you got another raspberry pi4，strongly recommend you can try on it and would it possible for you to upload your whole OS to us if the “last try” with no lucky result

which (should) allow me to use the camera on a 64-bit operating system which allows observatory applications to operate with more than 4GB of memory when required
=> The 32-bit should be no problem as well，ASIAIR pro is based on arm v7 32bit, no memory error when running ASI 6200MC live stack，that would ask much more memory than ASI294MM，we recommend you build a 32-bit version for temporary use

Vector

Hello @tech@zwo#46895,

OS Versions

=> Did you mean tried on 4 version OS? if you got another raspberry pi4，strongly recommend you can try on it

I have one Raspberry Pi v4 device. I have tried running libASICamera2.so for armv8 using the various test and demo programs on four different operating system versions, all mentioned in previous posts. Those operating system versions are:

Raspberry Pi ARM64 - 2020-05-28
Raspberry Pi ARM64 - 2020-08-24
Raspberry Pi ARM64 - 2021-04-09
The version that you uploaded for me to download and install

The only other Raspberry Pi devices that I own are a Raspberry Pi v1, and some Raspberry Pi Zero-W devices. Unfortunately, they are not 64-bit, they do not have a USB 3.0 port, and they are certainly not powerful or fast enough to test with even if they were 64-bit and did have a USB 3.0 port.

Additionally, as it has previously been shown, the Raspberry Pi 4 that I own will happily take full-resolution full-dpi images with the ASI294MM-P when it is running on a 32-bit operating system, which shows that this does not appear to be a hardware issue with the Raspberry Pi 4 or the ASI294MM-P camera, or the cable which connects them. This issue appears to be specific to execution within a 64-bit environment.

Tests have been performed with the OS versions above, including your own OS version, and the failure is consistent across all 64-bit versions of the Raspberry Pi operating system. Therefore, some potential scenarios that exist are:

There is something wrong with all 64-bit Raspberry Pi Operating Systems, or
There is something wrong with the ARMv8 libASICamera2.so driver for 64-bit operating systems, or
There is something wrong with the way that the camera's firmware communicates over USB, which happens to manifest more frequently in a 64-bit environment

Upload OS

@tech@zwo#46895 would it possible for you to upload your whole OS to us if the “last try” with no lucky result

I can do this, however all I do is download the image, perform a system software update, download and compile your SDK, and run your demo program. It is about as vanilla as it can get. I will need to make some space so that I can upload this for you, so I will get back to you on this request when I have done this.

32 Bit Environment

@tech@zwo#46895 which (should) allow me to use the camera on a 64-bit operating system which allows observatory applications to operate with more than 4GB of memory when required
=> The 32-bit should be no problem as well，ASIAIR pro is based on arm v7 32bit, no memory error when running ASI 6200MC live stack，that would ask much more memory than ASI294MM，we recommend you build a 32-bit version for temporary use

As I've mentioned, the 32-bit version of the operating system is fine for taking photos at full-resolution and full-dpi. The memory requirements are for other observatory control and management software, not for the camera. Attempting to reduce the memory footprint of the observatory software is another aspect that I am looking into, however this does not excuse the camera from failing to expose full-resolution and full-dpi images in a 64-bit environment.

While working towards a solution to the exposure failures on 64-bit systems, I have been using a 32-bit version of the raspberry pi operating system, and taking photos at increased binning to reduce their memory footprint so that the software does not crash due to an out of memory exception.

While I appreciate the suggestion, running in a 32-bit environment is not a suitable long-term solution.

Vector

Hello again @tech@zwo#46895

In addition to the responses in the post above, I have prepared an image for you to use.

Image Details

The image is based on the latest official 64-bit version of Raspberry Pi, which is Raspberry Pi ARM64 2021-05-28.
The image has been downsized with PiShrink. When it is extracted onto a card, it should automatically resize itself to fit the card that you install it on.
PiShrink has also compressed the image with gzip, so you will need the right program to extract it first before you can write it to an SD card. On Windows you can use 7-Zip. On Mac and Linux, you should already have access to the gunzip command to uncompress the file.

Things Done To This Image

Updating system packages and installing dependencies for the demo programs:

sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install -y libopencv-dev libncurses-dev

Configuring the Raspberry Pi with sudo raspi-config, including:
- Setting the Language (en_AU.utf-8)
- Setting the Timezone (Australia/Melbourne)
- Configuring the Display Resolution (1920x1080)
- Setting the host name
- Enabling the VNC Server
Downloading and extracting the ASI SDK v1.18 to /home/pi/Downloads/ASI_linux_mac_SDK_V1.18/

Linking the libASICamera2.so library by adding a path config file and rebuilding the library cache:

echo "/home/pi/Downloads/ASI_linux_mac_SDK_V1.18/lib/armv8" > /etc/ld.so.conf.d/asi-1.18.conf
ldconfig

Installing the udev rules by going to /home/pi/Downloads/ASI_linux_mac_SDK_V1.18/lib and running:
```
sudo install asi.rules /etc/udev/rules.d
```
Build the Demo programs by going to /home/pi/Downloads/ASI_linux_mac_SDK_V1.18/demo and running:
```
mkdir -p bin/armv8
make platform=armv8
```
Editing /home/pi/.ZWO/ASIconfig.xml to set <DebugPrint> to 01

Image Download

You can download the image here (~1.6GB):
https://drive.google.com/file/d/1br7623PL6CoU2ZjWLYl5XXe835zIzODE/view?usp=sharing

Usage

Uncompress the image with 7-Zip, gunzip, or something else that can uncompress gzipped files
Write the uncompressed image to an SD card using something like Balena Etcher or dd
Insert the SD card into the Raspberry Pi
Boot the Raspberry Pi
Log in (username=pi, password=raspberry) with ssh or vnc, or use an attached display/keyboard/mouse
Open an xterm window and cd to ~/Downloads/ASI_linux_mac_SDK_V1.18/demo
Run the various demo programs (test_gui2_snap, test_guid2_video, ...) with various settings, for example:
- Full resolution, full bit depth (8288 x 5644 @ 16bit)
- Half resolution, full bit depth (4144 x 2822 @ 16bit)
- Full resolution, half-bit-depth (8288 x 5644 @ 8 bit)

I have also written the image which you can download above to my SD card to confirm that it works, so you shouldn't have any problems with it. If you do, however, please let me know.

Thanks.

Tech@ZWO

Hello
We have download and

armv8-log.txt

16kB

tested on your OS
No exposure failure，attached the test whole log

Vector

I believe that I have made a breakthrough with this problem, and have figured out a temporary workaround.

Given that you were able to run the same operating system and same camera with no issue, I figure that there must be some difference in configuration from one device to the next. I looked at your log files and noticed that your Raspberry Pi v4 was a 4GB model, while mine was an 8GB model. This is interesting because I would have thought that if there was going to be an issue, it would be with the model with less memory, not more.

I started poking around other areas of the system. While doing that, I checked dmesg and noticed the following chunk of output:

[ 1992.819422] usb 1-1.2: new high-speed USB device number 5 using xhci_hcd
[ 1992.925118] usb 1-1.2: New USB device found, idVendor=04b4, idProduct=6572, bcdDevice=32.99
[ 1992.925135] usb 1-1.2: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[ 1992.925146] usb 1-1.2: Product: USB2.0 Hub
[ 1992.926855] hub 1-1.2:1.0: USB hub found
[ 1992.927283] hub 1-1.2:1.0: 4 ports detected
[ 1993.219546] usb 2-2: new SuperSpeed Gen 1 USB device number 3 using xhci_hcd
[ 1993.241026] usb 2-2: New USB device found, idVendor=03c3, idProduct=294e, bcdDevice= 0.00
[ 1993.241040] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 1993.241045] usb 2-2: Product: ASI294MM Pro
[ 1993.241052] usb 2-2: Manufacturer: ZWO
[ 2032.611067] usb 2-2: reset SuperSpeed Gen 1 USB device number 3 using xhci_hcd
[ 2032.782023] swiotlb_tbl_map_single: 94 callbacks suppressed
[ 2032.782034] xhci_hcd 0000:01:00.0: swiotlb buffer is full (sz: 16384 bytes), total 32768 (slots), used 32753 (slots)
[ 2032.782064] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.782447] xhci_hcd 0000:01:00.0: swiotlb buffer is full (sz: 16384 bytes), total 32768 (slots), used 32753 (slots)
[ 2032.782470] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.783196] xhci_hcd 0000:01:00.0: swiotlb buffer is full (sz: 16384 bytes), total 32768 (slots), used 32753 (slots)
[ 2032.784039] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.784803] xhci_hcd 0000:01:00.0: swiotlb buffer is full (sz: 16384 bytes), total 32768 (slots), used 32753 (slots)
[ 2032.784831] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.785286] xhci_hcd 0000:01:00.0: swiotlb buffer is full (sz: 16384 bytes), total 32768 (slots), used 32753 (slots)
[ 2032.785309] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.785682] xhci_hcd 0000:01:00.0: swiotlb buffer is full (sz: 16384 bytes), total 32768 (slots), used 32753 (slots)
[ 2032.785704] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.786175] xhci_hcd 0000:01:00.0: swiotlb buffer is full (sz: 16384 bytes), total 32768 (slots), used 32753 (slots)
[ 2032.786309] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.786691] xhci_hcd 0000:01:00.0: swiotlb buffer is full (sz: 16384 bytes), total 32768 (slots), used 32753 (slots)
[ 2032.786928] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.787836] xhci_hcd 0000:01:00.0: swiotlb buffer is full (sz: 16384 bytes), total 32768 (slots), used 32753 (slots)
[ 2032.788015] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.788904] xhci_hcd 0000:01:00.0: swiotlb buffer is full (sz: 16384 bytes), total 32768 (slots), used 32753 (slots)
[ 2032.789310] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.789969] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.790368] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.791411] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.791835] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.792262] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.792651] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.793092] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.793496] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.793885] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.795102] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.795970] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.796439] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.796879] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.797335] usb 2-2: usbfs: usb_submit_urb returned -11
[ 2032.798003] usb 2-2: usbfs: usb_submit_urb returned -11

The key lines here are:

[ 2032.782023] swiotlb_tbl_map_single: 94 callbacks suppressed

[ 2032.782034] xhci_hcd 0000:01:00.0: swiotlb buffer is full (sz: 16384 bytes), total 32768 (slots), used 32753 (slots)

[ 2032.782064] usb 2-2: usbfs: usb_submit_urb returned -11

SWIOTLB appears to refer to a memory management system which is responsible for allocating memory for (amongst other things) USB devices. There is a nice writeup here.

The error messages above appear to indicate that the swiotlb buffer is being maxxed out. I grabbed a copy of the Raspberry Pi Kernel and discovered that error code -11 corresponds to -EAGAIN. The kernel documentation for the USB subsystem indicates that this can occur for usb_submit_urb if there are 'too many queued ISO transfers'.

This behaviour seems to match up with what is happening on the device. I poked around in /sys and found that an entry for swiotlb existed under /sys/kernel/debug/swiotlb. There are two nodes within this folder: io_tlb_nslabs and io_tlb_used.

The value in io_tlb_nslabs was 32768, which corresponds to the total 32768 (slots) message above. The value in io_tlb_used was 1. When I ran the demo program and watched the content of io_tlb_used, the value flapped between fully allocated (32768) and not allocated (1).

This appears to indicate that as the camera driver is attempting to perform USB transfers, it is maxxing out the available memory allocation for USB transfer operations in some way. I am not an expert in the kernel or USB allocation, but given the messages seen above, the research I have performed, and the behaviour that I am experiencing, this appears to be the case.

As I don't have access to the source code for the driver, I cannot alter the source code to fix this problem. Instead, I started to look to see if there was a way that I could increase the size of the swiotlb buffer. It turns out that it can be done.

Increasing the `swiotlb` Buffer

On a regular computer you can supply a kernel argument in the bootloader (e.g., in grub) to set the size of swiotlb. Because the Raspberry Pi doesn't boot the same way, there is a different mechanism, which is the file /boot/cmdline.txt. This file allows you to supply command line arguments which are passed to the kernel at boot time.

I added the following to the end of the existing command line arguments in my /boot/cmdline.txt file:

swiotlb=131072

This number was chosen because it is the value of 128 * 1024 = 131072. (The default size was 32 * 1024 = 32768).

I restarted the Raspberry Pi, ran the demo program, and I WAS ABLE TO EXPOSE AN IMAGE AT 8288 x 5644 @ 16 BITS on the 64 bit operating system.

I checked the content of /sys/kernel/debug/swiotlb/io_tlb_used and while it was still flapping up and down, it was only maxxing out at about 46000 (this is an eyeballed measurement, because it changes so rapidly), and it was not hitting the limit in /sys/kernel/debug/swiotlb/io_tlb_nslabs listed as 131072.

As far as I can tell, the above change has resolved the issue for me.

Checks For You To Perform

I would be interested to know why your Raspberry Pi 4 does not experience these problems while mine does.

I am particularly curious as to whether your Raspberry Pi 4 is attempting to use the swiotlb buffer when allocating USB Request Blocks or not. I feel like this may be key to the reason why we are experiencing differences in the way that the devices work. My guess is that (for whatever reason) my Raspberry Pi 4 either has a smaller allocation to swiotlb than yours, OR, your Raspberry Pi 4 is not using the swiotlb buffer when allocating USB Request Blocks.

Either way, I think that this mechanism may be core to the reason that many ASI_EXP_FAILED errors occur, not just for me, but probably for other customers too.

Could you please output the content of your /boot/cmdline.txt file?
```
cat /boot/cmdline.txt
```
Could you please run this command to list the available swiotlb buffer on your Raspberry Pi 4?
```
sudo cat /sys/kernel/debug/swiotlb/io_tlb_nslabs
```
While running one of the demo programs to capture an 8288 x 5644 @ 16 bit image/video, could you please run the following command to watch the amount of the swiotlb buffer that is being used, and report the typical high allocation values?
```
watch -n 0.2 cat /sys/kernel/debug/swiotlb/io_tlb_used
```

I don't know if this problem indicates that there's an issue with the way that the Raspberry Pi 4 is configured by default (not enough buffer allocation) or if there is an issue with the way that your libASICamera2.so driver is allocating/submitting USB Request Blocks, but I believe that I have identified the source of these failures.

Vector

Tagging @tech@zwo in case you don't get a notification about the response above.

Tech@ZWO

Vector

mobaxterm-19216831airpro-20210607-161153.txt

33kB

Hello,please refer to this

Vector

@tech@zwo#47013

Thanks for running those tests. That's very interesting -- it seems that with the identical boot, kernel, and OS configuration, your Raspberry Pi is not making use of swiotlb at all when allocating USB Resource Blocks, while my Raspberry Pi is.

I wonder if this is a physical hardware difference between the two models (your 4GB and my 8GB Pi) or of it is working that way because of something else entirely.

astromatto

This comes unfortunately after one year, but I baked my own aarch64 image based on arch linux and I have no issues with the asi driver, important to mention, I run the raspberry kernel and not the arch one

« Previous Page Next Page »