bug report: possible memory leak in ASICamera2 SDK on linux?

elmerfudd

Hi,

I've been spending a lot of time trying to figure out why my camera program will run nicely for awhile and then hang after a few hours of execution. I think I might have boiled it down to a possible memory leak in the SDK.

It appears that about 2M of memory gets consumed each time an image is captured using the ASIGetDataAfterExp() function.

Attached below is the smallest program I could come up with that illustrates the problem.

Sam, could you take a look at this and either tell me if I'm using your library incorrectly or see if there is indeed a bug in the SDK?

Thanks and regards,

--Derek

// This program illustrates what seems to be a memory leak in the ZWO SDK running on linux with an ASI120MM camera.
// Compile and run it like this:
//
//     g++ -Wall -Werror -g -c -o bug.o bug.cc
//     g++ -o bug bug.o libASICamera2.a -lusb-1.0
//     ./bug
//
// 09 Sep 2015 Derek Lieber
//
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include "./ASICamera2.h"

int  
main()
   {
   typedef unsigned char  BYTE;
   typedef unsigned short WORD;

   int numDevices = ASIGetNumOfConnectedCameras();

   if (numDevices <= 0)
      {
      printf("no camera\n");
      exit(1);
      }

   if (numDevices != 1)
      {
      printf("unexpected cameras\n");
      exit(1);
      }

   int cameraIndex = 0;
   ASI_CAMERA_INFO info;
   ASIGetCameraProperty(&info, cameraIndex);

   int cameraId = info.CameraID;
   if (ASIOpenCamera(cameraId) != ASI_SUCCESS)
      {
      printf("can't open camera\n");
      exit(1);
      }

   int width = 1280; int height = 960; int binning = 1;
   ASI_IMG_TYPE type = ASI_IMG_RAW16;
   int num_pixels = width * height;
   int bytes_per_pixel = 2;
   
   if (ASISetROIFormat(cameraId, width, height, binning, type) != ASI_SUCCESS)
      {
      printf("can't set image format\n");
      exit(1);
      }

   WORD *image = new WORD;

   for (int cnt = 0;;)
      {
      ASIStartExposure(cameraId, ASI_FALSE);
      
      ASI_EXPOSURE_STATUS status = ASI_EXP_WORKING;
      while (status == ASI_EXP_WORKING)
         {
         usleep(100 * 1000);
         ASIGetExpStatus(cameraId, &status);
         }
         
      if (status != ASI_EXP_SUCCESS)
         {
         printf("exposure failed\n");
         exit(1);
         }
      
      if (ASIGetDataAfterExp(cameraId, (BYTE *)image, num_pixels * bytes_per_pixel) != ASI_SUCCESS)
         {
         printf("no data\n");
         exit(1);
         }

      printf("===%d===\n", ++cnt), fflush(stdout);
      
      // Here we display our program's memory size as it runs.
      // We do it by parsing the output of the ps command, which looks similar to this:
      //
      // F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
      // 0 S  1000 14015 14014  9  80   0 - 25916 hrtime pts/1    00 bug
      //                                    ^^^^^we print this field
      
      char cmd;
      snprintf(cmd, sizeof(cmd), "ps -l -p %d | tail -1 | awk '{ print $10 }'", getpid());
      system(cmd);
      
      // Watch this output as the program runs.
      // It looks something like this:
      //
      //  g++  -Wall -Werror -Wno-deprecated-declarations -g   -c -o bug.o bug.cc
      //  g++  -o bug bug.o  libASICamera2.a -lusb-1.0
      //  ./bug
      //  ===1===
      //  11039
      //  ===2===
      //  11552
      //  ===3===
      //  12065
      //  ===4===
      //  12578
      //  ===5===
      //  13091
      //  ===6===
      //  13604
      //  ===7===
      //  14117
      //  ===8===
      //  14630
      //  ===9===
      //  15143
      //  ===10===
      //  15656
      //  ===11===
      //  16169
      //  ===12===
      //  16682
      //  ===13===
      //  17195
      //  ===14===
      //  17708
      //  ===15===
      //  18221
      //  ===16===
      //  18734
      //  ===17===
      //  19247
      //  ===18===
      //  19760
      //  ===19===
      //  20273
      //  ===20===
      //  20786
      //  ===21===
      //  21299
      //  ===22===
      //  21812
      //
      // Notice that the program's size increases by 513 pages after each snapshot.
      // 1 page = 4KB = 4096 bytes, so 513 x 4096 = 2,101,248 bytes
      // Eventually the program runs out of memory and hangs.
      }

   return 0;
   }

elmerfudd

I did some googling and found the exact same problem was reported here:

http://indilib.org/forum/ccds-dslrs/761-virtual-address-space-asicam.html

The sequence of posts indicates that the problem was "fixed", however the final post asking "was this resolved" was never answered. I wonder if it was really fixed or maybe has resurfaced again?

Out of curiosity,I instrumented the library code to monitor calls to malloc() and free() and saw lots of calls between each ASIGetDataAfterExp().

I don't understand why there should be any dynamic memory management at all?

It seems like the SDK should be able to allocate all its buffers once the image dimensions have been selected and then run forever without any mallocs/frees. This would certainly improve the performance. And maybe it would fix whatever subtle memory management issue that's responsible for this bug?

elmerfudd

I'm still looking at this, trying to figure out why memory keeps growing.

Not having access to SDK source code makes analysing the problem difficult.

I tried running the program under gdb to see what's going on and saw some peculiar behavior: a thread is being created and destroyed for each image capture. See the snippet below, which is the output from gdb.

I'm not sure if this is part of the problem or not, but it's an incredibly inefficient algorithm, in terms of performance.

I also makes me wonder if this is tied in with the memory growth symptoms somehow.

--Derek


.
.
.


==  0 ==


== 1 ==


== 2 ==


== 3 ==


== 4 ==


== 5 ==


== 6 ==


.
.
.

elmerfudd

Sorry to keep posting like this, but here's one more tidbit pointing to a possible association between the threads that are being created and the memory consumption.

From "man pthread_create":


On Linux/x86-32, the default stack size for a new thread is 2 megabytes.

What's interesting about this is that 2 megs matches the increment in process size I'm seeing after each capture.

I wonder if it's possible that the thread stacks are not being reclaimed?

One possibility is that pthread_join() is not being called to reclaim the thread after it has exited.

From "man pthread_join":


Failure to join with a thread that is joinable (i.e., one that is not detached), produces a "zombie thread".  Avoid doing  this,  since  each  zombie thread  consumes some system resources, and when enough zombie threads have accumulated, it will no longer be possible to create new threads (or processes).

This post references a similar problem:

https://stackoverflow.com/questions/8170734/how-to-reclaim-memory-after-a-pthread-exit

And claims the solution is to use the following to set the thread attributes prior to creating it:

pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);

elmerfudd

I believe I've solved the problem.

Using dlsym() I inserted hooks to intercept calls to pthread_create() and pthread_join().

I found that pthread_join() is never being called, so the thread's memory is never reclaimed after it exits.

I then tried patching the call to pthread_create() to supply a PTHREAD_CREATE_DETACHED attribute.

This indeed fixed the memory leak.

Here's the program output with the patch disabled, showing process size after each image capture.

Note the increasing sizes.


installing hooks
create
===1===
11044
create
===2===
11557
create
===3===
12070
create
===4===
12583
create
===5===
13096

Here's the output with the patch enabled.

Note that the program size stays constant.

nstalling hooks
create
===1===
11045
create
===2===
11045
create
===3===
11045
create
===4===
11045
create
===5===
11045

So the bug is in the SDK library. It's probably best not to debate the wisdom of creating a new thread for every image capture, but if you're going to do that, you must pthread_join() them to reclaim their memory.

Please let me know what you think. Thanks.

--Derek

Below, for reference, is the temporary patch code that I used for testing.

// Program to illustrate thread-related memory leak in the ZWO SDK running on linux with an ASI120MM camera.
// Compile with:
// g++  -Wall -Werror -g   -c -o bugfix.o bugfix.cc
// g++  -o bugfix bugfix.o  libASICamera2.a -lusb-1.0 -ldl

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include "./ASICamera2.h"

#include <pthread.h>
#include <dlfcn.h>

// Intercept calls to pthread_create() and apply PTHREAD_CREATE_DETACHED attribute.
//
typedef int (*PTHREAD_CREATE       )(pthread_t *thread, const pthread_attr_t *attr, void *(*start_routine) (void *), void *arg);
static  int (*actual_pthread_create)(pthread_t *thread, const pthread_attr_t *attr, void *(*start_routine) (void *), void *arg);
        int   pthread_create        (pthread_t *thread, const pthread_attr_t *attr, void *(*start_routine) (void *), void *arg)
           {
           printf("create\n");

#if 1 // 1=apply bugfix, 0=don't

           static pthread_attr_t new_attr;
           pthread_attr_setdetachstate(&new_attr, PTHREAD_CREATE_DETACHED);
           attr = &new_attr;

#endif
           
           return actual_pthread_create(thread, attr, start_routine, arg);
           }

// Intercept calls to pthread_join() and report them.
//
typedef int (*PTHREAD_JOIN       )(pthread_t thread, void **retval);
static  int (*actual_pthread_join)(pthread_t thread, void **retval);
        int   pthread_join        (pthread_t thread, void **retval)
           {
           printf("join\n");
           int rc = actual_pthread_join(thread, retval);
           return rc;
           }

// Install function hooks.
//
void
bugfix_init()
   {
   printf("installing hooks\n");

   if (!(actual_pthread_create = (PTHREAD_CREATE)dlsym(RTLD_NEXT, "pthread_create")))
      {
      printf("bad dlsym");
      exit(1);
      }

   if (!(actual_pthread_join = (PTHREAD_JOIN)dlsym(RTLD_NEXT, "pthread_join")))
      {
      printf("bad dlsym");
      exit(1);
      }
   }

int  
main()
   {
   bugfix_init();

   typedef unsigned char  BYTE;
   typedef unsigned short WORD;

   int numDevices = ASIGetNumOfConnectedCameras();

   if (numDevices <= 0)
      {
      printf("no camera\n");
      exit(1);
      }

   if (numDevices != 1)
      {
      printf("unexpected cameras\n");
      exit(1);
      }

   int cameraIndex = 0;
   ASI_CAMERA_INFO info;
   ASIGetCameraProperty(&info, cameraIndex);

   int cameraId = info.CameraID;
   if (ASIOpenCamera(cameraId) != ASI_SUCCESS)
      {
      printf("can't open camera\n");
      exit(1);
      }

   int width = 1280; int height = 960; int binning = 1;
   ASI_IMG_TYPE type = ASI_IMG_RAW16;
   int num_pixels = width * height;
   int bytes_per_pixel = 2;
   
   if (ASISetROIFormat(cameraId, width, height, binning, type) != ASI_SUCCESS)
      {
      printf("can't set image format\n");
      exit(1);
      }

   WORD *image = new WORD;

   for (int cnt = 0;;)
      {
      ASIStartExposure(cameraId, ASI_FALSE);
      
      ASI_EXPOSURE_STATUS status = ASI_EXP_WORKING;
      while (status == ASI_EXP_WORKING)
         {
         usleep(100 * 1000);
         ASIGetExpStatus(cameraId, &status);
         }
         
      if (status != ASI_EXP_SUCCESS)
         {
         printf("exposure failed\n");
         exit(1);
         }
      
      if (ASIGetDataAfterExp(cameraId, (BYTE *)image, num_pixels * bytes_per_pixel) != ASI_SUCCESS)
         {
         printf("no data\n");
         exit(1);
         }

      printf("===%d===\n", ++cnt), fflush(stdout);
      
      // Here we display our program's memory size as it runs.
      // We do it by parsing the output of the ps command, which looks similar to this:
      //
      // F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
      // 0 S  1000 14015 14014  9  80   0 - 25916 hrtime pts/1    00 bug
      //                                    ^^^^^we print this field
      
      char cmd;
      snprintf(cmd, sizeof(cmd), "ps -l -p %d | tail -1 | awk '{ print $10 }'", getpid());
      system(cmd);
      }

   return 0;
   }

elmerfudd

Resolved. Sam and Yang sent me a new library that fixed the problem. Thanks guys!

silidan

elmerfudd wrote:
Resolved. Sam and Yang sent me a new library that fixed the problem. Thanks guys!

Did they also update the official SDK to download? (at least the Version name didnt change)

regards

Silidan

elmerfudd

silidan wrote:
elmerfudd wrote:Resolved. Sam and Yang sent me a new library that fixed the problem. Thanks guys!

Did they also update the official SDK to download? (at least the Version name didnt change)

regards

Silidan

I just checked, and the SDK on the download page has not been updated. --Derek

Sam@ZWO

elmerfudd wrote:
silidan wrote:
elmerfudd wrote:Resolved. Sam and Yang sent me a new library that fixed the problem. Thanks guys!

Did they also update the official SDK to download? (at least the Version name didnt change)

regards

Silidan

I just checked, and the SDK on the download page has not been updated. --Derek

we will upload here in software page for your guys to test soon

manjiw

Hello @Sam@ZWO,
Has this issue been fixed? I'm using v1.17 of the SDK and I am still seeing a memory leak issue after calling "ASIStartVideoCapture".

Software@ZWO

We have fixed this problem. The latest SDK version is v1.26. Please use the latest version. If you find this problem in use, please send your test code to us, and we will try to find the reason.

manjiw

Thanks. I've done a quick check and it seems to be fixed, will do some further testing to see if any issues arise.

Software@ZWO

manjiw 😃