11 Replies Latest reply: Nov 13, 2010 4:44 PM by ptc-1371556 RSS

    Import binary file to Mathcad 14 - size limit?

    jhardy1 Copper
      I have created a Mathcad 14 worksheet which reads and processes a large binary data file - specifically, a tile of NASA SRTM terrain elevation data. The worksheet works fine using the generally available 3 arc-second resolution files, which are 2.8 MB each, with over 1,000,000 records in each file. (Each file is a vector containing one integer entry for the elevation data of each point on a 1,201 x 1,201 grid, for 1,442,401 integers in total.)

       

      I would like to use the same worksheet to process some 1 arc-second data files, which have approximately 9 times the number of records (3,601 x 3,601 = 12,9607,201 grid points), and 9 times the file size (approx. 25 MB each).

       

      When I try to point the "Data Import Wizard" at one of the larger files, it first seems to work fine, but then it loses the data from the imported data file, and the worksheet stops working, as it has no data vector in the main data array. (The array shows only a single entry with a zero value, instead of the expected vector of data entries - see attached screen grab.)

       

      I suspect this is an issue with the file size exceeding some limit of Mathcad for handling large binary files, and / or exceeding memory limits.

       

      Does anyone know what are the effective limits for handling large binary files in Mathcad 14.0 M020? I couldn't find anything in the documentation.

       

      Thanks for any help!
      (Note: I have also posted this question on the Eng-Tips forum http://www.eng-tips.com/viewthread.cfm?qid=285290&page=1 - please don't feel obliged to respond to both! )
        • Re: Import binary file to Mathcad 14 - size limit?
          A.Non PTC Community Champion

          With 13 million points it's likely you have simply run out of memory.

           

          An array requires a contiguous block of memory, so it's possible that if you reboot your computer, then start only Mathcad, you may be able to read in the file.

          • Re: Import binary file to Mathcad 14 - size limit?
            LucMeekes Silver

            You may have hit a (or the) maximum allowed array or matrix size.

            In Mathcad 11 the maximum array index is 16,000,000-1 exactly. That's a program(med) limitation. Mathcad 11 tells you when an array is too large. Don't know for Mcad 14.

             

            Regards,

            Luc

              • Re: Import binary file to Mathcad 14 - size limit?
                A.Non PTC Community Champion

                In Mathcad 11 the maximum array index is 16,000,000-1 exactly. That's a program(med) limitation.

                I don't think so. The MC 11 help says:

                 

                The effective array size limit depends on the memory available on your  system. Approximately 8 bytes of memory are needed per array element. For most  systems, the array limit is at least 1 million elements. In no system is it  higher than 8 million elements.

                 

                Except that is definitely not correct, because I just created an array bigger than 8 million elements. Maybe that piece of the help is a left over from an earlier version. I thought the size was limited only by memory (which is certainly the case in MC 14 and MC15).

                  • Re: Import binary file to Mathcad 14 - size limit?
                    MikeArmstrong Diamond

                    I just tried to create an array with an index of 16,000,000 and it failed, then again my work laptop is painfully slow.

                     

                    Mike

                    • Re: Import binary file to Mathcad 14 - size limit?
                      LucMeekes Silver

                      Richard,

                       

                      The 8 million limit is most probably a left-over from older versions.

                      I've determined the limit in Mathcad 11 to be just 1 less than 16,000,000 back in 2006. if you search the old forum for "array", "size" and my name you should find it.

                      It's a good thing if PTC has increased the limit to available memory size in latest versions.

                       

                      Luc

                      • Re: Import binary file to Mathcad 14 - size limit?
                        jhardy1 Copper

                        Thanks, Richard, that is my issue.

                         

                        Doing a bit of experimentation, I find that the biggest vector I can create from scratch has 8,388,608 elements (from 0 to 8,388,607); that is, 2^24 elements in total.

                         

                        Mathcad 14 on-line help doesn't mention any 8 million element limit:

                         

                        The array size limit depends on the memory available on your system. Approximately 8 bytes of memory are needed per array element. For most systems, the array limit is at least 1 million elements. Note that the maximum allocatable memory in a 32 bit system is 4 Gb, which has to hold all open applications, your operating system, and whatever memory is consumed by your calculations.

                         

                        but on my system (Mathcad 14 M020, Windows XP SP3, 2 GB RAM) it seems to be a hard upper limit of 8,388,608 elements.

                         

                        Also, it seems that Mathcad is not handling dynamic reallocation of memory very well - if I try to create two such vectors, it throws an error of "not enough memory" for the second vector; but when I delete the offending statement which attempted to create the second vector, the first vector is also corrupted due to "not enough memory".

                         

                        Oh well, at least I now know where my problem lies - I'll have to see if I can manually parse the terrain file into smaller chunks of less than 8,000,000 elements.

                         

                        Thanks for the help, guys!

                          • Re: Import binary file to Mathcad 14 - size limit?
                            A.Non PTC Community Champion

                            I don't know why you are seeing that hard limit. In MC15 I just created a vector with 40,000,000 elements. I can create a matrix with 40,000,000 elements too (altohugh not at the same time!).

                              • Re: Import binary file to Mathcad 14 - size limit?
                                A.Non PTC Community Champion

                                OK. All is not as simple as it seems.

                                 

                                If I shut down Mathcad and restart it I can create two arrays of 40,000,000 elements each. If I try to increase the size to 50,000,000 elements I get an out of memory error. If I close Mathcad and restart it I can now create two arrays of 50,000,000 elements each!

                                 

                                BUT If I change the size to 500 elements the amount of memory Mathcad is sucking up does not change! If I disable the calculations of the arrays the amount of memory Mathcad is sucking up does not change! If I close the worksheet completely the amount of memory Mathcad is sucking up does not change! If I reload the worksheet and try to create the arrays again I get an out of memory error, so even at this point the memory sucked up by the previus worksheet is not freed!

                                 

                                Now Mathcad versions post 11 are based on the .NET framework, and the .NET framework is designed to handle all memory allocation. There's a very nice description of how it works here:

                                 

                                http://www.c-sharpcorner.com/UploadFile/tkagarwal/MemoryManagementInNet11232005064832AM/MemoryManagementInNet.aspx

                                 

                                If Mathcad were relying on the .NET memory management the observed behavior should not be possible. It would not even be possible to have fragmented memory. Well, it would, but only until you need a bigger chunk of memory, at which point .NET would do a garbage collect and compaction to leave you with a big contiguous block of memory. It might be possible that decreasing the array size or disabling the calculation would not free the memory, because it would not force .NET to do a garbage collection. When a worksheet is closed though Mathcad should force a garbage collection. And if I load the worksheet a second time and try to create the arrays again then .NET should perform a garbage collection anyway, without any prompting from Mathcad, because it needs the memory.

                                 

                                So the only conclusion I can come to is that Mathcad has it's own memory management implemented on top of .NET. Further, that memory management is obviously very poor, and that is why everyone mysteriously runs out of RAM when they are not expecting it. I guess one lesson from the above is that shutting down Mathcad and restarting it may drastically improve things. That should not be necessary though!

                                  • Re: Import binary file to Mathcad 14 - size limit?
                                    A.Non PTC Community Champion

                                    I guess one other possibility is that .NET is being used to manage the memory, but when a worksheet is closed the worksheet object is not being set to "nothing", and so .NET thinks the worksheet is still being used. It's hard to believe that such poor programming could come out of a major software company though.

                                • Re: Import binary file to Mathcad 14 - size limit?
                                  jhardy1 Copper

                                  OK, here’s an interesting thing ...

                                   

                                  Please see attached Mathcad 14 file. (Also saved as version 11 for the benefit of any users of earlier versions.)

                                   

                                  I create a vector with a user-defined number of elements – 2^23 by default, but you can easily increase this to 2^24, 2^25, or any other length of your choice. I populate this vector so that the value of the i’th element is equal to i, from 0 (I am using the default of zero-based arrays in my system) up to whatever you set. I then display the vector, count the number of rows, and extract the value of the last entry (which should be equal to the number of elements minus 1).

                                   

                                  With 2^23 (8,388,608) elements, everything generally seems to work fine – I can scroll down to the bottom of the vector, and it reports the correct value (8,388,607) for the last element. I don’t seem to get many problems for vectors of this length (but once I have “thrown” a memory error using a larger value, it can get unstable if I then try to crank it back to 2^23 elements).

                                   

                                  At 2^24 (16,777,216) elements, it starts to become unpredictable. Sometimes I can scroll down to the see the last element is 16,777,215. On other occasions, I can only scroll down to element number 8,388,607, but the worksheet correctly reports the value of the last element as 16,777,215. Sometimes, it will only display a single element at Row 0, but it will still correctly report the value of the last element as 16,777,215. No memory error messages are generated, even when I can only scroll down part way, or when I can only see the first element.

                                   

                                  At 2^25 (33,554,432) elements, it seems to always generate a message that there is “Not enough memory for this operation” in the statement where I populate the vector.

                                   

                                  Once I “throw” a “not enough memory” message, I have to set the number of elements to something “comfortably” smaller to be able to get the worksheet to run again; e.g.  I may have to reset it back to say 2^22 elements to clear the memory error, then I can crank it back up to 2^23 or 2^24, but I generally can’t go straight to 2^23 after getting a memory error at 2^25. Also, the longer I play with it, ramping the vector size up and down, the more unstable it gets – it looks like there is a memory leak somewhere, and I have to close the worksheet and reopen it, or even close Mathcad and restart it to get it to run again after a while.

                                   

                                  My system is using Windows XP SP3 with 2 GB RAM, and running Mathcad 14 M020. Might be interesting to see what people get with other system configurations ...

                            • Re: Import binary file to Mathcad 14 - size limit?
                              ptc-1371556 Copper

                              Hi

                               

                              This issue of running out of memory has been raised a number of times over the years in the collaboratory.  Often blame is fobbed upon NET but this is not right simply because many other programmes run in Windows without memory issues like in Mathcad.  I think we need to get right to the issue here and not skirt around it all the time. The limitations should be documented in its manuals etc as users are getting a false impression of what can be done.

                               

                              I have  used Mathcad v11, v13, v14 and v15 and the same issue exists across all these versions.  Mathcad falls over when you use large arrays and when you start processing them - after all just creating a large array and doing nothing with it is a totally pointless exercise and proves nothing.  Mathcad must have an internal structural problem with regard to memory, particularly stressing its internal structure when processing large arrays.  It really needs to fix the issue once and for all.  As time goes on, we do more computation and process ever larger sources of data: this is a non-reversible trend.  Mathcad or any other software provider must keep up with these real life demands.  Mathcad has hit the limit so far as I am concerned and it seems for a number of other users as well.  For PTC / Mathcad trading as an provider of high level technical software, this is a poor situation and undermines their credibilty.  

                               

                              Do not get me wrong, I really like the Mathcad programme.

                               

                              However, Mathcad is getting very frustrating for me to use and it is even more frustrating to see Mathcad / PTC do so little software development over the last few years to push Mathcad further along the envelope and make it generally more powerful.  Forum users has raised many useful suggestions on advancing Mathcad, many of which should be fairly straightforward to implement.   I do not know what is in Prime (I am concerned by some of the comments in this forum that Prime has some limitations compared to even Mathcad 15), but I think PTC really need to pull their finger out and get some serious development work done.  I suggest that PTC should make some announcements to its users about what it is doing.    Has PTC properly resourced its programming team on software development? - I suspect not given the limited development output over the last few years.  Perhaps there are also issues on how PTC is running on a business management aspect?   It seems to me Mathcad may be good for a certain level of work, but it will get left behind for larger scale work.

                               

                              I have gone through my learning curve with Mathcad and written a number of programmes over the last 7 years or so for my work.  I am loath to dump all that and re-learn something else like Maple or Mathematica.  So I am hoping that Mathcad does progress and go to another level.  As soon as possible, please!!!

                               

                              Regards

                               

                               

                              David