FAMOS (IMC) raw data in MathCAD?

[ClaudioPedrazzi]

Hi everyone,

is it possible to read (import) FAMOS raw data into Mathcad?  eventually with some kind of script?

Some measurement equipment produce this kind of raw format.  The format description can be found in the manufacturer web page:

http://www.imc-berlin.de

An example of what I mean can be found here, albeit with MATLAB:

File Exchange - MATLAB Central

I would have a description of the format, but it is in German. And it is pretty complex

Thanks for any hint!

Best regards

Claudio Pedrazzi

[StuartBruff]

Claudio Pedrazzi wrote:

Hi everyone,

is it possible to read (import) FAMOS raw data into Mathcad?  eventually with some kind of script?

Some measurement equipment produce this kind of raw format.  The format description can be found in the manufacturer web page:

http://www.imc-berlin.de

An example of what I mean can be found here, albeit with MATLAB:

File Exchange - MATLAB Central

I would have a description of the format, but it is in German. And it is pretty complex

It should be possible (shame Mathcad doesn't have regular expressions to make life easier, though).

Getting the file format requires creating an account (to download the Curve Window and Report Generator documentation containing the format), which I'm too lazy to do, so even the German version would probably be helpful.

Stuart

[Werner_E]

The first link (FAMOS) you provide ends in an 404 error.

The second link to imc did not lead directly to a description of the file format. While its possible that the description is hidden somewhere in the depths of this site, the willingness to search for is limited. Furthemore the site requires people to create a (free) account and register, which at least I am not willing to do.

You should provide a direct link and I guess it could be beneficial if you also provide a few sets of files you actually would like to be imported in Mathcad.

Then you should state, which version of Mathcad you are using. There are at least two very different flavors of Mathcad, regular Mathcad 15 (or below) and the newer and severely limited Mathcad Prime.

If the format you are interested in is a simple text format, it might be able to read it in directly in Mathcad and convert it to a number of matrices or whatever the end format in Mathcad should be. But if the format is so complex as you wrote, it probably could be hard to find anybody here who will do this for you.

The Matlab script you provided a link to script suggests that it should be possible to achieve in Mathcad, too, but it sure would be some work to convert that script to Mathcad.

An alternative could be to write a standalone converter in any programming language you are well versed in which converts the data in a format, Mathcad can read in directly.

WE

[ClaudioPedrazzi]

Sorry for the not working link to FAMOS Software...

http://www.imc-berlin.com/products/measurement-software/imc-famos/

I am using Mathcad 15.0

Attached are

a) the descriptions I have (please note, the file is NOT produced by FAMOS, it has just FAMOS-Format but is produced by measurement equipment of a supplier)

b) an example of a small file that I would like to read

And I am sorry for asking not so clearly.  My willingness to write software is also limited, unfortunately .  What I am actually asking is if there is a "black-box"-like solution, something almost ready to use, that I can try to use.

thanks again for any hint

Regards

Claudio

[MartinHanak]

Hi,

Sample for Forum.RAW is binary file. This means you cannot open it in Mathcad.

Try to get the file in text format.

MH

[ClaudioPedrazzi]

Hi Martin

thanks, I am aware of that.  That is the first thing I tried, to open with Notepad++.  So probably my question does not make sense.

One never knows, there could have been some tricks, or someone could have already developed a small program.

At least in german-speaking Europe the FAMOS software seems to be well known.

Best regards

CP

[Werner_E]

> Sample for Forum.RAW is binary file.

Sure it is. If you read the description of the file format, you would see that asking for the same data in text format is rather hopeless.

The format is quite flexible and accommodates to various number formats, etc.

> This means you cannot open it in Mathcad.

Sure we can read in the file using READBIN (type "byte") and you get a nx1 vector with the ascii values from 0 to.255 representing the file content. The demo RAW data could be read in quite quick an easy, but I could imagine problems with real bíg data files in GByte size as Mathcad always tries to read the whole file at once.

After reading in  comes the hard part. You have to write a parser in Mathcad to separate the different keys and the data held within. While sure possible it would mean quite some work to do.

Unfortunately I don't know of any ready to use solution.

Werner

[StuartBruff]

It is possible in principle to read and interpret the RAW (or dat) file, Claudio.

However, looking at the data, the last record on the list has the form

|CS,1,11895686,1,§¹K;6AG;q|D; o-;Àa:;™ü&;»‹8; ...  <etc, for the next 11 MBi>;

If I've read the notes for the CS key correctly, then it has format

CS,1,Key Length (=11895686),Index(=1),Raw Data(=§¹K;6AG;q|D; o-;Àa:;™ü&;»‹8; ...);  [Note: the | in between the q and the D is not a key marker, it's just one of the 4 bytes in a binary 32-bit number]

So, there are about 3 million numbers, which means that Mathcad 15 may run out of memory.  I'l have a look later .. got work to do!

Stuart

|CF,2,1,1; (Date Format)

|CK,1,3,1,1; (File is completely written)

|NO,1,7,1,0,,0,; (Data Origin)

|CG,1,5,1,1,1; (Data Field definition)

|CD,1,13,1,1,1,s,0,0,0; (Older version ...)

|NT,1,19,28, 7,2015,13,54,39; (Trigger time (28 Jul 2015 at 13:54:39)

|CC,1,3,1,1; (Component start, real, analog)

|CP,1,16,1,4,7,32,0,0,1,0; (Buffer 1, 4 bytes, float, 32 significant bits, standard mask, no offset, sequential data, not interspersed)

|Cb,1,36,1,0,1,1,0,11895684,0,11895684,1,0,0,; (Buffer Description ...  Buffer Length = 11895684)

|CR,1,13,0,0,0,1,3,m/s; (Value range, .. calibrated, range 3 m/s)

|CN,1,28,0,0,0,16,RMSV_20PRW_B801z,0,; (channel comment .. title = RMSV_20PRW_B801z)

[ClaudioPedrazzi]

Hi Stuart and Werner

that's cool!

I never used READBIN, I didn't even know its existence.  Thanks a lot!

So for the data quantity, I have chosen the smallest file that I have !!!

It should actually contain 2,973,921 time points!

So Stuart is right about the round three millions.  I have other files containing 18 millions points.

For information, the values are rms of vibration speed in one direction, registered by an instrument during a long measurement campaign.

To develop a parser appears to me pretty difficult, but I have to say, in principle what I would like to have is very simple:

-two columns, one with a time stamp, the other with the measured value.

- so that I can try to make plots, interpolations, correlations, and so on.

Thanks a lot again and have a nice day

Claudio

[StuartBruff · Accepted solution]

Claudio Pedrazzi wrote:

Hi Stuart and Werner

that's cool!

I never used READBIN, I didn't even know its existence.  Thanks a lot!

So for the data quantity, I have chosen the smallest file that I have !!!

It should actually contain 2,973,921 time points!

So Stuart is right about the round three millions.  I have other files containing 18 millions points.

For information, the values are rms of vibration speed in one direction, registered by an instrument during a long measurement campaign.

To develop a parser appears to me pretty difficult, but I have to say, in principle what I would like to have is very simple:

-two columns, one with a time stamp, the other with the measured value.

- so that I can try to make plots, interpolations, correlations, and so on.

Thanks a lot again and have a nice day

Claudio

If this file is representative of the data you're looking at, then writing a parser should be fairly straightforward - it's what I had to do to convert the initial parts to the nested array shown in my previous message.  

READBIN should allow you to read in segments of data rather than reading in the whole thing (READBIN("file", "type", [[endian], [cols], [skip], [maxrows]]).  Use [skip] to get to where the segment starts and [maxrows] to determine the size of the segment (eg, READBIN("file", "byte", 0, 1, 2000, 1000) to get a block of 1000 bytes starting at the 2000th byte) - I've used this before to read chunks of large files and, if you find the starting point for the CS raw data, you might even be able to set type as "float" to simplify the process).

Stuart

[ClaudioPedrazzi]

Thank you Stuart and Werner

you have helped me a lot.  I will try my luck with READBIN and with the trick of skipping and reading "forward", if I can find the time to invest for this project.

But at least I have now two answers, that I was looking for.

a) there is no ready-made solution for reading FAMOS formatted data (as for example for WAV or JPG, BMP files).  Also not available a third-party solution.

b) with an acceptable amount of programming it is possible to read the file and decode it.

For the moment, I just say thank you

Claudio

[ClaudioPedrazzi]

Hi everyone,

since I like to give something back to the forum, because I receive so much help from it, I am posting here some additional information that I found useful.  I have been made aware only now, that there is a free "imc-FAMOS-reader" available from imc.  And with that software one can read one ore more measured results in the RAW-format, plot them, explore them and see them in tabular form.  At the moment I am not aware of any possibility to export the data, once read. Unfortunately.

The reader is described here:

http://www.imc-berlin.de/produkte/messtechnik-software/imc-famos/reader/

Best regards

Claudio