SparkFun Forums 

Where electronics enthusiasts find answers.

Have questions about a SparkFun product or board? This is the place to be.
By Dave Wave
#123133
I am tinkering with a progect that is generating large files of data (on the order of 200K+ values). I want to convert the data from hex to dec, then preform some simple math on the value, then average five values for a result. I hope to end up with a data set that is a little more managable.

Short of trying to write my own program, is there an easy way to do this?

My version of excel is only good to 64K

Thanks,

-Dave
By Dave Wave
#123157
I think the problem might be trying to get any editing-type program to cut the data into chunks. Is there any way to process the data in a serial fashion to reduce the number of data points?

Thanks,


-Dave
By UhClem
#123160
Dave Wave wrote:Short of trying to write my own program, is there an easy way to do this?
Writing a program to do that is pretty simple. At least it is if you have done it before. But there are other options:

1) split - this is a standard unix program for splitting files up. Even if you aren't running Linux, you should still be able to locate a version suitable for your OS.

2) Dataplot is a heavy duty data processing tool which is way more than you need.

3) gnuplot This is a general purpose plotting tool which might do what you want. 200K values is not a problem but with large data sets it can get a bit slow.

4) Scilab A very general purpose tool.
By Dave Wave
#123183
fll-freak wrote:Perhaps.

It would be helpful if we had a clue what you are trying to do!

I am trying to process the data dump from a ulog.

Sample rate is 50HZ

Each channel is out put as a four digit hex number

There are three channels

I am trying to process this into a more usable form (convert to dec, average 5 readings together, then convert to G-load).

Any other ideas appreciated.

-Dave
By fll-freak
#123186
Do you have a Linux computer?
Do you have cigwin installed on a PC?
Do you know any languages (Perl, Python, C, C++, BASIC, FORTRAN, COBOL, SNOBOL, Lisp, Java, APL, MATLAB, IDL, ...) and have the tools to write a program in them?

If not, do you have any friends that can help you?
By Dave Wave
#123190
fll-freak wrote:Do you have a Linux computer?
Do you have cigwin installed on a PC?
Do you know any languages (Perl, Python, C, C++, BASIC, FORTRAN, COBOL, SNOBOL, Lisp, Java, APL, MATLAB, IDL, ...) and have the tools to write a program in them?

If not, do you have any friends that can help you?

I was really trying to avoid having to sit down and hack something out (it has been a looong time ) and free time is short, so I was looking for the easy way out.

-Dave
By Dave Wave
#123226
The answer would be yes to the first and no to the second.

However, I am not a fan of linux, so what would be the route with cigwin?

Thanks for the help,

-Dave
By fll-freak
#123231
Cigwin is a collection of Unix utilities (more, cat, ls, ...) for Windows. You would then be able to cobble together a series of filters to do your parsing, conversions, and basic math.

Even not being a fan of Linux, you still might want to go that route as it would give you access to things like Gnuplot.
By Dave Wave
#123238
I'll take a look at that and see if I can make it go. Failing that I will have to dust off my very rusty C++. Thanks for the help.


-Dave
By nickersonm
#123268
When I need to do stuff like this repetitively, and there's one or two operations I need that are not easily covered by standard utilities, I find it simplest to write a very small quick and dirty program. You're welcome to use the attached dataaverage.cpp if you wish. It just bins and averages a tab-separated dataset; you'd need to add a hex to dec conversion. Warning: it's completely unoptimized and probably has bad programming practices, but it works for what I needed.

I also second the recommendation of gnuplot for visualizing data. It offers simple and powerful scripting. There's no need to obtain the entire Cygwin package to get gnuplot, though; there's a Windows binary directly available.

- nickersonm
You do not have the required permissions to view the files attached to this post.
By Dave Wave
#123275
It looks like I'll take the C++ route.


Anybody have any recomendations for open-source/free compilers?

Thanks,

-Dave


PS thanks for the C++ example!