Custom datalogger and sample rates

[Goyds]

G'day All

I thought i might start by sharing some information before asking for help...

We've been developing a data logging and telemetry package for our 2013 electric vehicle based around and Mbed platform, CAN interfaces and a handful of remote capture boards giving around 80 data points. We have also used some Digi X-tend modules and have 2 way wireless telemetry which is nice for keeping an eye on battery temps etc. I can safely say that this isn't as big a project as we originally thought so if there are teams out there with a few electrical or robotics engineers who want to join this is a pretty good project. Total cost for us so far is around $800-$1000 AUD for the complete system (not including sensors other than GPS and accelerometer/gyro).

Anyway onto the question... I'm struggling a bit with the sheer quantity of data. We have several sample rates from 500Hz down to 10Hz with the data saved into a CSV file on a USB key however at 500Hz the data exceeds excels capacity after around 30 secs. What data analysis packages do you use/recommend? Has anyone tried anything in Matlab? Or am i missing the point and should just reduce the amount of data we gather?

The data we log is all the basics and a few others like fore/aft acceleration, gyro, GPS location and time, throttle/brake usage, steering angle, all wheel speeds, tire temps, Brake pad temps, motor parameters, battery parameters, errors and faults etc.

Cheers for the help.

Lloyd Morrissey
Chief Engineer
Team Swinburne 2012

[Jay Lawrence]

You could look into using SAS. It takes huge files and spits out summary info. Not sure on the specifics, just know that my girlfriend uses it at work to process enormous data sets.

Other than that I don't see why you would need 500 data points per second for anything. On the turbo cars we took up a fair bit of logging with turbo speed (can't remember the sample rate but we're talking over 200,000rpm...) but we were still able to make do with standard MoTeC M800 logging space. So I would first have a look at the data you need vs. what you're attempting to get.

Your list of logged items seems pretty reasonable, though I'm not sure how much is involved with your electric motor parameters.

[SNasello]

I agree with Jay, i don't really know what all you want to log at 500 Hz. The only sensor I would log that fast would be vertical accelerations, damper travel, or strain gauges.

That being said, Excel is probably not your best bet for data analysis. Just because you made your own custom data logger doesn't mean you also need to write an analysis software for it. I believe GEMS can import CSV files, and you can also check out the software from 2D-datarecording, which i used extensively. The 2D software is offered to all Formula Student teams, so license costs are not a problem.

[BeunMan]

I think you need to step away from excel files an start using either self-designed or use a format used by a data analyser (as mentioned by SNasello).

500hz is really high and almost never needed for any sensor. Check if your sensor can even provide these sampling rates.
Expect data rates in the order of 2Kbyte/sensor/second without any overhead and 4byte/sensor) which doesn't sound like a lot but for lossy wireless systems (like wifi) this can start to be a problem is the connection is not constant. I think you would get around ~512Kbyte/s of data at those rates and that is quite a lot to keep processing without errors and congestion.


Tristan
Chief Electronics 2010 DUT Racing

[Goyds]

Thank you for all the replies.

We are only measuring accelerations, shock displacement and tire temp at 500Hz, the rest are generally around 20 or 100hz with. We plan to analyze the tire temp data to look for hot spots on the rotating surface, otherwise these would probably only be 10-20Hz as well. We are not entirely sure what we will see or if it will be useful but that is kind of the point...

It sounds like i need to look into saving the data into a format that can be interpreted by an analysis package such as the motec, bosch or 2D one. I know that the older motec software could accept .csv files. I'll look into GEMS although it looks like the 2D one might be what i want.

[Goyds]

Actually I've just come across a few books on race car data analysis. Does anyone have any they like/recommend? This might be a better approach o work out what we really want/need.

Lloyd Morrissey
Chief Engineer
Team Swinburne 2012

[murpia]

The best value motorsport data analysis package I'm aware of is GEMS Data Analysis Pro:

http://www.gems.co.uk/?content...d=gems-data-analysis (http://www.gems.co.uk/?content=pages&id=gems-data-analysis)

It features ascii import, even if the feature list does not say so. However, the ascii import operates at a fixed sample rate, so if you wanted to look at 500Hz data you'd need to create 500Hz data for ALL channels (i.e. massive files).

The overlay feature works well though, so you could combine more than one dataset of different sample rates if really needed.

Like all this stuff, you have a cost / convenience / time tradeoff to make.

Regards, Ian

[BillCobb]

Yes, SAS (Statistical Analysis System) can handle the volume of data you have on hand. Once you learn SAS, you will practically know Matlab. SAS can even perform frequency response analysis using PROC SPECTRA. But, A student version of Matlab can swallow all the data, too. The reasons I suggest Matlab are:

1) It's a motorsports industry standard for exploratory measurements and modeling. So, there are very large numbers of 'shared' scripts which you can borrow and save huge amounts of time, all while enhancing your Matlab skills. (You learn the process by example).

2) Combined with Simulink, you can 'drive' your simulations with the data or data models you've prepared.

3) You can prepare .mat files of your data all ready to go into Matlab programs.

4) Matlab, like knowing a few other CAE programs, is not just an enhanceent of your education. If you plan to enter industry of any sort, it will be a requirement. Even if you wind up in a Bridge Road or Murrays plant or Heaven forbid, a Moon Dog factory, I can bet Matlab will get you to the front of the employment line, just like wearing a 'US Border Patrol' hat does in this country.

[jlangholzj]

Originally posted by BillCobb:
Yes, SAS (Statistical Analysis System) can handle the volume of data you have on hand. Once you learn SAS, you will practically know Matlab. SAS can even perform frequency response analysis using PROC SPECTRA. But, A student version of Matlab can swallow all the data, too. The reasons I suggest Matlab are:

1) It's a motorsports industry standard for exploratory measurements and modeling. So, there are very large numbers of 'shared' scripts which you can borrow and save huge amounts of time, all while enhancing your Matlab skills. (You learn the process by example).

2) Combined with Simulink, you can 'drive' your simulations with the data or data models you've prepared.

3) You can prepare .mat files of your data all ready to go into Matlab programs.

4) Matlab, like knowing a few other CAE programs, is not just an enhanceent of your education. If you plan to enter industry of any sort, it will be a requirement. Even if you wind up in a Bridge Road or Murrays plant or Heaven forbid, a Moon Dog factory, I can bet Matlab will get you to the front of the employment line, just like wearing a 'US Border Patrol' hat does in this country.

This.

I've also found that I'm learning a lot more using matlab for some of the other team members. Running tire data from calspan for my ss/frame guy has opened up a huge chunk of knowledge to me. I feel I understand things quite a bit better once I chug through matlab.

Also just the other day I took data from comp last year, exported my TPS,RPM and IJPW, open it up in matlab and with about 20 lines of code (30-40 if you make it look pretty) i've got a surface plot of a new alpha-n fuel map. We were previously speed-density but we've been working on making the switch. Thing fired right up and ran pretty darn well on it too!

Matlab also has some nice gui stuff that you can play around in for data gathering and display. It would be very possible for you to real-time graph either the simulated data that Mr. Cobb is talking about OR use it as a replay for the data you've gathered.

This next year I'm going to do a matlab workshop for the guys and get them going on it....its a very powerful tool once you learn it. Truthfully I don't even completely grasp everything in it yet.

[rollcentre]

National Instruments has a software package called DIAdem that is meant to work with very large sets of data in a 'channel' style environment, instead of the 'cell' style environment of Excel. I have had only a little exposure to it but liked what I saw. I think they have a short free trial period of the full-featured software.

[Goyds]

G'day All

I've settled on GEMS data analysis or actually Dlog99 for now. It seems easy to create files that it can open and it looks like it will meet our needs for now. We might look into GEMS pro at some later stage although if we need to do any heavy data analysis we will probably use matlab.

Thank you all for the advice, it was very helpful.

[BeunMan]

Another note on data rates: If you don't want to send the massive data stream for all sensors at 500hz you can always decide to convert the data by a program on the receiving side and thus limit your data rate. It's a form of time based compression by encoding, send it and then decoding the stream to fit the GEMS ascii text format.


Tristan
Chief Electronics 2010 DUT Racing

[Andrew Richards]

You might try concerto by avl. Also minitab would probably work ok and is really simple to use. Another that is supposed to work with large amounts of data but I have no experience with is origin lab which is a piggy back on excel. Better question is what do you want to do with your data?

[blacksaleen95]

AEM AQ-1 is the best bang for the buck datalogger. AEMData is on par with MoTec I2 Pro but for 1/10 the price. If you like the GEMS stuff check out the AEM line, it's very very similar to the GEMS.

[murpia]

Originally posted by blacksaleen95:
AEM AQ-1 is the best bang for the buck datalogger. AEMData is on par with MoTec I2 Pro but for 1/10 the price. If you like the GEMS stuff check out the AEM line, it's very very similar to the GEMS.
Yes, I wasn't aware of AEM until now (thanks!).

Their analysis software is clearly developed / licensed from GEMS and is the same except for the branding. I downloaded it and compared it to GEMS:

It has the powerful math channel capabilities of GEMS GDA Pro, but not the CSV file import feature. So it's 'locked' to the AEM hardware in the same way as MoTeC i2 Pro is, but the AEM hardware looks significantly cheaper.

If you want to get a feel for it, do this:

1) download & install GEMS GDA (standard) from gems.co.uk
2) download and install AEMdata from aemelectronics.com
3) locate the demo files from GEMS GDA, called 'GDA Demo.gdap' and 'GDADemo.stf'. They will be in the My Documents of the Windows account used to install the software (i.e. the Administrator not necessarily the User).
4) Copy 'GDA Demo.gdap' and rename it 'GDA Demo.daqproj'.
5) Open 'GDA Demo.daqproj' in AEMdata and when prompted choose the 'GDADemo.stf' data file.
6) You can then try out the software, just the same as trying out GEMS GDA, but with the added feature of the maths channels (menu 'Data->Edit Maths...' and read the comprehensive help.

Regards, Ian

[HenningO]

Originally posted by murpia:
Yes, I wasn't aware of AEM until now (thanks!).


Me neither, thanks! That price point is hard to beat! Almost want equip my daily driver with one just to keep tabs on how many %WOT I have during my rush hour commute (0% FYI).

I also tested AEMdata and as Ian points out, it is the same as GEMS GDA. There is also a demo project that comes with the AEMdata install.

A couple of questions (for Ian or someone else):

1. The time/distance graphs in AEMdata/GDA will show any channel as constant between two sample points, so when you zoom in you'll have steps in line. Anyone know if this can be configured to behave more like Motec/PI where you'll have a straight line between the two data points?

2. Does anyone have experience with the AEMnet that the AQ-1. It seems like it's based on CAN? It would be nice to be able to expand the 8 A/D ports with some more ports over a A/D-CAN interface?

I think this should enable more teams to get into data analysis and actually get something useful out of it - thanks to a powerful and easy to use analysis software.

[murpia]

Originally posted by HenningO:
I also tested AEMdata and as Ian points out, it is the same as GEMS GDA. There is also a demo project that comes with the AEMdata install.

A couple of questions (for Ian or someone else):

1. The time/distance graphs in AEMdata/GDA will show any channel as constant between two sample points, so when you zoom in you'll have steps in line. Anyone know if this can be configured to behave more like Motec/PI where you'll have a straight line between the two data points?

If you bring up the 'Properties' for a channel, either by right-clicking in a trace window or double-clicking in the channel list, there is a drop-down to select the interpolation mode.

In fact, the default (hold) which gives you 'steps' is actually the most correct way to display the data. Just because you _can_ interpolate between points doesn't mean you should, without proper justification. That means gaining considerable (often experimental, using lab kit) knowledge about the sampling performance and anti-alias filtering of your data acquisition.


I think this should enable more teams to get into data analysis and actually get something useful out of it - thanks to a powerful and easy to use analysis software.

I hope so too,
Ian

[HenningO]

Originally posted by murpia:
If you bring up the 'Properties' for a channel, either by right-clicking in a trace window or double-clicking in the channel list, there is a drop-down to select the interpolation mode.

In fact, the default (hold) which gives you 'steps' is actually the most correct way to display the data. Just because you _can_ interpolate between points doesn't mean you should, without proper justification. That means gaining considerable (often experimental, using lab kit) knowledge about the sampling performance and anti-alias filtering of your data acquisition.


Thanks! Agreed, although I find it easier to visually interpret the data/rate of change with it in a non-step mode.