SparkFun Forums

[TriBob]

I've been having some problems with the pre-allocation feature: The pre-allocation works, and pre-allocating a 1GB log file is very fast (well under a second), but the limited FAT library has problems "knowing" where the write point is when overwriting an existing file: The data goes to the right place at the beginning of the file, but the file can't be truncated to shorten it to the actual space used prior to closing. The library code to reset the file length seems to be broken.

Rather than fix the FAT code, I may try another FAT library (perhaps one that also supports FAT32).

But before doing that, I'll first take a look at the complete rewrite of the V1 code done by UhClem, which is located here: http://home.earthlink.net/~schultdw/logOmatic/ Porting it to the V2 should be straightforward.

[TriBob]

I'm using the Free (and current) Code Sourcery ARM EABI GCC toolchain, available for Linux and Windows (and Mac with minor effort) here:
https://sourcery.mentor.com/sgpp/lite/a ... iption3053
(Yes, if I were really hard-core I'd build the ARM GCC toolchain from source. But I'd rather build loggers, not compilers.)

After installing it, I modified this section of the V2 Makefile to use the right toolchain command names:

Code: Select all

## Multi-platform Code Sourcery toolchain (ARM EABI)
CC = arm-none-eabi-gcc
CPP = arm-none-eabi-g++
OBJCOPY = arm-none-eabi-objcopy
OBJDUMP = arm-none-eabi-objdump
SIZE = arm-none-eabi-size
NM = arm-none-eabi-nm

I just had one of those slap-the-forehead moments: There is no need to modify the SFE Makefile: It makes more sense to just create soft-links to the CodeSourcery binaries that match the generic "arm-elf-*" naming convention, since this will also work with other projects that use the generic ARM GCC compilers.

Here's what I did on my system: Change to the directory containing the Code Sourcery binaries (I used /opt), then run a one-line Bash command to to create the soft-links all at once:

Code: Select all

$ pushd /opt/CodeSourcery/Sourcery_CodeBench_Lite_for_ARM_EABI/bin/
$ for i in arm-none-eabi-*; do ln -s $i ${i/none-eabi/elf}; done

Be sure to add the above directory to your PATH.

[TriBob]

I've taken a detour to get some ideas out of my head concerning making the binary log files self-convert to CSV by wrapping the data with a Bash script.

The key concept is Bash's notion of "Here Documents". But for a 500 MB data file, Bash itself could take forever, so I decided to look at a double "Here Document" : The first will contain a converter in C that will be piped to the local GCC, and the second will invoke the compiled program and pass it the binary data. Here's what I've got so far (untested):

Code: Select all

==== Copied by firmware at file creation ====
   #!/usr/bin/env bash
==== Generated by firmware from config data ====
   # Produced by firmware version 0.0.0.0.0.1 alpha
   column_count = 4
   header_string = '"Time(TickCnt), AccelX, AccelY, AccelZ\r\n"'
   item_types = ("uint16_t", "int16_t", "int16_t", "int16_t")
   format_string = '"%hd, %hu, %hu, %hu\r\n"'
==== Copied by firmware ====
   # Code generator for output line item type casting
   print_items=""
   for((i=0; $i<$column_count; $i++)); do
     print_items="${print_items}(${item_types}[$i])ntohs(val[$i])"
     if (($i < ($column_count - 1))); then
       print_items="${print_items}, "
     fi
   done
   # Create the converter executable
   # Use "<<-" to permit code to be indented for readability.
   # Allow Bash variable expansion/substitution.
   gcc -xc /tmp/convertit - <<-THE_C_CODE
       #include <stdio.h>
       #include <netinet/in.h>
       int main (int argc, char **argv) {
           uint16_t val[${column_count}];
           fputs( ${header_string}, stdout );
           while (1) {
               switch (fread( &val, 2, ${column_count}, stdin)) {
                   case (2*${column_count}): // Full read
                       break;
                   case 0:                 // EOF with nothing left
                       return 0;
                       break;
                   default:                // EOF with extra data
                       fputs(stderr, "Unexpected EOF.");
                       return -1;
                       break;
               }
               printf( ${format_string}, ${print_items} );
           }
           return 0;   // NOT REACHED
       }
   THE_C_CODE
   # Pass the binary data to the converter
   # Hard-quote the Here tag to prevent subsequent expansion/substitution
   /tmp/convertit >./$1.csv <<'THE_BINARY_DATA'
==== Logging of Binary Data Starts ====
   ...
   ... hundreds of megabytes of semi-random data ...
   ...
==== Copied by firmware at end of logging ====
   THE_BINARY_DATA
   rm /tmp/convertit
   exit 0

My main concerns at this point are if Bash will be a bottleneck passing the binary data to the C program. I may need to have the C program read the file directly, which will complicate things a little.

[TriBob]

I neet to stop slapping my firehead so often, or I'll start bruising.

I just realized that an awesome ADC simplification should be possible: Acquire ALL the ADCs ALL the time in burst mode, then in the DONE IRQ handler use the config file info to determine which ones get logged, and which get ignored.

It's not like the ADCs will wear out or anything, and it greatly simplifies ADC configuration. Plus, the battery voltage is fed to an ADC that isn't logged, which makes its value available fro free every cycle.

[Mee_n_Mac]

I just realized that an awesome ADC simplification should be possible: Acquire ALL the ADCs ALL the time in burst mode, then in the DONE IRQ handler use the config file info to determine which ones get logged, and which get ignored.

Will you be sacrificing sampling rate for those times when all the channels aren't wanted ? N channels takes N times as long as 1 channel.

[TriBob]

I just realized that an awesome ADC simplification should be possible: Acquire ALL the ADCs ALL the time in burst mode, then in the DONE IRQ handler use the config file info to determine which ones get logged, and which get ignored.

Will you be sacrificing sampling rate for those times when all the channels aren't wanted ? N channels takes N times as long as 1 channel.

Not really: I plan to run acquisition at a fairly high rate, so that whenever you periodically read any ADC register, it is more recent than the last time you read it. Most people don't realize that 'perfect' ADC timing is rarely needed: 'Fast enough' is 'good enough' for the vast majority of cases, especially if the individual inputs aren't correlated (e.g., not measuring 2 aspects of a single signal, such as current and voltage, or I & Q amplitudes, or amplitude and phase).

For example, if I plan to read the ADC registers at 1 KHz, it doesn't matter if the ADC itself is running at 1 KHz or 1.1 KHz or 10 KHz or 100 KHz: The value in the register will always be 'fresher' than 1 KHz. Yes, it sounds sloppy, but getting rid of the need to synchronize to the ADC rate can free up lots of CPU time.

So, if I want 14 channels of data at 3200 Hz from 2 ADCs (8 from one and 6 from the other), I'd simply run both ADCs at a conversion rate of 26 KHz or higher. This scheme will work up to the max conversion rate of the ADC, which is a bit over 1 MHz for 10-bit data (or ~128 KHz/channel for 8 channels).

The ADCs seem to use very little power, so high sampling rates don't seem to have high power costs (though I don't yet have hard numbers for this). Getting rid of polling and ADC interrupts will let me use a simple timer interrupt, and also permit me to idle the processor between updates.

There are other possibilities to consider: I can use multiples of the desired sampling rate to increase the net sample width. I can sample at 2x my desired output rate, then use a single ADC 'done' interrupt to sum every pair of 10-bit samples to create an 11-bit output value. And if I wanted 12-bit data at 3200 Hz, I could run the ADCs at a 4x rate then sum 4 consecutive 10-bit samples to create each 12-bit sample. (That's one interrupt per SET of ADC conversions, not for each individual one.)

Unfortunately, the S/N can degrade faster than the bit depth increases (for a couple of reasons), so this technique is often self-limiting beyond a few bits of added resolution. (Adding 256 8-bit samples won't create a very good 16-bit value!) But I'm storing the data in 16-bits anyway, so increasing the bit depth a little beyond 10 bits can't hurt, especially if it is 'free'.

[TomKoorts]

TriBob,

I believe I’m basically undertaking the same task as you - combining the Logomatic v2 and ADXL345 to create an accelerometer data logger.

Since the ADXL345 requires some initialisation, can this be performed on the Logomatic or do I require something else?

I’m pretty new to this stuff but I’ve read through your thread and found some really helpful information.

Is there any advice you would give me in hindsight of your efforts?