SparkFun Forums

[mitchell_giese]

I recently purchased a OpenLog Artemis (DEV-16832 ROHS) to do some high baud rate serial data logging. I was previously using the DEV-13712 ROHS but the 115200 max baud was not enough for my needs.

My setup is extremely simple, I have a device that takes usb-c power and outputs serial data at 3.0v. I am connecting both the OLA board and my device to the same power source (shared ground) via usb-c cables and then running a jumper cable between the tx line on my device and the rx pin on the OLA. I have configured both of the devices to operate at 500,000 baud. I have also updated the OLA to the latest available firmware and confirmed that enableSD=1 and logSerial=1. All other logging is disabled.

I have confirmed that 3v of serial data is coming across the tx line, but when I power them both on I have had very limited success getting the OLA to log the data. When I press the reset button to wake it from deep sleep, I see the stat light blinking rapidly indicating that something is writing to the SD card. When I power down the device and remove that card, the serialLog file is 0KB most of the time. I have gotten it to work a few times, and validated that the data is correct, but I can't consistently repeat whatever it is I am doing.

Any help will be greatly appreciated!

[PaulZC]

Hi,

The only way to guarantee the serial log file is closed correctly is to add a stop-logging button on Pin 32. If you disconnect the power while the OLA is still logging, the file may be left open and will appear zero size. Please have a look at:

https://learn.sparkfun.com/tutorials/op ... figuration (Scroll down to the "Configure Power Options" section)

https://learn.sparkfun.com/tutorials/op ... iderations

Best wishes,
Paul

[mitchell_giese]

Thanks for the help Paul. I got the button wired up but I am still having some issues. If I let the logging go for a minute or 2 and then press the button it does nothing. The stat light continues to flash showing me data is still writing to the SD card. It is not until I actually disconnect the serial input and then press the button that the OLA goes to sleep and stops logging. Additionally, when reviewing the data after a successful power down from a 2+ minute logging session a large portion of the data seems to be corrupt. I have confirmed that the corruption is happening on the OLA side and that the average data rate should only be about 43 KB/s. Any more ideas as to how I might resolve this? Thanks!

[PaulZC]

Hi,

In OLA terms, 43KB/s is a _lot_ of data. I suspect what is happening is that the OLA code is getting stuck in the "if serial data is available, write it to SD card" loop and is never getting as far as servicing the stop-logging button. Also, if the SD card write takes longer than usual - which it does occasionally as the card performs some internal juggling - then at that data rate you will get a serial buffer overflow and data will be lost.

Using a freshly-formatted SD card may help. Pleased make sure you use the official SD Association Formatter:

https://www.sdcard.org/downloads/formatter/

I'm afraid the standard OpenLog Artemis firmware is not a good choice for your application. It might be possible to write dedicated code to resolve this. The code would need two threads, one to place the serial data into a large ring buffer and a second to write data from the buffer to SD card.

Best wishes,
Paul

[PaulZC]

Hi @mitchell_giese,

I've been experimenting with a special build of the OpenLog Artemis firmware, dedicated to high speed serial logging. Now, don't get your hopes up... It doesn't (yet) do what you need. It works well at baud rates up to 230400, with near continuous logging. I've tested it with 10Hz NMEA data from a u-blox ZED-F9P and the log files are clean at ~80% duty, ~20KB/s. The wheels come off at 460800 baud for reasons I don't yet understand.

If you are able to drop the baud rate on your sensor down to 230400, this firmware should work well for you.

https://github.com/sparkfun/OpenLog_Art ... ial_Logger

There are some bells and whistles I hope you will enjoy:
- You can echo the data back out on the TX pin if you want to
- It has built-in parsing / error checking for u-blox UBX and NMEA format data (I use this to check for dropped characters)
- The firmware will automatically open a new log file after a pre-defined interval or when you toggle Pin 11
- You can configure Pin 32 for stop-logging (like the regular OLA)
- If you enable the debug messages, you get periodic updates of the number of characters received
- You can timestamp the data using a pre-defined 'token' (like the regular OLA)

Please give it a try - and apologies again that it doesn't quite handle the data rate you need.

Best wishes,
Paul

[Snowjeb]

Hi Paul,

I am trying to use your openlog firmware. It seems exactly what I need. I want to log rawx data from a ZED-9F GPS module connected with QWIIC. I need raw, since I want the highest possible sample rate: ~30 Hz. Quite quickly I run into writing issues. Trying to understand your software, I think I can tweak:
#define sdWriteSize 512 --> maybe to 2048
#define fileBufferSize 16384 Have not found if the ZED9 can use a larger buffer
increase to: Serial.begin(230400);

Am I right, that these are the parameters to optimize? Do you have a suggestion?

Thanks,
Jeroen

[PaulZC]

Hi Jeroen,

Is ZED-9F a typo? Did you mean ZED-F9R? I haven't experimented with its "priority navigation mode" so I don't know if asking it to output RAWX at 30Hz from multiple constellations is possible or not. You will need to test it.

The RAWX frames can be large. When tracking all constellations, they can be over 2kBytes in size. Scale that up to 30Hz and you have a bit rate approaching 500 kBits/sec. And that doesn't include overheads. So you will struggle. 400kHz I2C won't be fast enough. 230400 Serial won't be fast enough either. The SPI interface can run at 125kBytes/sec, but I suspect you will run into a bottleneck trying to use one SPI bus for both GNSS and SD data...

If you really need that kind of speed, you may need to think about using a different board so you can use two SPI interfaces - one for GNSS and a separate one for SD.

We have a separate build of the OpenLog Artemis firmware dedicated to logging GNSS data. It might be a useful starting point for your tests?
https://github.com/sparkfun/OpenLog_Artemis_GNSS_Logger

The sdWriteSize is set by the SD library. Increasing it is unlikely to improve anything.

Yes, you will almost certainly need to increase the file buffer size. Again you will need to test this.

Increasing the serial speed may help, but my advice would be to print as little to the 'terminal' as possible so the code can concentrate on streaming the data to SD card.

Good luck with your project!
Paul

[Snowjeb]

Hi Paul,
Thank you overmuch for your quick reply!
I think the Zed-F9P (SparkFun GPS-RTK2 Board - ZED-F9P) can go up to 25 HZ in RAW with 1 or 2 constellations (GPS + (GAL or GLO or DBS)) and to 20 Hz in RTK mode with only GPS. Still have an accuracy of 0.01 m! I am a bit confused here, still very high accuracy and only 1-2 constellations. It's not easy to turn constellations of in Ublox software.
I try to use the OpenLog_Artemis_GNSS_Logger. Here the sdWriteSize is set (again?) The other settings I mentioned are from this code too.
So I gather: 1 either use SPI (2x) but then I need another Datalogger (Does SParkfun have one?) and 2. one constellation to reduce frames and try optimise buffer size, write to console (off).
Could a better SD card help?
Thanks for you help,

Jeroen

[PaulZC]

Hi Jeroen,

Please give the GNSS Logger code a try. It may do what you need? It will let you enable/disable constellations and set the rate for RAWX logging.

Moving to dual-SPI is a _much_ bigger project, but may be the only way to get the full data rate. You would need a single board that can support two SPI interfaces simultaneously: one for GNSS and one for SD. The SparkFun ESP32 Thing Plus C should do what you need, but I've never tried to use two SPI interfaces on that board. There are some tips on how to enable two SPI interfaces here:
https://github.com/espressif/arduino-esp32/issues/1219

The SparkFun GNSS Library supports SPI, but you would need to write your own code to configure the GNSS and then log the incoming data to SD card.

Using a good quality, freshly-formatted SD card will help, but it is not a "magic wand"...

Best wishes,
Paul