Posting data to SparkFun server

Everything pertaining to the data.sparkfun.com service, the phant project which powers it, and user projects which talk to the service.

Moderators: phalanx, toddtreece, Brennen@SparkFun

Post Reply
User avatar
sparky
Site Admin
Posts: 1337
Joined: Mon May 05, 2003 6:02 pm
Location: Colorado
Contact:

Posting data to SparkFun server

Post by sparky » Wed Apr 30, 2014 8:18 am

Hi all,

I wanted to give the forum readers a heads up about a data channel service that we've been working on. Check it out here: http://data.sparkfun.com.

I was thrilled by how easy Wunderground was to work with and then let down by the complexities of using other services. So we created Phant (as in elephants don't forget) to make it super easy for people to push data out to the cloud.

For example, if you have a remote float sensor monitoring the water level of a de-watering pit (as we do in the new SparkFun building) and you want to report the water level every few minutes, where do you push the data? I'm a hardware guy. I don't know how to setup a server to save my life, so data.sparkfun.com fills the gap. All you do is create a stream, record the keys, then have your remote sensor concatenate a string.

http://data.sparkfun.com/input/Jxyjr7Dm ... e%20forum!
(go ahead, you know you want to click it!)

You can see the stream ID [Jxyjr7DmxwTD5dG1D1Kv] and the private key [gzgnB4VazkIg7GN1g1qA]. I don't know how to maintain a SQL database but I know how to get my Arduino to stick some strings together and then post it to HTTP (via cellular, wifi, or ethernet).

Then you can see the resulting data on the public stream: http://data.sparkfun.com/streams/Jxyjr7DmxwTD5dG1D1Kv

It's all free, open source goodness so if you don't trust us, need more than 50MB of space or faster than 10s update rates you can deploy Phant on your own server.

You're the first group to really hear about it outside the walls of SparkFun so we'd be thrilled if you kicked the tires and let us know what you think. We want to keep it simple (stupid) but what could be better?

Cheers,
-Nathan
Pursue the cheese, win the excellence.

User avatar
sparky
Site Admin
Posts: 1337
Joined: Mon May 05, 2003 6:02 pm
Location: Colorado
Contact:

Re: Posting data to SparkFun server

Post by sparky » Wed Apr 30, 2014 8:25 am

For a larger data set example, checkout the Wimp Weather Station tutorial and the feed it reports to. Here's the (relatively) simple code that pushes the data to the channel:

Code: Select all

    //Now we form the large string to pass to sparkfun
    local strSparkFun = "http://data.sparkfun.com/input/";
    local publicKey = "dZ4EVmE8yGCRGx5XRX1W";
    local privateKey = "private_key=myPrivateKey";

    bigString = strSparkFun;
    bigString += publicKey;
    bigString += "?" + privateKey;
    bigString += "&" + winddir;
    bigString += "&" + windspeedmph;
    bigString += "&" + humidity;
    bigString += "&" + tempf;
    bigString += "&" + rainin;
    bigString += "&" + baromin;
    
    //Push to SparkFun
    local request = http.get(bigString);
    local response = request.sendsync();
Pursue the cheese, win the excellence.

User avatar
phalanx
Non-SFE Guru
Posts: 1948
Joined: Sun Nov 30, 2003 8:57 am
Location: Candia, NH

Re: Posting data to SparkFun server

Post by phalanx » Wed Apr 30, 2014 8:33 am

Hi Nate!

This is a great idea. Like you, my experience doesn't lie in the server side of the equation. I'll be curious to see the creative ways people utilize it.

-Bill

Kamiquasi
Posts: 113
Joined: Sat Jan 15, 2011 4:46 pm

Re: Posting data to SparkFun server

Post by Kamiquasi » Thu May 01, 2014 1:58 pm

Tire-kicking has commenced - looks like it's okay with unicode, basic attempts at inserting html, and at least Bobby-proof.

I've also set up an actual data set - SparkFun Product Mutations nothing only one item in there yet, but should gradually fill out.

IRC answered some Q&A:
The timestamp column is always present - no need to add one yourself.
If the data exceeds 50MB, the oldest data starts getting deleted.

lyndon
Support Volunteer
Posts: 1536
Joined: Thu Jul 17, 2008 6:37 pm
Location: Minnesota, USA

Re: Posting data to SparkFun server

Post by lyndon » Thu May 01, 2014 2:53 pm

This is awesome.

As luck would have it, I've been working on exactly the same thing in my spare time over the last week (well, a private version anyway). I'll be sure to check it out. Do you plan to support MQTT?

lyndon
Support Volunteer
Posts: 1536
Joined: Thu Jul 17, 2008 6:37 pm
Location: Minnesota, USA

Re: Posting data to SparkFun server

Post by lyndon » Thu May 01, 2014 5:16 pm

And it looks like you've been visited by little Bobby Tables ;-)

Kamiquasi
Posts: 113
Joined: Sat Jan 15, 2011 4:46 pm

Re: Posting data to SparkFun server

Post by Kamiquasi » Thu May 01, 2014 5:39 pm

oops.. it didn't like me submitting a value twice - e.g. &brewTemp=1&brewTemp=2 .. I killed your timestamp header :(

languer
Posts: 85
Joined: Fri Apr 09, 2010 12:48 pm
Location: Arizona

Re: Posting data to SparkFun server

Post by languer » Sat May 03, 2014 1:36 pm

Wow. Again, wow. This is so great and so welcomed. I had a wifly and ethernet shield, but I could not for the life of me make them talk to the outside world (outside of my router) without paying for a service.

Mee_n_Mac
Support Volunteer
Posts: 5628
Joined: Thu Apr 28, 2011 4:10 pm

Re: Posting data to SparkFun server

Post by Mee_n_Mac » Sat May 03, 2014 9:03 pm

I've been inspired by this to hack the "weather station" at the lake house. But instead of getting indoor and outdoor and lake water temps, it'll get outdoor and lake water temps and lake levels, posted in a publicly accessible way. I might even process the "fast" lake level data to give wave state and then add a anemometer ... data people going out to boat will find useful as it'll be timely (unlike the Weather Channel).

Something to do instead of drinki ... errr ... while drinking beer this summer !

User avatar
toddtreece
Posts: 2
Joined: Mon May 05, 2014 6:07 am
Location: Uniontown, MD
Contact:

Re: Posting data to SparkFun server

Post by toddtreece » Mon May 05, 2014 6:17 am

Kamiquasi wrote:oops.. it didn't like me submitting a value twice - e.g. &brewTemp=1&brewTemp=2 .. I killed your timestamp header :(
thanks for reporting the issue! it should now throw an error if you submit a field twice in the same request. i'll try to monitor this thread, but feel free to report bugs here: https://github.com/sparkfun/phant/issues

User avatar
toddtreece
Posts: 2
Joined: Mon May 05, 2014 6:07 am
Location: Uniontown, MD
Contact:

Re: Posting data to SparkFun server

Post by toddtreece » Mon May 05, 2014 8:35 am

lyndon wrote:This is awesome.

As luck would have it, I've been working on exactly the same thing in my spare time over the last week (well, a private version anyway). I'll be sure to check it out. Do you plan to support MQTT?
Thanks! MQTT support should be available in the next couple weeks.

If you would like to host your own private version of the server, check out: https://github.com/sparkfun/phant

Tubes
Posts: 2
Joined: Tue Oct 07, 2014 7:25 pm

Re: Posting data to SparkFun server

Post by Tubes » Wed Oct 22, 2014 5:56 am

Hey Sparky, this is all great stuff. Can you point me toward some examples on how to post data to data.sparkfun.com using a RedBoard and the Sparkfun SM5100B cellular shield? I am working on a project with these items and getting a head start would be great! Thanks.

Reuben

agilb
Posts: 12
Joined: Wed Apr 11, 2012 8:23 am
Location: Utah, USA

Re: Posting data to SparkFun server

Post by agilb » Wed May 13, 2015 7:34 am

I am curious and will start testing, but must all the fields be populated, or can one device update the wind speed while another updates the water level or temperature while making all the data available in the same log?

nigelibrown
Posts: 1
Joined: Wed Dec 09, 2015 9:30 am

Re: Posting data to SparkFun server

Post by nigelibrown » Thu Dec 10, 2015 1:43 am

Here is some series data from a recent collection, if you look at the 4th line you can see the timestamp is incorrect i would expect it to be 12:12 this is happening quite frequently?

0 -69,20.67,69.22,2015-12-01T00:15:11.854Z
1 -72,20.68,69.21,2015-12-01T00:14:41.208Z
2 -72,20.68,69.27,2015-12-01T00:13:22.406Z
3 -71,20.7,69.19,2015-12-01T00:11:12.790Z
4 -71,20.66,69.2,2015-12-01T00:11:23.927Z
5 -70,20.72,69.28,2015-12-01T00:10:22.886Z
6 -70,20.68,69.23,2015-12-01T00:09:23.621Z
7 -71,20.69,69.23,2015-12-01T00:08:23.265Z

Post Reply