Unable to post data regularly

Everything pertaining to the data.sparkfun.com service, the phant project which powers it, and user projects which talk to the service.

Moderators: phalanx, toddtreece, Brennen@SparkFun

Post Reply
clipcorner
Posts: 1
Joined: Wed Dec 16, 2015 12:20 pm

Unable to post data regularly

Post by clipcorner » Wed Dec 16, 2015 12:29 pm

Are there problems with the data server?
After successfully posting data every 15 minutes since November 14 (2015), since just after 19:15Z on 2015-12-15, my data is posting only intermittently & irregularly. Rarely every 15 minutes; typically gaps of 45 minutes up to 2 hours. Unfortunately I am remote from the device (hence its utility) so I'm not 100% certain it is sending data every 15 minutes. But in the month of close monitoring it never skipped a report, and I am able to verify that the remote wi-fi is functioning.

Miskatonic
Posts: 4
Joined: Wed Feb 18, 2015 8:33 pm

Re: Unable to post data regularly

Post by Miskatonic » Thu Dec 24, 2015 8:27 am

Is this still an issue? We have occasional server drops during maintenance, but if there is a large scale rolling outage, we know about it almost immediately from users.
Is it possible your power supply is on the fritz? Sometimes when battery/solar powered projects get to a low current star this appears, particularly with RF applications in the loop.

foxt
Posts: 3
Joined: Mon Jan 04, 2016 4:24 pm

Re: Unable to post data regularly

Post by foxt » Mon Jan 04, 2016 4:29 pm

This is still an issue at the moment. HTTPS posts are not being processed from my electric imp agent, but HTTP posts are. Over on the electric imp forum, Hugo has done a bit of investigating and suggested that the problem is on the sparkfun side. I know that others have reported the same problem with HTTPS posts from their imp agents. The problem seems to have started several days ago, when HTTPS posts were failing intermittently. Today, they fail 100% of the time (at least for me). Again, HTTP posts succeed, perhaps that is a clue?

agilb
Posts: 12
Joined: Wed Apr 11, 2012 8:23 am
Location: Utah, USA

Re: Unable to post data regularly

Post by agilb » Wed Jan 06, 2016 9:58 am

some of my streams have dropped and aren't posting any longer.

Minimizer
Posts: 9
Joined: Tue Aug 16, 2011 9:05 pm

Re: Unable to post data regularly

Post by Minimizer » Sat Jan 09, 2016 10:36 pm

Good to read I'm not alone here or missing something .. my posts also (from an Electric Imp agent) stopped abruptly on or about Dec. 30. Until then I'd been successfully posting using the basic SF code here ..

https://learn.sparkfun.com/tutorials/pu ... ectric-imp

hfiennes
Posts: 2
Joined: Sun Jan 10, 2016 10:15 pm

Re: Unable to post data regularly

Post by hfiennes » Sun Jan 10, 2016 10:23 pm

As @minimizer and @foxt have noted, some imp users are seeing issues posting with https (and only https) to data.sparkfun.com. Moving to http avoids the issue, though is obviously not ideal.

I took some tcpdumps from the imp server end and I can often see the sparkfun end disconnecting immediately after the TLS client hello is sent. The issues are a lot more frequent every 5 mins or so, for a few seconds near the top of the minute, eg:

2016-01-01T04:15:03-05
2016-01-01T04:20:01-04
2016-01-01T04:25:01-03
2016-01-01T04:30:02-06
2016-01-01T04:35:01-06

...which to me implies that lots of people are posting with https at aligned times and the server is getting a bit overloaded/upset just for a few seconds.

I can provide more info if needed, our customers are doing many many many thousands of requests to data.sparkfun.com per day :)

foxt
Posts: 3
Joined: Mon Jan 04, 2016 4:24 pm

Re: Unable to post data regularly

Post by foxt » Tue Jan 12, 2016 4:13 pm

Thanks for the insight. I switched to http requests to get around the problem, but I would like to switch back. I'll look at programming the imp to try to connect at different times and see if that helps. In the meantime, this had been working well for months and just started having an issue late December. Do we know if anything changed on the sparkfun side recently?

Miskatonic
Posts: 4
Joined: Wed Feb 18, 2015 8:33 pm

Re: Unable to post data regularly

Post by Miskatonic » Wed Jan 13, 2016 5:21 pm

We are looking at the IT side of things. There has been some DOS activity on Phant lately that has caused some issues. Also we are seeing users "over throttle " their feeds. If you are over the 100post/15mins, your feed will shutdown for an hour. This is how we keep bandwidth up for all users. The particulars of this can be read here: http://phant.io/docs/input/limit/

foxt
Posts: 3
Joined: Mon Jan 04, 2016 4:24 pm

Re: Unable to post data regularly

Post by foxt » Thu Jan 14, 2016 3:26 pm

Thanks for the response. My imp project posts to sparkfun via the imp agent service every 15 minutes. Do my posts appear to be coming from imp, or from me? I suspect they look like they are coming from imp, and when thinking about the great tutorials etc. that show how to make the imp work with sparkfun, I bet that there are many many people like me. Does that make the imp service look like one of those users who "over throttle"?

hfiennes
Posts: 2
Joined: Sun Jan 10, 2016 10:15 pm

Re: Unable to post data regularly

Post by hfiennes » Thu Feb 04, 2016 12:42 am

I just looked in our request logs and things appear to be working much better now. I've not gone back to see when things started working again, but I'm seeing successful connects with HTTPS consistently now. Thanks!

Post Reply