I've noticed some pretty severe latencies with the summary data these last couple of days. Yesterday, the 15-minute summary data was 6-7 hours behind the realtime data. As of right now, it appears stuck about 26 hours behind. Is there a problem on the back end? I don't see any real continuity issues with my realtime data.
I've deleted my key from the URLs below, but for illustration's sake...
It looks like things were fixed sometime shortly before noon today, although as the following graph from the Widget shows, about 26 hours worth of summary data were simply never generated, from roughly 7am on the 29th to 9am on the 30th. Bummer. Our meter only spits out a pulse every 10 gallons, so the data appear rather sparse, but if you look at the bottom of the graph, you can see little tick marks where the 0's are, and even these 0's are missing for about 26 hours, indicating no records. I checked the realtime data many times during this period (the URL in my original post), so I know my Push was sending data and the pulse counts were incrementing.
IMG_0578.PNG (116.17 KiB) Viewed 12227 times
At this point I'm prepared to conclude that there was some back end outage that caused the summary data to not be generated for more than a day, but it would still be good for my sanity to have some validation.
Hmm. The summary data seems to be having issues again. It would be great if somebody from EKM would chime in on this. It seems there are some serious back end issues afoot. Looking now, not only is my 15-minute summary data missing, the whole year's gone blank. Only an hour ago, I was looking at a day's worth of useful data. 300+ gallons. This problem is not just in the widget. The API build URLs are returning nothing at the moment. Please let us know what's up.
Hey, Im sorry to leave you hanging! I just saw your posts in the forum.
It sounds like you have experienced 3 overlapping issues:
1. The Summary API is sometimes not closely coupled in time to the real-time API. The summary API will always be a little behind the real-time API but should not ever be more than 60-90 minutes behind. Yes, we have been working on the summary API recently and have had a few customers bring up this lag, we hope to have it more closely coupled within a week or two.
2. The gap in your dataset, Im not clear what this would be about. We did have an issue recently where 2 of our servers were not communicating with each other, I would guess it was during the time period that you are showing here. Will you let us know if you see this again?
3. The Widget not showing the historical data from the Summary API. Yes, more than likely your data was in the Summary API, but the widget was not able to show it. It had something to do with CORS, we did a roll-back and now it seems to be working again. Will you try your widget again? You might need to tweak your widget URL a bit or look at a historical data set that you have not looked at recently to avoid getting cached data.
Thanks Jameson. I was just in the process of updating the support ticket to mention that the data became available a few minutes ago. Unlike last week, they appear to have been getting generated even though they were unavailable. I'm not generally using the Widget, and have instead built a Python app that polls the 15-minute summary data at regular intervals. That code adds all the no-cache business to the http headers and has not so far had issues with caching (that I'm aware of). I definitely have seen that with the Widget, though.
Is there a service bulletin mailing list or some other Announcements-type place where these sorts of issues are posted about? If not, there's my Suggestion Box submission.
Im afraid we don't have an API status page for the real-time API and the summary API. At this point the quickest is just a short email to us (with the API call or widget link you are using) and we will get back to you.
You could go to our public account and see if the real-time data and summary API data is working there, if we have a lot of issues in that account, then you will know we have broader issues than one that just affects your account for example. Here is the link:
Hi Jameson, EKM folks - Me again. (Sorry to be the perennial cry-baby when it comes to this particular issue.) Would it be possible to expand the raw query limit beyond 1000 in lieu of a fix to this summary data latency issue? So that at least a whole 24 hours can be examined without needing to rely on the often-catatonic summary data generator? As of now, my summary data hasn't been updated for over 12 hours, which is honestly... *shrug*... pretty lame. My hope is to always be able to see the last 24-hours of usage. It's not terrible if things are half an hour or even a couple of hours behind, but when they're frequently 6-12 hours behind, the value of a query with 15-minute granularity is severely eroded. I keep thinking to myself that I should build my own database of raw results and just keep it up-to-date with raw queries, but then it just doesn't seem like I should have to do that. I'd much rather see the API keep its implicit promise here and be more timely with the summary results.
Although if I could have at least 1500 raw readings, I'd (probably) shut up and never ask about this again.
A further observation from last night: The summary data did eventually catch up, very shortly after I sent the above message. It looked fine. Curiously, later in the evening, I looked at the graph in the widget and that (approximately) 12-hour period had disappeared from the data, as had several short periods after.
IMG_1317.PNG (172.27 KiB) Viewed 7109 times
It then reappeared sometime overnight (excerpted from a two-day query):
Screen Shot 2022-09-05 at 8.44.26 AM.png (61.94 KiB) Viewed 7109 times
Note that, in addition to the period in question, you can see in the images above that a random smattering of records after 12pm had also disappeared and then reappeared.
Sorry to hear that you experienced the trouble with the Summary API last weekend. Yes, there was an outage with the Summary server having to do with Mongo DB, we did get it fixed.
We can change your meter read rate to once every 90 seconds or once every 120 seconds etc, so that your 1000 real-time reads span more overall time. That way your real-time meter reads will cover more than 24 hours.
I can see your screenshot that shows a gaps in the data, then the second screenshot shows that the data got filled in. In the second screenshot, I only see the gaps being filled in, I dont see the data being altered, is that what you are also seeing?
Again, sorry for the trouble, let us know if you see any other issues. Thanks
Hi, Jameson - Thanks for the helpful reply as always. 90-second reads would be faaAAAAAaabulous! What do I need to do?
As far as what I observed: The summary data was running about 12 hours behind. It then caught up. Some time later, those 12 hours worth of summary data disappeared again, as did a few random 15-minute summary results in the following hours. Very odd. And then a few hours later, it was all back again.
This wasn't all bad, though. It exposed a bug in my own code where I was assuming that the summary results were contiguous. When those 12 hours disappeared again, my gallons counter merged two days worth of usage and spit out a number that had me running out the door to look for the leak, heh.