Summary data is severely latent

[jaholmes]

I've noticed some pretty severe latencies with the summary data these last couple of days. Yesterday, the 15-minute summary data was 6-7 hours behind the realtime data. As of right now, it appears stuck about 26 hours behind. Is there a problem on the back end? I don't see any real continuity issues with my realtime data.

I've deleted my key from the URLs below, but for illustration's sake...

The following URL shows my last 1000 reports, constrained to the Pulse_Cnt_1 field:
https://api.ekmpush.com/readMeter?key=& ... ulse_Cnt_1

Scanning through these, I don't see any major discontinuities or other problems. It looks like what I'd expect. The latest report is current as of my writing:

Code: Select all

Good	Date		Time		Time_Stamp_UTC_ms	Pulse_Cnt_1
1	2021-09-30	14:47:39.618	1633013259618		22784

The following URL shows my 15-minute summary data:
https://summary.ekmmetering.com/summary ... ulse_Cnt_1*

The latest of these is, as of my writing, roughly 26 hours in the past:

Code: Select all

Server: sumdb-c1-s2-r2    
Summary Info	Pulse Cnt 1
Start TS: 1632919519517 Zoom Out 
End TS: 1632920359480 
Start Date: Wed Sep 29 2021 12:45:19 GMT+0000 (GMT) 
End Date: Wed Sep 29 2021 12:59:19 GMT+0000 (GMT)
...etc.

Ouch. Any idea what the holdup is?

[jaholmes]

It looks like things were fixed sometime shortly before noon today, although as the following graph from the Widget shows, about 26 hours worth of summary data were simply never generated, from roughly 7am on the 29th to 9am on the 30th. Bummer. Our meter only spits out a pulse every 10 gallons, so the data appear rather sparse, but if you look at the bottom of the graph, you can see little tick marks where the 0's are, and even these 0's are missing for about 26 hours, indicating no records. I checked the realtime data many times during this period (the URL in my original post), so I know my Push was sending data and the pulse counts were incrementing.

IMG_0578.PNG (116.17 KiB) Viewed 2053 times

At this point I'm prepared to conclude that there was some back end outage that caused the summary data to not be generated for more than a day, but it would still be good for my sanity to have some validation.

[jaholmes]

Hmm. The summary data seems to be having issues again. It would be great if somebody from EKM would chime in on this. It seems there are some serious back end issues afoot. Looking now, not only is my 15-minute summary data missing, the whole year's gone blank. Only an hour ago, I was looking at a day's worth of useful data. 300+ gallons. This problem is not just in the widget. The API build URLs are returning nothing at the moment. Please let us know what's up.

One day:

day.png (84.37 KiB) Viewed 1980 times

By month:

year.png (78.54 KiB) Viewed 1980 times

[Jameson]

Hey, Im sorry to leave you hanging! I just saw your posts in the forum.

It sounds like you have experienced 3 overlapping issues:

1. The Summary API is sometimes not closely coupled in time to the real-time API. The summary API will always be a little behind the real-time API but should not ever be more than 60-90 minutes behind. Yes, we have been working on the summary API recently and have had a few customers bring up this lag, we hope to have it more closely coupled within a week or two.

2. The gap in your dataset, Im not clear what this would be about. We did have an issue recently where 2 of our servers were not communicating with each other, I would guess it was during the time period that you are showing here. Will you let us know if you see this again?

3. The Widget not showing the historical data from the Summary API. Yes, more than likely your data was in the Summary API, but the widget was not able to show it. It had something to do with CORS, we did a roll-back and now it seems to be working again. Will you try your widget again? You might need to tweak your widget URL a bit or look at a historical data set that you have not looked at recently to avoid getting cached data.

[jaholmes]

Thanks Jameson. I was just in the process of updating the support ticket to mention that the data became available a few minutes ago. Unlike last week, they appear to have been getting generated even though they were unavailable. I'm not generally using the Widget, and have instead built a Python app that polls the 15-minute summary data at regular intervals. That code adds all the no-cache business to the http headers and has not so far had issues with caching (that I'm aware of). I definitely have seen that with the Widget, though.

Is there a service bulletin mailing list or some other Announcements-type place where these sorts of issues are posted about? If not, there's my Suggestion Box submission.

[Jameson]

Im afraid we don't have an API status page for the real-time API and the summary API. At this point the quickest is just a short email to us (with the API call or widget link you are using) and we will get back to you.

You could go to our public account and see if the real-time data and summary API data is working there, if we have a lot of issues in that account, then you will know we have broader issues than one that just affects your account for example. Here is the link:

https://api.ekmpush.com/account.html?key=MTAxMDoyMDIw

Someday we might get our act together and post statuses to something like twitter, but we need more mental bandwidth. Thanks for the suggestion in any case!

Feel free to post your python script to the forum if you think others would find it useful.