Summary data is severely latent

For more advanced meter data users. Learn from others and ask questions about how to read meters, how to use the EKM Push Data, and how to display the meter data in useful ways.
jaholmes
Posts: 17
Joined: Sat Mar 21, 2015 9:38 pm

Summary data is severely latent

Post by jaholmes »

I've noticed some pretty severe latencies with the summary data these last couple of days. Yesterday, the 15-minute summary data was 6-7 hours behind the realtime data. As of right now, it appears stuck about 26 hours behind. Is there a problem on the back end? I don't see any real continuity issues with my realtime data.

I've deleted my key from the URLs below, but for illustration's sake...

The following URL shows my last 1000 reports, constrained to the Pulse_Cnt_1 field:
https://api.ekmpush.com/readMeter?key=& ... ulse_Cnt_1

Scanning through these, I don't see any major discontinuities or other problems. It looks like what I'd expect. The latest report is current as of my writing:

Code: Select all

Good	Date		Time		Time_Stamp_UTC_ms	Pulse_Cnt_1
1	2021-09-30	14:47:39.618	1633013259618		22784
The following URL shows my 15-minute summary data:
https://summary.ekmmetering.com/summary ... ulse_Cnt_1*

The latest of these is, as of my writing, roughly 26 hours in the past:

Code: Select all

Server: sumdb-c1-s2-r2    
Summary Info	Pulse Cnt 1
Start TS: 1632919519517 Zoom Out 
End TS: 1632920359480 
Start Date: Wed Sep 29 2021 12:45:19 GMT+0000 (GMT) 
End Date: Wed Sep 29 2021 12:59:19 GMT+0000 (GMT)
...etc.
Ouch. Any idea what the holdup is?
jaholmes
Posts: 17
Joined: Sat Mar 21, 2015 9:38 pm

Re: Summary data is severely latent

Post by jaholmes »

It looks like things were fixed sometime shortly before noon today, although as the following graph from the Widget shows, about 26 hours worth of summary data were simply never generated, from roughly 7am on the 29th to 9am on the 30th. Bummer. Our meter only spits out a pulse every 10 gallons, so the data appear rather sparse, but if you look at the bottom of the graph, you can see little tick marks where the 0's are, and even these 0's are missing for about 26 hours, indicating no records. I checked the realtime data many times during this period (the URL in my original post), so I know my Push was sending data and the pulse counts were incrementing.
IMG_0578.PNG
IMG_0578.PNG (116.17 KiB) Viewed 34094 times
At this point I'm prepared to conclude that there was some back end outage that caused the summary data to not be generated for more than a day, but it would still be good for my sanity to have some validation. ;)
jaholmes
Posts: 17
Joined: Sat Mar 21, 2015 9:38 pm

Re: Summary data is severely latent

Post by jaholmes »

Hmm. The summary data seems to be having issues again. It would be great if somebody from EKM would chime in on this. It seems there are some serious back end issues afoot. Looking now, not only is my 15-minute summary data missing, the whole year's gone blank. Only an hour ago, I was looking at a day's worth of useful data. 300+ gallons. This problem is not just in the widget. The API build URLs are returning nothing at the moment. Please let us know what's up.

One day:
day.png
day.png (84.37 KiB) Viewed 34021 times
By month:
year.png
year.png (78.54 KiB) Viewed 34021 times
Jameson
Posts: 860
Joined: Fri Nov 04, 2011 7:42 pm
Location: Santa Cruz, CA
Contact:

Re: Summary data is severely latent

Post by Jameson »

Hey, Im sorry to leave you hanging! I just saw your posts in the forum.

It sounds like you have experienced 3 overlapping issues:

1. The Summary API is sometimes not closely coupled in time to the real-time API. The summary API will always be a little behind the real-time API but should not ever be more than 60-90 minutes behind. Yes, we have been working on the summary API recently and have had a few customers bring up this lag, we hope to have it more closely coupled within a week or two.

2. The gap in your dataset, Im not clear what this would be about. We did have an issue recently where 2 of our servers were not communicating with each other, I would guess it was during the time period that you are showing here. Will you let us know if you see this again?

3. The Widget not showing the historical data from the Summary API. Yes, more than likely your data was in the Summary API, but the widget was not able to show it. It had something to do with CORS, we did a roll-back and now it seems to be working again. Will you try your widget again? You might need to tweak your widget URL a bit or look at a historical data set that you have not looked at recently to avoid getting cached data.
Jameson
EKM METERING
http://www.ekmmetering.com
831.425.7371
jaholmes
Posts: 17
Joined: Sat Mar 21, 2015 9:38 pm

Re: Summary data is severely latent

Post by jaholmes »

Thanks Jameson. I was just in the process of updating the support ticket to mention that the data became available a few minutes ago. Unlike last week, they appear to have been getting generated even though they were unavailable. I'm not generally using the Widget, and have instead built a Python app that polls the 15-minute summary data at regular intervals. That code adds all the no-cache business to the http headers and has not so far had issues with caching (that I'm aware of). I definitely have seen that with the Widget, though.

Is there a service bulletin mailing list or some other Announcements-type place where these sorts of issues are posted about? If not, there's my Suggestion Box submission. :)
Jameson
Posts: 860
Joined: Fri Nov 04, 2011 7:42 pm
Location: Santa Cruz, CA
Contact:

Re: Summary data is severely latent

Post by Jameson »

Im afraid we don't have an API status page for the real-time API and the summary API. At this point the quickest is just a short email to us (with the API call or widget link you are using) and we will get back to you.

You could go to our public account and see if the real-time data and summary API data is working there, if we have a lot of issues in that account, then you will know we have broader issues than one that just affects your account for example. Here is the link:

https://api.ekmpush.com/account.html?key=MTAxMDoyMDIw

Someday we might get our act together and post statuses to something like twitter, but we need more mental bandwidth. Thanks for the suggestion in any case!

Feel free to post your python script to the forum if you think others would find it useful.
Jameson
EKM METERING
http://www.ekmmetering.com
831.425.7371
jaholmes
Posts: 17
Joined: Sat Mar 21, 2015 9:38 pm

Re: Summary data is severely latent

Post by jaholmes »

Hi Jameson, EKM folks - Me again. (Sorry to be the perennial cry-baby when it comes to this particular issue.) Would it be possible to expand the raw query limit beyond 1000 in lieu of a fix to this summary data latency issue? So that at least a whole 24 hours can be examined without needing to rely on the often-catatonic summary data generator? As of now, my summary data hasn't been updated for over 12 hours, which is honestly... *shrug*... pretty lame. :( My hope is to always be able to see the last 24-hours of usage. It's not terrible if things are half an hour or even a couple of hours behind, but when they're frequently 6-12 hours behind, the value of a query with 15-minute granularity is severely eroded. I keep thinking to myself that I should build my own database of raw results and just keep it up-to-date with raw queries, but then it just doesn't seem like I should have to do that. I'd much rather see the API keep its implicit promise here and be more timely with the summary results.

Although if I could have at least 1500 raw readings, I'd (probably) shut up and never ask about this again. :)
jaholmes
Posts: 17
Joined: Sat Mar 21, 2015 9:38 pm

Re: Summary data is severely latent

Post by jaholmes »

A further observation from last night: The summary data did eventually catch up, very shortly after I sent the above message. It looked fine. Curiously, later in the evening, I looked at the graph in the widget and that (approximately) 12-hour period had disappeared from the data, as had several short periods after.
IMG_1317.PNG
IMG_1317.PNG (172.27 KiB) Viewed 28976 times

It then reappeared sometime overnight (excerpted from a two-day query):
Screen Shot 2022-09-05 at 8.44.26 AM.png
Screen Shot 2022-09-05 at 8.44.26 AM.png (61.94 KiB) Viewed 28976 times
Note that, in addition to the period in question, you can see in the images above that a random smattering of records after 12pm had also disappeared and then reappeared.

Grumble, grumble. :/
Jameson
Posts: 860
Joined: Fri Nov 04, 2011 7:42 pm
Location: Santa Cruz, CA
Contact:

Re: Summary data is severely latent

Post by Jameson »

Sorry to hear that you experienced the trouble with the Summary API last weekend. Yes, there was an outage with the Summary server having to do with Mongo DB, we did get it fixed.

We can change your meter read rate to once every 90 seconds or once every 120 seconds etc, so that your 1000 real-time reads span more overall time. That way your real-time meter reads will cover more than 24 hours.

I can see your screenshot that shows a gaps in the data, then the second screenshot shows that the data got filled in. In the second screenshot, I only see the gaps being filled in, I dont see the data being altered, is that what you are also seeing?

Again, sorry for the trouble, let us know if you see any other issues. Thanks
Jameson
EKM METERING
http://www.ekmmetering.com
831.425.7371
jaholmes
Posts: 17
Joined: Sat Mar 21, 2015 9:38 pm

Re: Summary data is severely latent

Post by jaholmes »

Hi, Jameson - Thanks for the helpful reply as always. 90-second reads would be faaAAAAAaabulous! :) What do I need to do?

As far as what I observed: The summary data was running about 12 hours behind. It then caught up. Some time later, those 12 hours worth of summary data disappeared again, as did a few random 15-minute summary results in the following hours. Very odd. And then a few hours later, it was all back again.

This wasn't all bad, though. It exposed a bug in my own code where I was assuming that the summary results were contiguous. When those 12 hours disappeared again, my gallons counter merged two days worth of usage and spit out a number that had me running out the door to look for the leak, heh.
Post Reply