Page 1 of 2

Data Storage Capacity

Posted: Fri Aug 18, 2017 1:43 pm
by romemeek
Hello,

I have two questions. How long is the data from my push device stored on the cloud server?
Is there a limit to the amount of data that can be stored?

Thanks!

Re: Data Storage Capacity

Posted: Thu Aug 31, 2017 4:11 pm
by Jameson
We store the latest 1000 real time reads. By default meters are read once per minute, so this is the latest 1000 minutes of data.

Here is more information about our Real-Time API: http://documents.ekmmetering.com/api-do ... altime-api

===

We then take these real-time reads and then summarize them down into 15 minute chunks that are available via our Summary API. We store these 15 minute chunks forever.

Here is more information about our Summary API: http://documents.ekmmetering.com/api-do ... ummary-api

===

Let me know if this does not answer your question.

Re: Data Storage Capacity

Posted: Sat May 05, 2018 3:48 am
by Caruana
That's good to know, Jameson. Are there any plans to expand the amount that's kept indefinitely or is there no need to do that?

Re: Data Storage Capacity

Posted: Mon May 07, 2018 1:45 pm
by Jameson
We store the Summary Data indefinitely.

Are you talking about having more than 1000 reads in the EKM Push real-time data? At this point we do not have a plan to expand this. If someone wants 2000 minutes of real-time data (instead of the 1000 minutes that they get with the Push system by default), they can either change their meter read rates to once per 2 minutes (in this case 1000 reads would be 2000 minutes of data), or they can store all of the reads that come into the Push system using the EKM Dash software.

Re: Data Storage Capacity

Posted: Mon Mar 06, 2023 7:28 am
by jgmoroz
I am new with your systems and doing a proof of concept experiment to demonstrate, to our Water department Village Board, a system that would store data to calculate the Gallons of water per kWh. like to begin with minimum communication and data storage system consisting of:
1.EKM-Omnimeter Pulse UL v.4 with a water meter input (can't find the storage capability of storing all 42 measurements) in 15
minute increments.
2. RS485 to USB converter
3. My pc that would not reside at the site but I would bring it weekly to collect the data in a csv format.
4. Make the calculations offline using Excel and sharing the results with the board.

Please advise.

Thanks
Jim

Re: Data Storage Capacity

Posted: Mon Mar 06, 2023 11:00 pm
by Jameson
Hello Jim,

I think your proposal would work, but it would be rather a manual process and you would not be able to gather much data if your PC was not on the site 24/7. If it were me, I would take a look at the Push3 system instead. You could have a Push3 gateway on the site reading the Omnimeter Pulse v.4 (including the water meter data). Ideally the EKM Push3 gateway would be connected to the internet and would be sending data to the cloud once per minute. But if you do not have internet or do not want to send data to the cloud, you do not have to. The EKM Push3 gateway has an SD card slot. If you were to use a 16GB, 32GB, 64GB or larger SD card (or USB stick) with your Push3 you could store tons of data when your Push3 is offline. Then you can go and get the data once in a while. Heads up though you might have to be decent at manipulating the data as the data on the SD card or USB stick is stored in json format.

You would definitely have enough data over time to be able to easily calculate the kWh per gallon. Having a lot of data would help you to see if you could improve the kWh per gallon over time with efficiency upgrades to your systems.

If your Push3 is connected to the internet then you just use the cloud as your storage (the SD card is really just a backup in that case). The data is real-time, is much richer and more complete, we have software solutions to be able to view it and get into Excel, etc (its basically better all the way around if you do have internet for this). It is super easy to get the data in csv format, get it into excel, and share that csv data or data from the EKM Widget, www.encompass.io with your board members.

Re: Data Storage Capacity

Posted: Tue Mar 07, 2023 9:56 am
by jgmoroz
When the data storage is expressed as 1000 reads, does this mean that each read is 42 measurements?

I understand that this is stored in the cloud and I appreciate the very large storage capability of the ekm Push 3.

Please help me understand how many 15 minute interval sets of 42 measurements can I store in the meter itself.

I am probably being to picky, but would like to consider carefully my first minimal architecture with a meter and 485/USB converter.

Thanks
Jim

Re: Data Storage Capacity

Posted: Tue Mar 07, 2023 10:40 am
by Jameson
When the data storage is expressed as 1000 reads, does this mean that each read is 42 measurements?
When we are referring to storing the latest 1000 intervals, this is the latest 1000 real-time (once per minute in most cases) meter read intervals. This would come to 1000 minutes of storage if your meter read interval is 1 per minute. You can extend this to 5000 minutes by changing the read interval to once per 5 minutes.

Each meter interval has the 15+ (in the case of a v3 Omnimeter) or 42+ (in the case of a v4 or v5 Omnimeter) metered values included with it.

We then take these meter intervals and summarize them into 15 minute summaries. We use these summaries to calculate 15 minute, hourly, daily, weekly, and monthly summary API results.

We store your summary data in 15 minute chunks for 2 years, then we reduce this to hourly summaries that we store for 4 years, then we reduce this to daily summaries (stored in UTC time) that we store forever. So you will be able to get daily, weekly, and monthly summaries (stored in UTC time) forever.

Here is more information about our Summary API data: https://documents.ekmmetering.com/api-d ... ummary-api

Re: Data Storage Capacity

Posted: Wed Mar 08, 2023 11:06 am
by jgmoroz
Thank you for your explanation and links to help me understand the storage in general.

Summarizing what I can store in the meter:
1,000 intervals of 42 measurements averaged over a 15 minute time period synchronized to UTC.
The chronological storage capacity is 1,000 / 96 or 10.5 days. So I would need to read and store the .cvs data roughly every 10 days manually to a thumb drive in my computer. Correct?

I am very old school, and understand that APIs have evolved into standard code is an interface between two systems. My understanding of the software is that the API would function to communicate the readings according to specific parameters that I would specify and a thing called a Widget would store them in a thumb drive?

Perhaps the best way to learn this is to purchase the minimal system and learn from experience.

Comments please
Thanks
Jim

Re: Data Storage Capacity

Posted: Fri Mar 10, 2023 1:37 pm
by Jameson
Hi Jim,

Thank you for bringing your post to my attention.

It sounds like we are not aligning on our understanding. Here is what we store:

In the meter: Not much at all. Just the latest values kWh, volts on each phase, etc. 6 months of once per month kWh reads (only 6 data points)

In the real-time Push data (typically once per minute): 1000 reads (can be thought of as 1000 minutes)

In the historical Summary Push data: 15 minute > 2 years, hourly > 4 years, [daily, weekly, monthly} > forever.

https://help.ekmmetering.com/support/so ... a-logging-
https://help.ekmmetering.com/support/so ... unt-portal