Data Storage Capacity

Forum is open to all other questions, answers, and discussions that have no other topic.
romemeek
Posts: 5
Joined: Tue Jul 25, 2017 7:50 am

Data Storage Capacity

Post by romemeek »

Hello,

I have two questions. How long is the data from my push device stored on the cloud server?
Is there a limit to the amount of data that can be stored?

Thanks!
Jameson
Posts: 859
Joined: Fri Nov 04, 2011 7:42 pm
Location: Santa Cruz, CA
Contact:

Re: Data Storage Capacity

Post by Jameson »

We store the latest 1000 real time reads. By default meters are read once per minute, so this is the latest 1000 minutes of data.

Here is more information about our Real-Time API: http://documents.ekmmetering.com/api-do ... altime-api

===

We then take these real-time reads and then summarize them down into 15 minute chunks that are available via our Summary API. We store these 15 minute chunks forever.

Here is more information about our Summary API: http://documents.ekmmetering.com/api-do ... ummary-api

===

Let me know if this does not answer your question.
Jameson
EKM METERING
http://www.ekmmetering.com
831.425.7371
Caruana
Posts: 1
Joined: Sat Apr 28, 2018 2:42 am

Re: Data Storage Capacity

Post by Caruana »

That's good to know, Jameson. Are there any plans to expand the amount that's kept indefinitely or is there no need to do that?
Jameson
Posts: 859
Joined: Fri Nov 04, 2011 7:42 pm
Location: Santa Cruz, CA
Contact:

Re: Data Storage Capacity

Post by Jameson »

We store the Summary Data indefinitely.

Are you talking about having more than 1000 reads in the EKM Push real-time data? At this point we do not have a plan to expand this. If someone wants 2000 minutes of real-time data (instead of the 1000 minutes that they get with the Push system by default), they can either change their meter read rates to once per 2 minutes (in this case 1000 reads would be 2000 minutes of data), or they can store all of the reads that come into the Push system using the EKM Dash software.
Jameson
EKM METERING
http://www.ekmmetering.com
831.425.7371
jgmoroz
Posts: 5
Joined: Mon Mar 06, 2023 7:09 am

Re: Data Storage Capacity

Post by jgmoroz »

I am new with your systems and doing a proof of concept experiment to demonstrate, to our Water department Village Board, a system that would store data to calculate the Gallons of water per kWh. like to begin with minimum communication and data storage system consisting of:
1.EKM-Omnimeter Pulse UL v.4 with a water meter input (can't find the storage capability of storing all 42 measurements) in 15
minute increments.
2. RS485 to USB converter
3. My pc that would not reside at the site but I would bring it weekly to collect the data in a csv format.
4. Make the calculations offline using Excel and sharing the results with the board.

Please advise.

Thanks
Jim
Jameson
Posts: 859
Joined: Fri Nov 04, 2011 7:42 pm
Location: Santa Cruz, CA
Contact:

Re: Data Storage Capacity

Post by Jameson »

Hello Jim,

I think your proposal would work, but it would be rather a manual process and you would not be able to gather much data if your PC was not on the site 24/7. If it were me, I would take a look at the Push3 system instead. You could have a Push3 gateway on the site reading the Omnimeter Pulse v.4 (including the water meter data). Ideally the EKM Push3 gateway would be connected to the internet and would be sending data to the cloud once per minute. But if you do not have internet or do not want to send data to the cloud, you do not have to. The EKM Push3 gateway has an SD card slot. If you were to use a 16GB, 32GB, 64GB or larger SD card (or USB stick) with your Push3 you could store tons of data when your Push3 is offline. Then you can go and get the data once in a while. Heads up though you might have to be decent at manipulating the data as the data on the SD card or USB stick is stored in json format.

You would definitely have enough data over time to be able to easily calculate the kWh per gallon. Having a lot of data would help you to see if you could improve the kWh per gallon over time with efficiency upgrades to your systems.

If your Push3 is connected to the internet then you just use the cloud as your storage (the SD card is really just a backup in that case). The data is real-time, is much richer and more complete, we have software solutions to be able to view it and get into Excel, etc (its basically better all the way around if you do have internet for this). It is super easy to get the data in csv format, get it into excel, and share that csv data or data from the EKM Widget, www.encompass.io with your board members.
Jameson
EKM METERING
http://www.ekmmetering.com
831.425.7371
jgmoroz
Posts: 5
Joined: Mon Mar 06, 2023 7:09 am

Re: Data Storage Capacity

Post by jgmoroz »

When the data storage is expressed as 1000 reads, does this mean that each read is 42 measurements?

I understand that this is stored in the cloud and I appreciate the very large storage capability of the ekm Push 3.

Please help me understand how many 15 minute interval sets of 42 measurements can I store in the meter itself.

I am probably being to picky, but would like to consider carefully my first minimal architecture with a meter and 485/USB converter.

Thanks
Jim
Jameson
Posts: 859
Joined: Fri Nov 04, 2011 7:42 pm
Location: Santa Cruz, CA
Contact:

Re: Data Storage Capacity

Post by Jameson »

When the data storage is expressed as 1000 reads, does this mean that each read is 42 measurements?
When we are referring to storing the latest 1000 intervals, this is the latest 1000 real-time (once per minute in most cases) meter read intervals. This would come to 1000 minutes of storage if your meter read interval is 1 per minute. You can extend this to 5000 minutes by changing the read interval to once per 5 minutes.

Each meter interval has the 15+ (in the case of a v3 Omnimeter) or 42+ (in the case of a v4 or v5 Omnimeter) metered values included with it.

We then take these meter intervals and summarize them into 15 minute summaries. We use these summaries to calculate 15 minute, hourly, daily, weekly, and monthly summary API results.

We store your summary data in 15 minute chunks for 2 years, then we reduce this to hourly summaries that we store for 4 years, then we reduce this to daily summaries (stored in UTC time) that we store forever. So you will be able to get daily, weekly, and monthly summaries (stored in UTC time) forever.

Here is more information about our Summary API data: https://documents.ekmmetering.com/api-d ... ummary-api
Jameson
EKM METERING
http://www.ekmmetering.com
831.425.7371
jgmoroz
Posts: 5
Joined: Mon Mar 06, 2023 7:09 am

Re: Data Storage Capacity

Post by jgmoroz »

Thank you for your explanation and links to help me understand the storage in general.

Summarizing what I can store in the meter:
1,000 intervals of 42 measurements averaged over a 15 minute time period synchronized to UTC.
The chronological storage capacity is 1,000 / 96 or 10.5 days. So I would need to read and store the .cvs data roughly every 10 days manually to a thumb drive in my computer. Correct?

I am very old school, and understand that APIs have evolved into standard code is an interface between two systems. My understanding of the software is that the API would function to communicate the readings according to specific parameters that I would specify and a thing called a Widget would store them in a thumb drive?

Perhaps the best way to learn this is to purchase the minimal system and learn from experience.

Comments please
Thanks
Jim
Jameson
Posts: 859
Joined: Fri Nov 04, 2011 7:42 pm
Location: Santa Cruz, CA
Contact:

Re: Data Storage Capacity

Post by Jameson »

Hi Jim,

Thank you for bringing your post to my attention.

It sounds like we are not aligning on our understanding. Here is what we store:

In the meter: Not much at all. Just the latest values kWh, volts on each phase, etc. 6 months of once per month kWh reads (only 6 data points)

In the real-time Push data (typically once per minute): 1000 reads (can be thought of as 1000 minutes)

In the historical Summary Push data: 15 minute > 2 years, hourly > 4 years, [daily, weekly, monthly} > forever.

https://help.ekmmetering.com/support/so ... a-logging-
https://help.ekmmetering.com/support/so ... unt-portal
Jameson
EKM METERING
http://www.ekmmetering.com
831.425.7371
Post Reply