Missing / Odd data in HA

[jamesonuk]

The data I am seeing on HA energy dashboard doesn't match what I am seeing in the Bright App.

I have setup the electricity consumption today sensor in the dashboard but as you can see some hours simply have no data and others don't match the figures I see in Bright

Eg 08:00-09:00 is showing as 3.5KWh but even allowing for the fact data is 30/60 minutes old when it turns up at HA I cannot see any way to get the 3.5 kWh figure

There there is no data until 13:00-14:00 in HA even though data is showing in Bright app.

Have I set this up wrong?

[HandyHat]

This should be fixed as of v1.0.0 and up. Please say if this issue continues to occur!

[ronnyTodgers]

Ahh sorry found this thread after posting my own - I'm using version 1.29.1 and getting something similar
#300

[changeofaspect]

I am getting the same issue and using v1.0.2. Firstly, a big thank you for putting in the time and effort to create and update this integration. I offer the following comments based on an observation and not a criticism.

I was puzzled by the difference in figures shown on the Bright App and the data displayed in HA. I buy electric from OVO and the data showing on their website matches exactly with the Bright App so the Bright App data is believable. OVO update their website once per day, the most recent data being yesterday. I will not see today's half hourly figures until tomorrow. The Bright App for the majority of time updates every 30 minutes throughout the day, but not always. I have observed the behaviour of the Bright App over several days and discovered that it quite often lags behind, sometimes by several hours and will then play catch up. It is worse in the mornings from 05:00 through to 10:00 where it is typically 3 hours behind.

The times shown on the Bright App are the beginning of the 30 minute period (OVO do the same) therefore when the Bright App says 16:00, the consumption is for the period from 16:00 to 16:30. The App may update at 16:33 but it might be at 17:03. So it could be as little as 3 minutes after the end of the period or it might be 33 minutes. That varies throughout the day, but in the mornings the lag increases. Worth noting that looking at the latest Hourly figure on the Bright App, it may have changed when you next look! It seems that the first 30minutes is shown as the last Hour reading and when the second 30minute period has finished, the Hourly display will be updated to show the whole hour.

So what does the integration make of this? By setting the stats graph to 5 minute granularity (changes), I see a series of spikes representing each update. In general, when the Bright App has caught up and there is no lag, the integration updates at xx:05 and xx:35. But when the Bright App is lagging, the integration will update at xx:00 or xx:30 and then follow that with another update 5 minutes later. It is possible to have four updates in the hour and the hourly bar chart displays the sum of the updates received during that hour. This explains why my charts and figures look different to the Bright App.

A further issue occurs just after midnight. The integration, which updated at 23:35, resets to zero at midnight. However the 23:35 update, at best was for the period 23:00 to 23:30 but frequently is for the period 22:30 to 23:00. After midnight, further updates arrive every half hour but the first, second, possibly third of these still relate to the previous day. The DCC or Hildebrand then seem to do their own reset which is seen on the HA graphs as drop (reduction) in the accumulating total. I have seen this can occur at 00:30, 01:00 or 01:30. the effect is that the Daily Total is lower than it should be by the sum of the consumption updates received after midnight minus the first reading of the new day (ie, for the period 00:00 to 00:30). Throughout the day, the total so far will always be lower than it should by this amount.

The combination of these two issues means that unfortunately the figures shown in HA do not align with the Bright App.

[ronnyTodgers]

I think you have nailed the issue here - The bright app and loop both delay the data so you only see several hours old time slots and it looks as though this might be because when you grab data it's a mix of the most recent and occasionally data belonging to previous timeslots. The apps enforce a delay to try and ensure the the slots are fully populated before they are shown. As James notes Bright does quite a bit of revision work as it goes along.

What I'm seeing in HA is that while the total consumption is correct, big chunks of usage are showing up several hours after where they should be. Does the API provide the timeslot that usage relates to? I suppose it must do to support the app but it looks like this integration assumes that when you grab data at 13.33 it refers to the 13.00 to 13.30 slot and this seems to be frequently incorrect.

In my first full day we had heavy usage from 7 to 8, but it showed up in the 10-11 bar on HA.

[ronnyTodgers]

I would be very surprised if the Glow display with local MQTT isn't actually instant and reliably so.

I'm wondering about the USB CAD thing because I don't really want the display, but then that doesn't do local MQTT so you still get when is likely a big uncontrollable source of noise of the meter deciding to upload the data - then it has to go through multiple server before it reaches the one that will send the MQTT.

If you want accurate instant then I reckon the display and local MQTT is the route to use, or get one of those flash counter things and tape it to the meter for the daily info!

[jamesonuk]

If you want accurate instant then I reckon the display and local MQTT is the route to use, or get one of those flash counter things and tape it to the meter for the daily info!

I bit the bullet and ordered one. All setup and logging now (was a bit of a faff to add to HA as you need to add all the sensors manually).
It is much more accurate but isn't quite 100%. It looks like this is the same issue with latency and HA expecting a current value but like this morning I see this

which other than a bit of rounding matches what is shown in the Bright app except around 4am.
Not sure why (perhaps the IHD lost connection to the meter?) I am missing a little consumption 03:00 - 04:00 all consumption 04:00 to 05:00 and then all of that data arrives 05:00 - 06:00 giving me an inflated figure there.

It seems to be spot on at day level (subject to a bit of rounding) which is much better but it is expensive just to get this data 😲

[ronnyTodgers]

I think the lost connection to the meter sounds most plausible - if it happens every night then that's a bit of a pain! does the same appear on the display of the IHD itself? And yes the day level accurately really should be achievable via DCC but not at present by the looks of it

[changeofaspect]

I having been discussing the CAD with Jane Wilson at Glowmarkt as I wanted more info before buying. If you are using local MQTT (ie. not Cloud MQTT), then you should get instantaneous power (updates every 10 seconds) and energy consumption for each minute and half hour period calculated locally by the CAD. This half hourly data presumably is what is used by HA. If you have Solar PV and Generation exceeds consumption, then the data will be negative for instantaneous export, per minute and per half hour. Some meters only give cumulative export. Then overnight, Bright will download from the DCC the actual half hourly figures for the previous day and overwrite the figures held in the CAD. I don't believe HA is updated to show the revised figures, not sure how it would do that as it works in realtime. So on the current day, the CAD and HA will align but when looking back at the previous day, the CAD and the Bright App will align but might not align with the figures previously stored in HA. Also, when the overnight correction occurs, any negative figures (Export to grid) will become zero.

So, if you are using cloud MQTT instead of local then the gap would match the Bright App. But if you are using Local MQTT, then the gap can only be produced locally and not by Bright/DCC so that looks like the CAD has lost connection with the meter. It would be interesting to see if it does it again.

[jamesonuk]

It would be interesting to see if it does it again.

I am guessing it probably has (there is very little change of there being no consumption in our house over an hour)
Not too fussed as the daily totals seem to match up and the hourly stats are nearly spot on.
I will do some digging if I ever get time just to figure out what is happening here and whether it is the fact that messages are just not coming through or whether they are coming through showing no energy is being used

[jamesonuk]

This seemed to get a lot better when I installed 1.0 but the data is looking to be quite a bit out again.
If I have read the above correctly then this is partly down to latency but when looking at day level the figures should aggregate up OK?

Day	Bright App	HA	Diff
16th	30.88	30.24	-0.64
17th	40.94	40.39	-0.55
18th	39.47	39.21	-0.26
19th	36.25	33.85	-2.4
20th	39.18	30.46	-8.72
21st	37.95	31.51	-6.44
22nd	41.46	39.69	-1.77

So overall that is 8% difference over the week (one day is over 12% out). That variance really limits it's usefulness.

This is obviously an issue in the latency of the DCC data combined with how HA handles power measurements which is a great shame. Obviously the bright app works to fix historic data but I guess it would need a (big) change to HA in order for it to behave the same way.

Looks like I need to fork out on a CAD if I want HA to be accurate.

Thanks for the effort in developing this though but I am not sure this is something that can be resolved?

[ronnyTodgers]

I think that might be a slightly different issue related to the zeroing of the days measurement which is done in this plugin at 1am or 1.30 so - damn I had assumed the day level would be right. I think the way to fix this without having to update old data in HA is to buffer the hourly DCC data (probably for at least 3 hours) before reporting it back to HA - that said it sounds like even telling HA the data you are reporting is for a previous timepoint is far from easy

[ronnyTodgers]

For what it's worth I think you need the local MQTT from the CAD, not the USB displayless version if you want it to be instant and accurate.

[changeofaspect]

I am seeing around 10% discrepancy each day as well. It has definitely been the worst week since getting my Bright App with Sat 21st Jan lagging behind by 7 hours in the morning and still 4 hours adrift in the early afternoon. I have been picking up on comments on the Glowmarkt forum and it seems that the Local MQTT CAD displays real time power updated every 10 seconds from the Smart Meter. But it also 'calculates' the half hourly usage as well. Then overnight, the real figures are downloaded from Bright and over-write the calculated half-hourly figures in the CAD. It sounds like this has the potential to give HA some reasonably close figures in real time, although the following day, there may be some discrepancy with what shows on the App/CAD compared the figures recorded in HA. It looks like I will be purchasing a CAD as well.

[trevorwarwick]

I think I'm also seeing more variance between what's shown in the Bright App and HA since I installed the recent update (1.03). I've only had it for a day so it may be coincidence, but will keep an eye on it.