By default, the history in Home Assistant shows the status changes of the last 10 days and supplements older data for certain entities from the Long Time Statistic (LTS). In this article you will find out which entities these are and how the data is stored in the Home Assistant.

Home Assistant has its own database tables for saving historical values, which means that the energy dashboard, for example, can also display data older than 10 days. But not only the energy dashboard can use the long-term statistics data.

Entities with long-term statistics (LTS)

The prerequisite for recording long-term statistics data (LTS) is the attribute "state_class" with one of the properties: "measurement", "total" or "total_increasing".

A look at the developer tools / states tells us whether an entity also records LTS data in addition to status changes.

If the attributes are missing, they can be added via the customize.yaml file. As an example, I have added the properties for a Riemann sum integral sensor for long-term statistics by adding the sensor to the customize.yaml file after creating it and adding the properties: 

In customize.yaml, the sensor can be addressed with its ID, here "sensor.heating_energy" and supplemented with the properties "state_class" and "device_class":

sensor.heating_energy:
  state_class : total_increasing
  device_class: energy
  last_reset: "2000-01-01T00:00:00+00:00"
  unit_of_measurement: kWh

To be able to display the long-term statistics data with on-board tools, i.e. without additional integration, Home Assistant offers the Lovelance card "Statistics diagram":

Here is an example of the PV production of my balcony power plant: daily and monthly:

Unlike the energy dashboard, the time period or the days to be displayed cannot currently be customized in the frontend: The values are fixed in the map. With the December 2023 update, the history can now also access the LTS data, so I assume that further improvements will certainly follow in this direction. Until then, certain custom repositories, such as the history-explorer-card, can be used.

Explore statistics data interactively in the dashboard: History-Explorer-Card

See: github.com/alexarch21/history-explorer-card

What is behind the LTS function?

Internally, Home Assistant uses various database tables to store data. The "states" table stores all status changes, which not only produces a large amount of data over a longer period of time, but would also affect the performance of Home Assistant. For this reason, Home Assistant deletes data that is older than 10 days. Responsible for cleaning up the status data is the setting: purge_keep_days, which cleans up and deletes data older than 10 days, see: www.home-assistant.io/integrations/recorder/. However, changing this setting is not necessarily a good idea, as a longer period produces more data and Home Assistant is slowed down if the value is too high.

In order to still be able to access data older than 10 days, Home Assistant determines the average, minimum, and maximum values for the sensors and writes them to its own tables:

The average, minimum and maximum values are stored in the "statistics_short_term" table every 5 minutes. The statistics_short_term table also discards data that is older than 10 days, just like the original "states" table. This is different for the "statistics" table, which stores the statistics data average, minimum, and maximum values for each hour and does not delete them. A look at the SQLite database shows us the data:

Home Assistant uses Unix timestamps as the format for the date and time:

To convert the Unix timestamp to a readable date, I have published a small JavaScript tool on this page: Convert Date: Unix Timestamp

Home-Assistant's internal database structure can also be surmised from the following example: If I try to set 5 minutes as the time period in the statistics card, it can only display up to 10 days. If I were to select an 11th day, the card would no longer display anything:

However, if I switch to "hour", the map obviously uses the "statistics" table and also displays data older than 10 days:

Conclusion

The behavior of Home Assistant when saving sensor data is more or less a compromise between the details saved and the age of the data: All status changes are available for the last 10 days; beyond that, the statistics table can only be accessed at hourly intervals. In terms of functionality, Home Assistant has made some progress here in the past, but Home Assistant cannot keep up with a timeseries database such as InfluxDB when it comes to saving historical data or with a solution such as Grafana when it comes to visualization. If you can live with the fact that data older than 10 days is only available on an hourly basis, Plotly Graph is still a worthy alternative.