By default, the history in Home Assistant shows the status changes of the last 10 days and supplements older data for certain entities from the Long Time Statistic (LTS). In this article you will find out which entities these are and how the data is stored in the Home Assistant.

Home Assistant has its own database tables for saving historical values, which means that the energy dashboard, for example, can also display data older than 10 days. But not only the energy dashboard can use the long-term statistics data.

Entities with long-term statistics (LTS)

The prerequisite for recording long-term statistics data (LTS) is the attribute "state_class" with one of the properties: "measurement", "total" or "total_increasing".

A look at the developer tools / states tells us whether an entity also records LTS data in addition to status changes.

If the attributes are missing, they can be added via the customize.yaml file. As an example, I have added the properties for a Riemann sum integral sensor for long-term statistics by adding the sensor to the customize.yaml file after creating it and adding the properties: 

In customize.yaml, the sensor can be addressed with its ID, here "sensor.heating_energy" and supplemented with the properties "state_class" and "device_class":

sensor.heating_energy:
  state_class : total_increasing
  device_class: energy
  last_reset: "2000-01-01T00:00:00+00:00"
  unit_of_measurement: kWh

If the attributes are available, the statistics tables are filled.

Historical data in the history

The easiest way to display long-term statistical data is in the history:

The current data of the last 10 days is displayed slightly weaker in the history, the data from the history tables slightly stronger.

Included in Home Assistant: Statistics chart

Without additional integration, historical data can be visualized via the Lovelance card "Statistics chart":

Here is an example for the PV production of my balcony power plant: daily and monthly:

Unlike the energy dashboard, the time period or the days to be displayed cannot currently be customized in the frontend: The values are fixed in the map. Until there are further improvements in Home Assistant, certain custom repositories, such as the Plotly Graph Card, can be used. Plotly Graph enables interactive scrolling, zooming in and out of charts. Initially, I only used Plotly Graph for current sensor data, but Plotly can also handle historical data:

Plotly Graph

The "period" option in Plotly Graph adjusts the data points to the zoom factor and uses more or less detail from the statistics tables. The Plotly Graph Card can be installed via the Home Assistant Community Store (HACS).

Ploty Graph must be adapted in the YAML editor for the setting. Here is an example for the visualization of a temperature sensor:

type: custom:plotly-graph
title: Temp
entities:
  - entity: sensor.aussen_temperature
    yaxis: y2
    showlegend: true
    show_value: true
    fill: tozeroy
    name: ww🌡
    period: auto
    fillcolor: rgba(255, 255, 0, 0.1)
    line:
      color: orange
      smoothing: 0.5
      shape: spline
      width: 1
hours_to_show: 800
refresh_interval: auto
layout:
  xaxis:
    rangeselector:
      buttons:
        - count: 3
          step: hour
        - count: 12
          step: hour
        - count: 30
          step: day
        - count: 365
          step: day
  yaxis:
    zeroline: true
    fixedrange: true
  legend:
    'y': 1.05
  margin:
    l: 35
    r: 43

Without the "period" option, Plotly uses the current status data. Originally, I also presented the History Explorer Card here. Since Home-Assistant can now access the statistics data via its history and Plotly Graph can be used to create much more flexible cards, I no longer use the card and have removed it here too.

What is behind the LTS function?

Internally, Home Assistant uses various database tables to store data. The "states" table stores all status changes, which not only produces a large amount of data over a longer period of time, but would also affect the performance of Home Assistant. For this reason, Home Assistant deletes data that is older than 10 days. Responsible for cleaning up the status data is the setting: purge_keep_days, which cleans up and deletes data older than 10 days, see: www.home-assistant.io/integrations/recorder/. However, changing this setting is not necessarily a good idea, as a longer period produces more data and Home Assistant is slowed down if the value is too high.

In order to still be able to access data older than 10 days, Home Assistant determines the average, minimum, and maximum values for the sensors and writes them to its own tables:

The average, minimum and maximum values are stored in the "statistics_short_term" table every 5 minutes. The statistics_short_term table also discards data that is older than 10 days, just like the original "states" table. This is different for the "statistics" table, which stores the statistics data average, minimum, and maximum values for each hour and does not delete them. A look at the SQLite database shows us the data:

Home Assistant uses Unix timestamps as the format for the date and time:

To convert the Unix timestamp to a readable date, I have published a small JavaScript tool on this page: Convert Date: Unix Timestamp

Home-Assistant's internal database structure can also be surmised from the following example: If I try to set 5 minutes as the time period in the statistics card, it can only display up to 10 days. If I were to select an 11th day, the card would no longer display anything:

However, if I switch to "hour", the map obviously uses the "statistics" table and also displays data older than 10 days:

Conclusion

The behavior of Home Assistant when saving sensor data is more or less a compromise between the details saved and the age of the data: All status changes are available for the last 10 days; beyond that, the statistics table can only be accessed at hourly intervals. In terms of functionality, Home Assistant has made some progress here in the past, but Home Assistant cannot keep up with a timeseries database such as InfluxDB when it comes to saving historical data or with a solution such as Grafana when it comes to visualization. If you can live with the fact that data older than 10 days is only available on an hourly basis, Plotly Graph is still a worthy alternative.