2

I've seen that many ML datasets have competitions (like imageNet). I've been looking for some kind of competition or state-of-the-art LSTM solutions for The Airline Passengers dataset but all I can find are really bad solutions (like next month will be like the current month) with some handwavy arguments that one could obtain a good solution if one tweaked the parameters. The best ensemble LSTM solution I've found is Taming the Chaos in Neural Network Time Series Predictions, but I'm looking for a single LSTM solution.

So my question is, what are the known state-of-the-art LSTM solutions (maybe a timeline of them if there is any) and if there are none, how come? Looking at the data it seems like a simple problem to solve. Are there any other well known time series forecasting datasets that one benchmarks LSTM solutions against?

Natanael
  • 23
  • 3
  • Are you looking for an LSTM fine-tuning guide? Something like: https://wandb.ai/metromert/Rabbidity/reports/LSTM-Hyperparameter-Finetuning--VmlldzozMDQyMzU4 or https://machinelearningmastery.com/tune-lstm-hyperparameters-keras-time-series-forecasting/ – Nicolas Martin Mar 31 '23 at 13:21
  • I'm more looking for an already optimized solution since then I can see that it can be done and how such a solution would look like. If the solution was for me to fine-tune it myself I would have to doubt my fine-tuning abilities. I mean both MINST and Titanic have such solutions (although not LSTM since it's another type of problem), why does not the Airline passengers have have such solutions? – Natanael Apr 01 '23 at 15:22
  • MNIST and Titanic are just very classic datasets used as school cases for decades. – Nicolas Martin Apr 01 '23 at 20:29
  • You mean that this would not be true for the Airline passengers dataset in the realm of timeseries forecasting? – Natanael Apr 02 '23 at 10:47
  • I think so, but I can be wrong. – Nicolas Martin Apr 03 '23 at 12:27

4 Answers4

0

Not quite sure about the specific task of Airline passengers, but in my experience, while Transformers have overshadowed RNNs recently, this statement is mostly true within the academic research community, and not so much in the industry (Setting aside NLP). Regardless, None of the Vanilla-RNNs are now considered SOTA, but variants that somehow utilize attention, or some other tricks are indeed SOTA comparable. Some notable examples are

  • Temporal Pattern Attention for Multivariate Time Series Forecasting (paper)
  • DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Network (paper)
  • Multivariate time series forecasting via attention-based encoder-decoder framework (paper)
  • one of the newer variants of ConvLSTM

I believe that one of these will perform nice out-of-the-box.

As per your other question, head over to the paper on Informer, they provide a comparison using many types of time series data - which might help.

Hadar
  • 137
  • 7
0

Your problem could be reframed more generally as time series forecasting. In time series forecasting, there are many different commonly used datasets and many different models. Papers With Code has benchmarks and implementations over time for time series forecasting.

Brian Spiering
  • 20,142
  • 2
  • 25
  • 102
0

It is difficult to provide a definitive answer as research in this area is ongoing, and the performance of models can be highly dependent on the specific task, data, and evaluation metrics. However, some recent studies have proposed effective LSTM-based approaches for time series forecasting tasks.

In 2017, the paper "DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks" introduced a novel LSTM-based architecture called DeepAR, which combines a recurrent neural network (RNN) with an autoregressive model to improve the accuracy and uncertainty estimates in time series forecasting. This approach achieved state-of-the-art results on a range of time series datasets, and you can find some implemetnations online for the Airline dataset.

Another study proposed an LSTM-based approach called "Multi-Horizon Time Series Forecasting with Hierarchical Attention Recurrent Neural Networks" (HARNet). HARNet uses a hierarchical attention mechanism to selectively attend to relevant temporal features and has achieved good results on several benchmark time series datasets, including the M4 competition dataset, which is a large-scale forecasting competition that includes various types of time series data.

Regarding why the state-of-the-art solutions for the Airline Passengers dataset may not be as widely reported as those for other datasets like ImageNet, is the fact that time series forecasting is a more specialized compared to image recognition, which is a more general and widely studied area in machine learning. Additionally, the performance of a model on a particular dataset can vary depending on factors such as the size and complexity of the dataset, the quality of the data, and the evaluation metrics used, among others. Some models just do better than others, depending on the problem.

In terms of other well-known time series forecasting datasets that one can benchmark LSTM solutions against, there are several popular benchmarks, including the M4 competition dataset mentioned earlier, as well as the M3 and M5 competitions. These datasets include a diverse range of time series data, including economic and financial indicators, energy demand, and retail sales, among others.

Nemo_the_scientist
  • 469
  • 11
0

The Airline Passengers dataset is a well-known time series forecasting dataset. Regarding the Airline Passengers dataset specifically, there may not be as much attention given to it in terms of competition or research, as it is a relatively small and simple dataset, with only 144 data points and the dataset exhibits clear seasonal patterns, which can be easily captured using simpler modelling techniques. and a well-known dataset that has been used for decades. So it's possible that many researchers have simply not found it to be a particularly interesting or challenging problem to solve.

One reason there may not be a single state-of-the-art LSTM solution for this dataset is that time series forecasting is a complex task and there is no one-size-fits-all solution. Different models and techniques may work better for different datasets, depending on factors such as data characteristics, seasonality, trends, etc. Additionally, hyperparameter tuning can have a significant impact on the performance of a model, and there is no universally optimal set of hyperparameters that will work for all datasets.

However, there have been many research papers and competitions focused on this problem, so I can provide some examples.

In addition to the Airline Passengers dataset, there are several other well-known time series forecasting datasets that are often used to benchmark the performance of different models, including:

  1. M4 Competition dataset: A large collection of time series data from various domains with different characteristics used in the M4 forecasting competition. The competition has spurred much research into forecasting methods, including LSTM-based approaches.

The M4 dataset is a collection of 100,000 time series used for the fourth edition of the Makridakis forecasting Competition. The M4 dataset consists of time series of yearly, quarterly, monthly and other (weekly, daily and hourly) data, which are divided into training and test sets. The minimum numbers of observations in the training test are 13 for yearly, 16 for quarterly, 42 for monthly, 80 for weekly, 93 for daily and 700 for hourly series. The participants were asked to produce the following numbers of forecasts beyond the available data that they had been given: six for yearly, eight for quarterly, 18 for monthly series, 13 for weekly series and 14 and 48 forecasts respectively for the daily and hourly ones.

  1. Energy forecasting dataset: A dataset containing hourly electricity demand and weather data for a large metropolitan area, used for predicting electricity demand.

This dataset contains 4 years of electrical consumption, generation, pricing, and weather data for Spain. Consumption and generation data was retrieved from ENTSOE a public portal for Transmission Service Operator (TSO) data. Settlement prices were obtained from the Spanish TSO Red Electric España. Weather data was purchased as part of a personal project from the Open Weather API for the 5 largest cities in Spain and made public here.

  1. Traffic forecasting dataset: A dataset containing hourly traffic volume data from a major urban freeway, used for predicting traffic volume.

This dataset contains the traffic data in San Bernardino from July to August in 2016, with 170 detectors on 8 roads with a time interval of 5 minutes. This dataset is popular as a benchmark traffic forecasting dataset.

Traffic Prediction Dataset

  1. Retail forecasting dataset: A dataset containing daily sales data from a retail chain, used for predicting sales.

Some state-of-the-art LSTM-based approaches for time series forecasting include:

  • Transformers for Time Series Forecasting: This paper introduced a transformer-based model for time series forecasting called Temporal Fusion Transformers, which outperforms traditional LSTMs on several benchmark datasets.

  • DeepAR: This model was introduced by Amazon in 2017 and is based on an LSTM network. It uses a probabilistic forecasting approach and has been shown to outperform traditional statistical methods and other deep learning models on several benchmark datasets.

  • GRU-D: This model was introduced in 2017 and is based on a gated recurrent unit (GRU) network. It includes a denoising module to remove noise from the input data, and has been shown to outperform traditional LSTMs and other deep learning models on several benchmark datasets.

  • LSTM-FCN: This model was introduced in 2018 and combines a LSTM network with a fully convolutional neural network (FCN).

Recent papers that have used LSTMs for time series forecasting include:

  • An Efficient Deep Learning Framework for Stock Price Forecasting Based on Stacked Bidirectional LSTM with Attention Mechanism by Huang et al. (2022)
  • Gated Autoregressive Recurrent Neural Network for Time Series Forecasting by Yeo et al. (2021)
  • A Hybrid Approach for Accurate Load Forecasting Using LSTM Neural Networks and Particle Swarm Optimization by Sharma et al. (2021)
Pluviophile
  • 3,520
  • 11
  • 29
  • 49