Deep Learning Project to Predict Stock Prices

Question

So I have a background in computer programming and a little in machine learning in general. What I would like to do is create a fun project in A.I. with deep learning.

I have a dataset that has a whole bunch of stock prices at a certain date, with a bunch of features for each entry to go with it. I also have some "experts" who made predictions on whether the stock will go up or down. As my dataset grows I can evolve the game to make selections from multiple stocks...etc

Essentially what I would love to do is create an A.I. app that will be fed the same data that the "experts" had and see if I can create something more accurate and beat them at it. Is this a viable approach?

Answer 1

Essentially what I would love to do is create an A.I. app that will be fed the same data that the "experts" had and see if I can create something more accurate and beat them at it. Is this a viable approach?

Sure, you can use one or more supervised learning techniques to train a model here. You have features, a target variable and ground truth for that variable.

In addition to applying ML you have learned, all you need to do to test your application fairly is reserve some of the data you have with expert predictions for comparison as test data (i.e. do not train using it).

I would caveat that with some additional thoughts:

• You haven't really outlined an "approach" here, other than mentioning use of ML.

• Be careful not to leak future data back into the predictive model when building a test version.

• Predicting stock and markets is hard, because they react to their own predictability and many professional organisations trade on the slightest advantage they can calculate, with experienced and highly competent staff both gathering and analysing data.

---

Not directly part of the answer, but to anyone just starting out and discovering machine learning, and finding this Q&A:

Please don't imagine rich rewards from predicting markets using stats at home, it doesn't happen. If you think that this is a route to "beating the market" be aware that you are far from the first to think of doing this, and such a plan can be summarised like this:

1. Market Data + ML
2. ???
3. Profit

You can fill in the ??? by learning loads about financial markets - i.e. essentially by becoming one of the experts. ML is not a short-cut, but it might be a useful tool if you are, or plan to be, a market analyst.

Answer 2

typically I would add this as a comment but, since my score threshold < 50, I am unable to do so - hence the "Answer" response

If you're interested in running ML algorithms against historic and future index prices, you might be interested in Quantopian - Kaggle for Finance Quants.

At Quantopian, you can upload, test, and compare your results with other ML Finance Quants. Additionally, you'll learn about certain financial metrics/ratios that are native to the financial sector.

Answer 3

You can take a look at this.

https://towardsdatascience.com/simple-stock-price-prediction-with-ml-in-python-learners-guide-to-ml-76896910e2ba

This is worth a look too.

https://www.analyticsvidhya.com/blog/2018/10/predicting-stock-price-machine-learningnd-deep-learning-techniques-python/

Finally, consider this technique, directly below. This is portfolio optimization, not stock price prediction. After working on Wall Street for the past 15 years, I've found stock price prediction to be quite useless. Portfolio optimization, on the other hand, can be lucrative. I've, personally, made millions and millions in profits, using simple techniques like the one described below.

https://www.pythonforfinance.net/2017/01/21/investment-portfolio-optimisation-with-python/

Answer 4

I think you are missing some points about the 'experts' you talk about :

• They have been around a long time, they already make a lot of money out of it, and they want to stay it that way

• They already use advanced techniques for that (quantitative finance), and always try the new things like ML

• They are at the top of their academic field / employ people that are the top of their academic field

• They have all the hardware / software they want for doing so

Yes, you can use some ML techniques to predict the stock market, but those experts you seems to dismiss are way ahead, see this Quant SE questions from 2011 : https://quant.stackexchange.com/questions/111/how-can-i-go-about-applying-machine-learning-algorithms-to-stock-markets

Actually, applications of ML in finance know a recent boom because those experts start going public with their methods, see Advances in Financial Machine Learning, and its companion implementation MLfinLab)

Answer 5

Essentially what I would love to do is create an A.I. app that will be fed the same data that the "experts" had and see if I can create something more accurate and beat them at it. Is this a viable approach?

In theory yes, but not in practice. Plesae consider that a zillion of uber-skilled people already tried to do it, and NOBODY have ever been able to beat the stock market. I applied RNNs to cryptocurrency trading and the results were mediocre.

The inventor of Keras François Chollet, at the end of his RNN forecast chapter in Deep Learning with Python, Manning, p. 224, wrote:

Markets have very different statistical characteristics than natural phenomena such as weather patterns. Trying to use machine learning to beat markets, when you only have access to publicly available data, is a difficult endeavor, and you’re likely to waste your time and resources with nothing to show for it.

Always remember that when it comes to markets, past performance is not a good predictor of future returns—looking in the rear-view mirror is a bad way to drive. Machine learning, on the other hand, is applicable to datasets where the past is a good predictor of the future.

I'm sorry to be so negative about this, but I think he's 100% right.