2

I wrote a blog post where I calculated the Taylor Series of an autoregressive function. It is not strictly the Taylor Series, but some variant (I guess). I'm mostly concerned about whether the derivatives look okay. I noticed I made a mistake and fixed the issue. It seemed simple enough,but after finding an error, I started to doubt myself.

$$f(t+1) = w_{t+1} \cdot f(t) $$

$$y^{*}_{t+1} = f(t+1)-{\frac {f'(t+1)}{1!}}(-t-1+t)$$

$$y^{*}_{t+1} = w_{t+1} f(t) + \dfrac{d}{df(t)}w_{t+1}f(t) + \dfrac{d}{dw_{t+1}}w_{t+1}f(t)$$

$$y'_{t+1} = w_{t+1} f(t) + w_{t+1} + f(t)$$

The details can be found in the blog post:

EDIT 7/6/20:

The AR form:

$$y^{*}_{t+1}=c+\sum _{{I=0}}^{L}w _{t+1-i}y_{{t-i}}+\varepsilon _{t}$$

f(t) is a recursive dense layer, y is the predicted output, and w are the weights, and L are the number of lag components. For the simple case where the next value only depends on the previous value, I got the following result.

$$f(t+1) = w_{t+1} \cdot f(t) $$

$$y^{*}_{t+1} = f(t+1)-{\frac {f'(t+1)}{1!}}(-t-1+t)$$

$$y^{*}_{t+1} = w_{t+1} f(t) + \dfrac{d}{df(t)}w_{t+1}f(t)$$

$$y'_{t+1} = w_{t+1} f(t) + w_{t+1}$$

EDIT 7/7/20:

The function f(t) represents y(t) with an error term. The error term might have some random process, but I'm going to assume that the errors are independent.

$$f(t+1) = w_{t+1} \cdot y(t) + \epsilon_t$$

EDIT 7/9/20:

Changed the dimensionality of w_t+1 to w_t.

$$f(t+1) = w_{t} \cdot f(t) $$

$$y^{*}_{t+1} = f(t+1)-{\frac {f'(t+1)}{1!}}(-t-1+t)$$

$$y^{*}_{t+1} = w_{t} f(t) + \dfrac{d}{df(t)}w_{t}f(t)$$

$$y'_{t+1} = w_{t} f(t) + w_{t}$$

Osama Rizwan
  • 203
  • 1
  • 2
  • 7
targetXING
  • 121
  • 7
  • 1
    I am not sure I understand the post, nor why the chain rule (or perhaps you mean the product rule) for derivatives would result in the 3rd equation... – Nikos M. Jul 05 '20 at 16:46
  • Oh you are right, it is actually the partial derivative + the product rule. – targetXING Jul 05 '20 at 21:02
  • The purpose was to linearize the recursive formula using the Taylor Series approximation. I couldn't find any resources on how to do this so I took a stab at it. – targetXING Jul 06 '20 at 01:30
  • 1
    Where is the actual AR (auto-regressive) or ARMA (auto-regressive-moving average) model? Where are these equations. it seems you start directly from trying to find the adaptive rule for the weights. For example what is $f(t)$? I presume $y_t$ is the output and $w_t$ is the weight. But where is the actual model? Please update your question with the model equation first, then try to find the adaptive rule for weights – Nikos M. Jul 06 '20 at 09:06
  • You're right, I think I was getting caught up in the derivatives and forgot the model form. I think it makes sense now. – targetXING Jul 07 '20 at 00:03
  • It is also true that I was caught up in trying to make it adaptive. – targetXING Jul 07 '20 at 00:44
  • Ok, now we have the model. Still where does $f(t)$ fit into the AR model? What does it mean "*f(t) is a recursive dense layer*"? It seems you make some assumptions that are not stated, yet these should be clear and transparent – Nikos M. Jul 07 '20 at 09:21
  • I defined f(t). Thanks for your help. This was the first time I tried to formulate a new model. – targetXING Jul 09 '20 at 04:56
  • 1
    OK but I dont make much sense of this definition of $f(t)$. Why does it multiply $y_t$ with $w_{t+1}$ (doesnt make much sense, dimensional analysis, for example, fails)? You already have an error term $\epsilon_t$ in the definition of the AR model. Why is this function needed, what purpose does it fit? – Nikos M. Jul 09 '20 at 06:33
  • Hmm, I was thinking that since y(t) is an actual data point, it would be confusing to analyze it as a function. But I can see that it is confusing. – targetXING Jul 09 '20 at 17:25
  • Oh, I think I get your comment now... I think it was just easier for me to think about the weights at t+1 transforming y_t into y_t+1, but you're right. Dimensionally, it might be better to represent it as w_t. – targetXING Jul 09 '20 at 19:36
  • Hmm I still dont get what purpose does $f(t)$ serve.. But, there is another problem. Is the model discrete time or contnous time? it seems you mix both. You use $t$ as discrete variable (eg $t$, $t+1$) but you want to apply continous derivatives, instead of discrete ones. If one wants to sample or transform a continous system to discrete one (eg sample at $\Delta{t}$ intervals) then for example $y_n = y(n \Delta{t})$ and so on. $y(t+1)$ is confusing, it could be $y(t+\Delta{t})$, but cannot arbitrarily be set to 1, like it is a discrete variable eg $t=n$ – Nikos M. Jul 10 '20 at 09:28
  • Hmm, it’s definitely continuous because f(t) can output all real numbers. The sample rate could be anything I.e. T samples per second, and t +1 represents the next sample after t. – targetXING Jul 10 '20 at 13:26
  • The output may be continous but we are talking about the time variable. Plus one cannot pre-assign the sample interval $\Delta{t}$ to any value beforehand the analysis has to be carried out generally / abstractly if it is to be valid. Again you mix continous with discrete time analysis. Discrete time is difficult analysis but there are ways to pass from contious to discrete but this is not one of them – Nikos M. Jul 10 '20 at 15:49
  • I posted the question here: https://www.reddit.com/r/mathematics/comments/hotjhe/derivative_of_continuous_function_ft_with/ – targetXING Jul 10 '20 at 21:38
  • Interestingly, I got two answers 1) stating continuity doesn't matter 2) a suggestion to use finite difference f(t)w(t) - f(t-1)w(t-1). I ran this version of the model in tensorflow and got really cool results! – targetXING Jul 10 '20 at 21:39
  • 3) Itô’s Lemma, https://en.wikipedia.org/wiki/Itô%27s_lemma – targetXING Jul 10 '20 at 22:33
  • why dont you post this question on https://math.stackexchange.com/questions? It seems to have more informed community. I mean post your original question (not only about the derivative part) and see what you get – Nikos M. Jul 11 '20 at 08:46

1 Answers1

0

I guess the differentiation is wrong.

$y^{*}_{t+1} = w_{t+1}f(t) + \frac{df(t+1)}{d(t+1)}$

$y^{*}_{t+1} = w_{t+1}f(t) + \frac{d(w_{t+1}f(t))}{dt}*\frac{dt}{d(t+1)}$

$y^{*}_{t+1} = w_{t+1}f(t) + w_{t+1}\frac{df(t)}{dt} + f(t)\frac{dw_{t+1}}{dt}$

SrJ
  • 818
  • 3
  • 9