Dueling DQN - can't understand its mechanism

Question

I am trying to understand the purpose of Dueling DQN. According to this blogpost:

our reinforcement learning agent may not need to care about both value and advantage at any given time - this seems to be what I can't understand.

Let's assume we are in state $S_t$ and we select an action which has the highest score. This is the promised total reward that we will get in the future if we take the action.

Notice, we don't yet know the V or A of the future state $S_{t+1}$ (where we will end up after taking this best action), so decoupling V and A in any state, including S_{t+1} seems unnecessary. Additionally, once we do get to work with them we still seem to recombine them during $S_{t+1}$ into a single Q-value, just as was noted in the blog post.

So, to complete my thought: V and A seem to be a "hidden intermediate step", that still gets combined into Q, so we never know it's even there. Even if network somehow benefits from one or the other, how does it help if both streams still end up as Q?

A slightly unrenated thought, 'V' is the score of the current state only. 'A' is the total future expected Advantage, for a particular action, right?

Can someone provide a different example to the sunset?

Edit after answer was accepted:

Found a friendly explanation about this architecture here.

Also, if someone struggles to understand what V, Q and A is, read this answer, and my comment under it.

Answer 1

A slightly unrelated thought, 'V' is the score of the current state only. 'A' is the total future expected Advantage, for a particular action, right?

Not quite. $V$ is the total (discounted) future expected reward, assuming starting in state $s$, following current policy (in a control problem, usually best guess so far at the optimal policy) into the future. That includes selecting current action, $a$, according to the policy being assessed. The advantage function $A(s,a)$ (blog post has the arguments wrong for $A$) is the difference in value between selecting $a$ according to the current policy and selecting a specific action $a$. The value of $A$ also sums all future rewards assuming that the current policy is then followed into the future after this, maybe different, selection. Note that when the policy is the optimal one, and $V(s)$ is accurate, then $A(s,a)$ should always be zero or negative; optimal actions score zero, non-optimal ones will be negative.

Even if network somehow benefits from one or the other, how does it help if both streams still end up as Q? ... Can someone provide a different example to the sunset?

The more prosaic explanation is that the function decomposition is always technically correct (for an MDP). Coding the network like this incorporates known structure of the problem into the network, which otherwise it may have to spend resources on learning. So it's a way of injecting the designer's knowledge of reinforcement learning problems into the architecture of the network.

Conceptually this is similar to designing CNNs for computer vision with local receptive fields because we know edges and textures can be detected this way in images. Although CNNs have more than just that benefit, one of the positive aspects of the design for vision tasks, is that they structurally match known traits of the problem being solved.

Value-based RL control methods (as opposed to policy-gradient methods) work due to "generalised policy iteration", where the agent is constantly assessing the current values of a policy, then using those value estimates in order to make improvements. The split between $V$ and $A$ functions fits very well with that conceptually. The $V$ function is generally being adjusted to assess the current policy as accurately as possible, whilst positive values in $A$ function identify likely changes to the policy.