A well-studied coloring problem is to assign colors to the edges of a graph G so that, for every pair of vertices, all edges of at least one shortest path between them receive different colors. The minimum number of colors necessary in such a coloring is the strong rainbow connection number (src(G)) of the graph. When proving upper bounds on src(G), it is natural to prove that a coloring exists where, for every shortest path between every pair of vertices in the graph, all edges of the path receive different colors. Therefore, we introduce and formally define this more restricted edge coloring number, which we call very strong rainbow connection number (vsrc(G)). In this paper, we give upper bounds on vsrc(G) for several graph classes, some of which are tight. These immediately imply new upper bounds on src(G) for these classes, showing that the study of vsrc(G) enables meaningful progress on bounding src(G). Then we study the complexity of the problem to compute vsrc(G), particularly for graphs of bounded treewidth, and show this is an interesting problem in its own right. We prove that vsrc(G) can be computed in polynomial time on cactus graphs; in contrast, this question is still open for src(G). We also observe that deciding whether vsrc(G) = k is fixed-parameter tractable in k and the treewidth of G. Finally, on general graphs, we prove that there is no polynomial-time algorithm to decide whether vsrc(G) ≤ 3 nor to approximate vsrc(G) within a factor n 1−ε , unless P=NP.