We analyze the asymptotic symmetry of higher spin gravity with M × M matrix valued fields, which is given by rectangular W-algebras with su(M ) symmetry. The matrix valued extension is expected to be useful for the relation between higher spin gravity and string theory. With the truncation of spin as s = 2, 3, . . . , n, we evaluate the central charge c of the algebra and the level k of the affine currents with finite c, k. For the simplest case with n = 2, we obtain the operator product expansions among generators by requiring their associativity. We conjecture that the symmetry is the same as that of Grassmannian-like coset based on our proposal of higher spin holography. Comparing c, k from the both theories, we obtain the map of parameters. We explicitly construct low spin generators from the coset theory, and, in particular, we reproduce the operator product expansions of the rectangular W-algebra for n = 2. We interpret the map of parameters by decomposing the algebra in the coset description.