Reinforcement Learning (RL) has shown promising performance in environments for both robotic control and strategic decision making. However, they are usually treated as separate problems with different objectives. In this work, we propose the use of Reinforcement Learning to solve both control and strategic problems as one, in a multi-agent robotic soccer environment. We use the IEEE Very Small Size Soccer (VSSS) challenge from the Latin American Robotics Competition (LARC) as a study case. In the VSSS, two autonomous teams of wheeled robots compete by pushing the ball around to score goals. To unify both control and strategy problems, our approach gives full control of the actuators' speed to the RL algorithm whilst keeping the broader objective of winning the game. Our method achieves win rates as high as 93% against hand-coded heuristic strategies. In this work we contribute by developing an RL agent that can learn from self-play and generalize against new opponents. Our methodology uses multiagent Reinforcement Learning with self-play in order to build up the knowledge for complex tasks. We also developed a simulated environment for the robotic soccer game.