ArieNot sure what's wrong with your motherboard, but if it's broken and still under warranty, RMA.
With some minor effort, your memory might just run at 3600Mhz (CL16 should be fine), Ryzens will like that. Staying on this motherboard and dropping in a 5800X3D will give you the highest gaming performance offered on this socket.
It will depend on the games you play and what kind of FPS you're looking for whether to upgrade the GPU as well.
After I RMA the motherboard, I want to look at getting the ryzen 5700x with 2x16gb 3600 ram sticks
Would using my ryzen 3600 stock cooler be ok or is it better to just buy a cpu cooler?