I'm thinking about switching from designer to product planner

I strongly recommend it!

Come on in, come on in ~

That said, if you yourself think that a designer means a graphic designer, or if your organization understands the role that way, I'd urge you to escape that organization and get a bit more experience first.

At least in the IT market, a designer is not a stylist — that's my personal opinion.


Similarly,
You should aim to be a programmer, not just a developer
You should aim to be a service designer, not just a manager

Professor Kim Jin-woo, author of HCI 3.0, once said this at his author talk: When he tells people he does HCI, they tilt their heads in confusion, but when he says he "calculates UX," they understand.
Just as title and rank are different things.