Yes, but just because you can log in via a Telegram Code, doesn't mean you have to. The attacker could get access to your SIM card, and select an option to get the code via an SMS message and not Telegram, and get access to your account. Look at the screenshot (from Telegram X), tapping the "Haven't received the code?" button will send the code via SMS.
The only solution is to get rid of phone numbers. the way iMessage allows you to de-register your number completely and use only an email address