Two-factor authentication systems aren’t as foolproof as they seem. An attacker doesn’t actually need your physical authentication token if they can trick your phone company or the secure service itself into letting them in.

Additional authentication is always helpful. Although nothing offers that perfect security we all want, using two-factor authentication puts up more obstacles to attackers who want your stuff.

RELATED: Secure Yourself by Using Two-Step Verification on These 16 Web Services

The two-step authentication systems on many websites work by sending a message to your phone via SMS when someone tries to log in. Even if you use a dedicated app on your phone to generate codes, there’s a good chance your service of choice offers to let people log in by sending an SMS code to your phone. Or, the service may allow you to remove the two-factor authentication protection from your account after confirming you have access to a phone number you configured as a recovery phone number.

This all sounds fine. You have your cell phone, and it has a phone number. It has a physical SIM card inside it that ties it to that phone number with your cell phone provider. It all seems very physical. But, sadly, your phone number isn’t as secure as you think.

If you’ve ever needed to move an existing phone number to a new SIM card after losing your phone or just getting a new one, you’ll know what you can often do it entirely over the phone — or perhaps even online. All an attacker has to do is call your cell phone company’s customer service department and pretend to be you. They’ll need to know what your phone number is and know some personal details about you. These are the kinds of details — for example, credit card number, last four digits of an SSN, and others — that regularly leak in big databases and are used for identity theft. The attacker can try to get your phone number moved to their phone.

There are even easier ways. Or, For example, they can get call forwarding set up on the phone company’s end so that incoming voice calls are forwarded to their phone and don’t reach yours.

Heck, an attacker might not need access to your full phone number. They could gain access to your voice mail, try to log in to websites at 3 a.m., and then grab the verification codes from your voice mailbox. How secure is your phone company’s voice mail system, exactly? How secure is your voice mail PIN — have you even set one? Not everyone has! And, if you have, how much effort would it take for an attacker to get your voice mail PIN reset by calling your phone company?

With Your Phone Number, It’s All Over

RELATED: How to Avoid Getting Locked Out When Using Two-Factor Authentication

Your phone number becomes the weak link, allowing your attacker to remove two-step verification from your account — or receive two-step verification codes — via SMS or voice calls. By the time you realize something is wrong, they can have access to those accounts.

This is a problem for practically every service. Online services don’t want people to lose access to their accounts, so they generally allow you to bypass and remove that two-factor authentication with your phone number. This helps if you’ve had to reset your phone or get a new one and you’ve lost your two-factor authentication codes — but you still have your phone number.

Theoretically, there’s supposed to be a lot of protection here. In reality, you’re dealing with the customer service people at cellular service providers. These systems are often set up for efficiency, and a customer service employee may overlook some of the safeguards faced with a customer who seems angry, impatient, and has what seems like enough information. Your phone company and its customer service department are a weak link in your security.

Protecting your phone number is hard. Realistically, cellular phone companies should provide more safeguards to make this less risky. In reality, you probably want to do something on your own instead of waiting for big corporations to fix their customer service procedures. Some services may allow you to disable recovery or reset via phone numbers and warn against it profusely — but, if it’s a mission-critical system, you may want to choose more secure reset procedures like reset codes you can lock in a bank vault in case you ever need them.

Other Reset Procedures

RELATED: Security Questions Are Insecure: How to Protect Your Accounts

It’s not just about your phone number, either. Many services allow you to remove that two-factor authentication in other ways if you claim you’ve lost the code and need to log in. As long as you know enough personal details about the account, you may be able to get in.

Try it yourself — go to the service you’ve secured with two-factor authentication and pretend you’ve lost the code. See what it takes to get in. You may have to provide personal details or answer insecure “security questions” in the worst case scenario. It depends on how the service is configured. You may be able to reset it by emailing a link to another email account, in which case that email account may become a weak link. In an ideal situation, you may just need access to a phone number or recovery codes — and, as we’ve seen, the phone number part is a weak link.

Here’s something else scary: It’s not just about bypassing two-step verification. An attacker could try similar tricks to bypass your password entirely. This can work because online services want to ensure people can regain access to their accounts, even if they lose their passwords.

For example, take a look at the Google Account Recovery system. This is a last-ditch option for recovering your account. If you claim to not know any passwords, you’ll eventually be asked for information about your account like when you created it and who you frequently email. An attacker who knows enough about you could theoretically use password-reset procedures like these to get access to your accounts.

We’ve never heard of Google’s Account Recovery process being abused, but Google isn’t the only company with tools like this. They can’t all be entirely foolproof, especially if an attacker knows enough about you.

Whatever the problems, an account with two-step verification set up will always be more secure than the same account without two-step verification. But two-factor authentication is no silver bullet, as we’ve seen with attacks that abuse the biggest weak link: your phone company.