Accessing Retirement Accounts Early

We look at Roth IRA conversions as a mechanism to reach into our retirement funds sooner, and lower our tax obligations along the way.

This post is long and quite wonky if you ask me. If you make it to the end and need a supportive hug, I’m here for you. - sincerely, emotional me.

Coming of age

How often do you actually look at your pay stub?

I remember getting my first paycheck from my job at the local video store, and thinking “Taxes? FICA!? Who the heck is FICA? And why are they taking my money!?” After all, FICA didn’t clean up after that kid who threw up on the new Lethal Weapon 2 poster. That was me.

My adolescent concerns over high taxes and big government were short-lived once I cashed that paycheck at my local bank. The bank teller skillfully counted out a stack of bills right in front of me, and then she placed it in my hand. The bills were crisp and untainted. It felt like a sin to fold them. They even smelled new. Yes, I smelled them. It was a viscerally satisfying experience.

It was a coming-of-age moment, a farewell to the money-making schemes of my childhood, the most disgusting of which was dumpster diving for aluminum cans that I could cash in at my local recycling center.

Flush with cash at the age of 16, I stopped worrying about taxes and instead focused on my take-home pay from that day forward. It was the first time I pushed the autopilot button in my life, turning a blind eye to all those subtractions from my gross pay. For decades, my default response to taxes was to capitulate, to never question whether I was paying too much. I was doing my part.

Don’t get me wrong, I’m not anti-tax. We’ve paid a lot of taxes over the years. And I personally like roads, schools, parks, and all the many services available at the local, state, and federal levels. This isn’t a post where I rant about wasteful government spending and unfair tax policy, but rather a resolution of sorts to take advantage of the unique tax-saving levers available to early retirees.

Tax-saving things we did right, even if accidentally so

There were relatively few tax-saving levers to pull as regular W2 workers, but we managed to do some of the basics that didn’t require too many financial acrobatics.

We deferred some taxes

Most of the time, we contributed the maximum amount to tax-advantaged retirement accounts (e.g. a 401K). We aren’t avoiding taxes on this money altogether, but we did kick the can down the road, deferring our tax bill until we’re in a lower tax bracket.

We avoided some taxes, too

For most of our working life, we had a high-deductible health plan and were eligible to contribute money to a Health Savings Account (HSA). HSAs are unique because they allow both tax-free contributions and tax-free withdrawals on qualifying expenses. In other words, this is money we will never have to pay taxes on.

We saved some after-tax money in a brokerage account (e.g. eTrade)

In the 4 years leading up to early retirement, we drastically cut the cost of our lifestyle and used the extra dollars to pay off debt and build up a post-tax investment account. We automated monthly investments into low-cost index funds. While not directly a tax-saving move, boosting our savings and taxable investments is critical to accessing money in our retirement accounts early.

Tax blunders, and other things we did wrong

Fear and short-term thinking

Some years we hardly contributed to our 401K at all, or we “went to cash,” fearing a big market downturn. That worked against us nearly every time. We paid higher taxes and missed out on dividends and capital gains while our money sat in cash earning .03%.

Raiding the livelihood of our future selves

In one instance, we cashed about $30,000 out of one of our legacy retirement plans, paying both the taxes and early withdrawal penalties.

Just writing that sentence makes me also want to throw up on a Lethal Weapon 2 poster.

Throwing money at our problems, without thinking critically

One year, the IRS sent us a letter telling us we owed them more money, and we just paid it. We didn’t have an accountant look at it. We didn’t even look at it ourselves. We just assumed we made a mistake, wrote the check, and moved on.

In spite of ourselves, we’ll end up a-sittin’ on a rainbow

I want to share our failures because the internet loves to shine its light brightest on people who are seemingly making all the perfect moves. It’s a facade. No one is perfect. This stuff is hard. Forgive yourself.

So how do we plan to get to our money early and pay so little in taxes?

Well first, we need a very quick, overly simplistic, and incomplete primer on IRAs, 401Ks, and Roth IRAs before we go much further.

Standard IRAs and 401Ks

Standard IRAs and 401Ks allow you to save money for retirement before the taxman touches it. They lower your taxable income today, but you have to pay taxes on the withdrawals in retirement. These are called tax-deferred investments.

1. You are not taxed on the contributions you make to IRAs and 401Ks.

2. You are taxed on the withdrawals you make from IRAs and 401Ks.

Roth IRAs

Roth IRAs allow you to save money for retirement after the taxman touches it. You are paying taxes up front on the contributions. But when you take a distribution, the money is tax-free.

1. You are taxed on the contributions you make to a Roth IRA.

2. You are not taxed on the withdrawals you make from a Roth IRA.

Is this too wonkish? Am I losing you here? You look pale. Are you feeling sick? Do you need a Lethal Weapon 2 poster?

Solving the early retirement problem - how do we get early access to our IRA funds?

Retiring early presents problems. We need to cover our living expenses in our early fifties, but we are too young to access the money in our IRAs and 401Ks, and our Social Security benefits are still 15 to 20 years away.

Enter the Roth conversion.

The Roth Conversion

We can’t directly withdraw money from our IRAs early (before 59 1/2) unless we pay a penalty. But we are allowed to move money from our IRAs to our Roth IRAs, penalty-free. This is called a Roth Conversion.

We have to pay taxes on any Roth conversions we do, so it is important to do Roth conversions in years when our income is really low. The lower our tax bracket, the fewer taxes we will have to pay.

Allen? It's me, er.. you.. emotional you. How exactly are we supposed to keep our income low and still, you know, eat?

Oh, hi emotional me, I was wondering when I would see you in this post.

Remember those savings that we built up in our brokerage account? When we retire early and spend from our savings, we are spending money that has already been taxed. Our income, on paper, is very low. But our actual cash flow is just fine.

Ok, but if we do this, we're basically moving money from one retirement account to another, and on top of that, we're paying taxes. 

It feels like we are taking money out of one pocket, giving some of it away, and putting what's left in the other pocket. 

How does this really help us?

Roth conversions aren’t for everyone. But for us, they seem ideal.

We need to access our retirement money early. Unlike standard IRAs, we are allowed to access Roth conversion money before we turn 59 1/2. So it checks that box.

Are we giving up a little money today in order to pay taxes on those transfers? Sure, but the taxes will be low and then our money grows tax-free forever.

Building the ladder

There is one little snag in our plan.

I knew it :(

Cool it, Eeyore.

So, yes, we technically can access our Roth conversion money before retirement age, but only after 5 years have passed since the money was converted. On top of that, each Roth conversion that we perform gets its own 5-year clock, so as early retirees:

1. We need to do conversions every year.

2. We need enough savings to cover the first five years of living expenses while our conversions marinate.

A sample Roth conversion ladder. During the first five years of conversions, living expenses are funded by savings. After five years, living expenses can be funded, tax and penalty-free, from earlier Roth conversions.

Checking in with the nerd

For us, this seems like the best strategy to reduce our taxes over the course of our retirement. But let’s consult the nerd.

Hey, spreadsheet nerd me, are you there?

Yes sir. I'm here.

What do the numbers say?

Well, sir, it should take us around 20 years to fully convert our IRAs over to Roth IRAs. Our effective tax rate on those conversions should be around 7.6%.

Ok, and after that?

Well, sir, after that our income will come from our tax-free Roth IRAs and Social Security. Our effective tax during that period of time should be around 3%.

If, and this is a BIG IF, we're fortunate enough to live until 95, it all averages out to a tax bill of around 5.3%.

Big If!? What are you trying to say?

Have you seen my spreadsheet schedule for cardio workouts? And the diet tracker? And our latest bloodwork pivot table? I really think that we're not doing enough to -

Thank you, spreadsheet nerd me! That’ll do! We don’t have time to talk about the workout schedule or how we’re not eating enough kale. Maybe next time.

Disclaimer: I’m not a certified financial advisor, CPA, or tax strategist, nor am I acting as your fiduciary. I don’t even do my own taxes. Any investment or tax-related musings are my thoughts, and my thoughts only. They exist for your entertainment. Consult your own tax and financial advisors on your situation.