Return flight ticket or open ended ticket

I am a bit confused about the question. Processing an application for a passport has nothing to do with your ticket, you do not need a ticket to get a passport, nor do you need a passport to get an airline ticket, I guess I would take care of the passport first to be safe.

Not all airlines offer open ended tickets, I would start with whatever airline looks like it has the best prices, and approach them. Thailand does require that you have a return or onward ticket, not sure if on open ended ticket meets that requirement, contact the Thai embassy regarding that. They also only allow you (US Resident I assume) 30 days anyway (you can extend in some cases another 30 days) so picking a return date may not be that difficult.

Another potential option would be to purchase a refundable ticket that allows changes without penalty, pick a date and if it changes, go through the change process.

Most countries want you to have a return ticket booked when you arrive. Many will not even allow you into the country if you don't have one except in special circumstances like you are there on a work permit visa.

As far as US passports go, there is no requirement to have any travel plans at all when you apply. You cannot get an appointment at a passport issuance office without a plane ticket in hand, but that is not how you should be planning on applying because that process is limited to persons leaving the country due to funerals and other unexpected last minute travel requirements. Most Post Offices (the real ones, not the privately run mail box places) handle passport applications. Since you say it will be months before you leave, there is no need for expedited service or to use one of those companies that advertise getting your passport quickly for you.

EDIT: Sorry, was thinking renewal when I stated you could mail it in. I have removed that.

You can RENEW a passport by mail, but you need to APPLY for a new passport IN PERSON. Post office is convenient for most people who do not live/work in a city with a Passport Office. And I'm pretty sure the charge is the same either place.

The "hefty surcharge" is what the Passport Office charges you when you apply there. They call it the "expedite service." You can pay that at the Post Office, but it is not required unless you want your passport really quick.

If you are a US citizen and need a passport, here is the official US government website, which will have all the forms, procedures, fees, and options for expediting (not needed if you are not traveling for a few months): https://travel.state.gov/content/passports/en/passports.html

There are many other websites that look official, but they just charge more for the same thing.

You do not need a ticket to get a passport; anyone who meets the requirements and submits the proper documents and fees can get a passport, even if they never intend to leave the US.

If you start to fill out the online application, it will tell you as part of the process if you are technically a "renewal" (done by mail and cheaper) or a "new application" (done at a receiving center like a post office). Just follow all the steps and they'll tell you exactly what to do and where to do it.