Thirsting Blade question about number of attacks

If you have a weapon in each hand and Extra Attack, neither light, you can use Extra Attack to attack with each of them on your main action.

But, if you have Thirsting Blade, and no Extra Attack, can you attack once with the non-Pact weapon, and then make another attack with your Pact Weapon, or would you be locked in to a single attack after you make the non-Pact attack?