Obviously I don't mean for the entire trip, but on a city by city basis. We are looking at Zagreb, Budapest, Brasov, Cluj and Bucharest specifically. There are some things outside of the city limits we want to see, mostly castles, but the budget may be a bit tight for a car rental. If it's a choice between not seeing them and getting a car, a car it will be but I had a bad experience with my last car rental on vacation and just would prefer not to if its feasible.
Your Itinerary is a little different than ours was, but I'm sure you can profit from our experience. We took the train from Budapest to Zagreb, walked and used the trolley in Zagreb, bussed from Zagreb to Plitvice Lakes National Park (not to miss), bussed from Plitvice to Split, walked Split, and took ferries down the Dalmation Coast to Hvar, then on to Korcula by ferry, and bus to Dubrovnik. Never needed a car, and when you arrive at your destinations using public transportation, there are locals waiting to meet the train, bus, or ferry to take you home and put you up in their pensiones, so much more fun, cheaper, and a greater experience than hotels. Renting a car puts you in your own bubble and isolates you from local people. Public transportation puts you one question away from a conversation with those around you. We never had a single problem figuring out how to get where we wanted to go, nor felt a car would have been a plus. You are immersed in the places you traveled to see when you are on foot and dependent on public transportation.
Croatia and Bosnia were both parts of Yugoslavia, the birthplace of all Yugo's. While on a walking tour of Sarajevo, the guide asked, "Do you know why they put a window defroster in the rear window of a Yugo?" "To keep your hands warm while you are pushing it!" He also explained that the Yugo was the only car sold with a city bus map included as standard equipment.
Mentioned in this answer:
Zagreb has excelent public transportation and very inexpensive taxi. I do not suggest renting a car while in the city, it is almost impossible to park in the old part if town.
If you decide to rent a car gor going outside the city, book it from the US to get better deals. Keep im mind that they druve stuck shift card and you need to specify you want an automatic car.
Make sure you visit Plitvice Lakes National Park while in Croatia.
In my opinion renting a car would be cheaper but it depends on how far from the city would you like to go. The railway line is centralised. it means every train goes from and to Budapest and usually there are no connections between the lines. Public transfer can be very expensive and if you are of 2 or more I think it will be cheaper and much more comfortable to rent a car. If you don't mind the time (+ trains schedules are not reliable) and your destinations are near to each other then would be only the public transfer a good choice.
I hope it was of help. Have a nice trip :)