We live and work in the Washington, DC area. We have a 19, 9 and 6 year old. We enjoy visiting Baltimore, MD (aquarium and historical places) Philadelphia, PA (historical places and museums), Hershey, PA (Hershey Park), Lancaster County (Amish), PA, Williamsburg, VA (historical places), the Smithsonian Museums in Washington, DC, and hiking in the Shenandoah Mountains (WV and VA). If history is what you want to see and you have limited time, I would prioritize the museums and monuments in Washington, DC and Philadelphia, PA. Amish Country is lovely but there is not a lot to do other than shop and go on tours about how the Amish live.
A must is a tour of the Capital. Contact your Congressman to set up a private tour. They will also book your tickets for the White House, Library of Congress, etc. 3 days is not really enough to do it all. New York absolutes are: Tribute to the World Trade Center, a Broadway Play or two and of course the Rockets.
You probably think New Jersey is just a place to drive through on the way from New York to Philadelphia. But one of its best points of interest is the Grounds For Sculpture -- an enormous sculpture garden located near Trenton, on what once was the New Jersey State Fairgrounds. Definitely worth a couple of hours at least.