Welcome to The Department of Fatherland State Security I mean Homeland Security.
Not enough to tell if this was a VIPR team or not, but my guess is it was not as they usually only show up with the local cops in tow. There are separate rules for the "inspection" (not "search") of things passengers bring onto ferries, and those "inspections" are usually carried out by rent-a-cops.* I've never been able to find anything written down that addresses the difference between an "inspection" and a "search". I have seen the regs that allow "inspections" of vehicles and (for lack of a better term) luggage, but did not see anything that permitted the "inspection" to go farther than looking at the top layer when the luggage is opened.
From the description you provide, it was a perfectly legal violation of 6th Amendment rights in furtherance of anti-terrorism security theater.
* While accumulated anecdote will never = data, I've never heard of anybody but rent-a-cops performing this security work.