Silly argument? Thomas Jefferson would be surprised to hear that. See his memo to Geo Washington regarding whether the new fedgov had the authority to charter a bank.
The entire Bill of Rights is a limitation on government. The delegated authorities are elsewhere. Just because something is prohibited in the Bill of Rights emphatically does not mean its opposite is authorized. Read the history of ratification. The constitution came out of the convention without a bill of rights. Numerous, maybe all, Framers were opposed to a bill of rights. Their argument was that a bill of rights was not necessary, that the fedgov would have only the powers delegated, and, if a power was not delegated, it couldn't be exercised. They intended the constitution to stand alone. The Bill of Rights only came along after a lot people objected and threatened to derail ratification.
Also, your argument can be applied to any right. For example, journalists can be denied free press as long as they receive due process (another surprise to Thomas Jefferson who saw the federalist party--his political enemies--lose the next election and fade away after passing the Sedition Act.) By your logic, self-incrimination can be compelled as long as due process is followed. By your logic, a defendant can be denied representation at trial as long as due process is followed. By your logic, people can be subjected to cruel and unusual punishments as long as due process is followed.
You can argue for the constitutionality of denying felons their RKBA if you want; but, you're going to have to find another clause.
Now, please stop the intellectual dishonesty of declaring you've disposed of a silly argument merely by repeating your old argument without actually addressing the points raised.