When it comes to presses everyone has their own preference. There are many manufacturers, Lee, Dillion, Hornady, RCBS are some of the most prominent. I admit Lee is my preferred brand. They offer a great selection of different presses and dies and even have package deals that give you the press and dies for a specific cal.. Here's a link to their web site.
I would suggest either a simple turret press or a progressive press. The progressive press does all the functions at once and basically spits out a bullet ready to fire with each pull of the lever. Some beginners feel a little apprehensive with these, but the learning curve is very gentle.
As to the powder, cases, and bullets you can either get them at any decent gun store or you can order them. Gun shows can also have some decent deals(though true deals are getting harder to find). If you choose mail order I would recommend Natchez Shooters Supplies
I will warn you their website is a horror to find anything, but keep looking it's worth it. Here's a link to the Lee Pro 1000s offered by Natchez. You will see that a full press set up in any of the most common cal. is around $150. You should be looking at around $300 total(depending on what accessories you choose) to start. I have in no way given you all the things you can get, just a quick overview of what you will need. Hope it helps.
It's usually better to buy your powder and primers at a store, if you mail order them they will charge you a hazmat fee in addition to the price and shipping.