Well, it seems this topic has a lot of speculating about things but here are the actual facts:
1. In Windows Vista with UAC Internet Explorer 7 and more so 8 are more secure then other browsers like Firefox and Opera just because of Protected Mode as all the web pages run at the lowest integrity level.
Any security hole in Internet Explorer 7/8 in the page code can not write anything and or do bad things while in Protected Mode. You would have to run an unsafe control outside of Protected Mode for that to happen and Internet Explorer will not do that unless you tell it too.
2. Activex in Internet Explorer is no more unsafe the Netscape NPAPI Plugin system that other browsers like Firefox and Opera use.
Both systems allow code to be installed and run on your computer.
This code can do anything from writing files to displaying popup ads.
It just so happens that Internet Explorer was the one that got the majority of malware written for it. It's hard to say for sure why but the fact that most people use Internet Explorer is probably a big factor.
3. For Internet Explorer's invasiveness it depends on how you look at things.
Internet Explorer consists of two parts the backend engine that renders web pages and the front end you use when you run iexplore.exe
The backend engine is included with Windows and is used by many things built-in like Windows Explorer to third-party programs like Quicken and IETab.
The frontend is just a normal browser like Firefox that uses the backend and can be removed without issue and in fact in Windows 7 you will be able to remove it.
It just so happens that things like Portable Firefox include the backend (Gecko Rendering engine) and the frontend (Firefox) in one package that can be moved from one system to another.
There really is no need for a Portable Internet Explorer when it's installed on the computer.
4. The invasiveness of Mozilla Firefox is not that much.
ScottW is correct. All you need to do is either uncheck 'Set As Default Browser' in the installer or after the install go to the "Default Programs" in Control Panel and set Internet Explorer or another browser as the default and you will never see any part of Firefox launching for anything without you doing it.
As for Google Chrome, that would qualify as invasive because a process (GoogleUpdater.exe) is installed and always runs in the background without any permission from you.
For Safari you can stop it's updater (Apple Software Update) in the installer or after the install by uninstalling it though it's a little hidden and hard to figure out.
5. As for the study that said Firefox users were smarter.
That is just warping of statistics, overall there are more Internet Explorer users and Firefox users so while it would seem like more smart people use Firefox the truth is plenty of Internet Explorer users are smart and in fact by volume of users, there are likely more smart people using Internet Explorer.
6. What browser should you use?
Whatever works best for you.
I happen to be using three at the moment.
Internet Explorer with IE7Pro, Firefox with many extensions and the Google Chromium Trunk Build (I use it because It doesn't include the Google installer or updater).