Led by a surge in the amount of data being analyzed in data warehouses, the world’s largest databases are pushing new heights as they double and triple in size, according to a survey.
Survey: Biggest Databases Approach 30 Terabytes
About The Author
Eugenia Loli
Ex-programmer, ex-editor in chief at OSNews.com, now a visual artist/filmmaker.
Follow me on Twitter @EugeniaLoli
So what makes anyone think that the TIA (Total Information Awareness) project was ever really dropped, even if officially?
Yes, please note that this only discusses PUBLICLY KNOWN databases. I seem to recall seeing a 16 TB tape library unit from the _1980’s_ in the NSA’s Cryptography Museum. I would be very surprised if the worlds largest database wasn’t located in nearby Ft. Meade, and isn’t at least an order of magnitude larger than what this article discusses.
-braddock
From the article…
The program requires that the databases must be in production and contain at least 1 terabyte of data (or 500 megabytes of data if running on Windows).
Why special considerations for windows???
Aren’t you supposed to be “Missing In Action”..?!
Sorry, couldn’t help it!!!
“”Why special considerations for windows???””
Processor limitations?
Windows implies x86 hardware, which comes with a 4Gb (Without using paging extensions) address space, afaik Windows uses the top 2Gb for kernel addresses, with the rest user space sections. So maybe Windows applications are limited to 500mb of data area (Perhaps someone more knowledgable can clarify this).
Even Linux is limited to 3Gb of application address space on x86 (Top 1Gb is used for the kernel, same PAE caveat as before). So trying to run a terabyte sized DB on x86 architecture involves a lot of shuffling around of data, which would slow everything to a crawl. So perhaps they’re trying to be fair in comparison to processors with a 64-bit address space.
I have seen numbers indicating that on average 20% of all data is incorrect due to poor data-entry, fake data etc.
Just think about it – 20% of 30 Terabytes is 6 Terabytes of useless/garabage data. Wow !!!
Largest database:
Babar:
This database is arguably the largest in the world: as of Mon Nov 10 00:01:28 2003, over 834.1 TB has been stored in 710021 files.
info:
http://www.slac.stanford.edu/BFROOT/www/Public/Computing/Databases/…
Processor limitations?
Windows implies x86 hardware
And what about the 64-bit SQL Server 2000 for IA64?
http://www.microsoft.com/sql/64bit/default.asp
“”And what about the 64-bit SQL Server 2000 for IA64? “”
Is this the first architecture you think of when you hear someone comment on the Windows operating system? I doubt it.
Is this the first architecture you think of when you hear someone comment on the Windows operating system? I doubt it.
No, but it’s certainly the first platform that comes to mind when discussing the largest Windows-driven databases in the world.
“”No, but it’s certainly the first platform that comes to mind when discussing the largest Windows-driven databases in the world.””
Well in that case you have a point, and the 500mb criteria seems bizarre.
You should see me l33t clustered pr0n database of thumbnailed sized… oops.
Any of these MySQL Databases?
MySQL has troubles with anything over 30MB, let along 30TB.
I find the best and fastest database to be Intersystem’s Cachè. http://www.intersystems.com
“MySQL has troubles with anything over 30MB, let along 30TB.”
Oh well i’m having 4gb databases atm with mysql with no problems at all, I will tell you when I reach 30TB ok?
MySQL has troubles with anything over 30MB, let along 30TB.
Some peoples run multi-TB database on MySQL. My database is “only” 230 MB and is running without a hitch. If you have a problem with 30 MB, then you misconfigured something – don’t blame MySQL for it.
Let’s not forget that MS is also distributing a version of Windows specially for powerful servers (Datacenter Edition). My guess is that 500MB is a typo and that they really meant 500GB.
The myth that MS SQL server can’t compete in the large database arena gets blown out of the water! a 5.3 TB SQL server 2000 database..thats pretty impressive. I would still like to see the hardware setup for it.
Hardware setup – simple.
A large Sun box with an “Intel Inside” and a “Designed for Windows” badges tacked on with scotch tape.