Databases

• When you've got small amounts of data (megabytes) you can store it in the RAM during processing.
• When you have more (gigabytes), you can swap it out to hard disk. This can slow down processing, but is usually not a problem. (Thrashing)
• In addition to storing it during processing, you might have different uses for the data.
• Databases fit in here.
• They're a field unto themselves, but most medium sized organisations have databases. I think all large ones do.
• You often extract data from them for other uses.
• However, these databases are often in use; they're being constantly updated.
• If you want to do some learning on them, you often make a data warehouse.
• This can be specially constructed (aligned for fast retrieval) for the data you're interested in.
• They can also be cleaned up (with bad items removed, and redundant entries combined).
• They get updated rather infrequently (e.g. every week or more).