In our Intro to 64-bit, we mention that the limit for a 32-bit process (by default) in Win-32 is 2 GB. Likewise, we mention that the limit for a 64-bit process is 8 TB. I recently received a mail in which the author was telling me that he wrote a 64-bit app that requested memory by means of malloc and that it would stop around 6 GB, which was nowhere close to the 8 TB we talked about.
After some e-mails back and forth I found the specs for his system: EM64T with 4 GB of memory. It just so happens, that the memory your application can use is limited by how big your page file is + the amount of physical RAM you have.
In this case, the machine had 4 GB of RAM, the page file size was about 4 GB. If you open your task manager and look under the performance tab, you will find stats for "Limit" which is how much memory is available in your system. This number should be very close to your page file size + your physical RAM.
You can change the Page file size under the performance properties of your computer, and by doing so, you could modify the settings so that your "Limit" can increment.
In our tests we incremented the page file size to 11 GB. When our "memory hog" application was running, the limit was dynamically increased and could allocate about 17 GB of RAM. Keep in mind that the more paging you are doing, the slower things will be.