R Error: Cannot Allocate Vector of Size N GB (2 Examples) | How to Increase the Memory Limit
This tutorial shows how to increase or decrease the memory limit in R.
The tutorial contains two examples for the modification of memory limits. To be more specific, the article consists of the following topics:
So let’s take a look at some R codes in action:
The Starting Point: Running Memory Intense Function with Default Specifications
Let’s assume we want to create a huge vector of randomly distributed values using the rnorm function in R. Then we could use the following R code:
x <- rnorm(4000000000) # Trying to run rnorm function # Error: cannot allocate vector of size 29.8 Gb
Unfortunately, the RStudio console returns the error message: “cannot allocate vector of size 29.8 Gb”.
Even though there is no general solution to this problem, I’ll show in two examples how you might be able to fix this issue. So keep on reading!
Example 1: Garbage Collection Using gc() Function in R
The first possible solution for this problem is provided by the gc function. The gc function causes a garbage collection and prints memory usage statistics.
Let’s do this in R:
gc() # Apply gc() function # used (Mb) gc trigger (Mb) max used (Mb) # Ncells 531230 28.4 1156495 61.8 1156495 61.8 # Vcells 988751 7.6 8388608 64.0 1761543 13.5
Have a look at the previous output of the RStudio console. It shows some memory usage statistics that might be helpful to evaluate your RAM usage.
Perhaps, the call of the gc function solves our memory problems:
x <- rnorm(4000000000) # Trying to run rnorm function # Error: cannot allocate vector of size 29.8 Gb
Unfortunately, this is not the case in this example. So let’s try something else…
Example 2: Increase Memory Limit Using memory.limit() Function
Another solution for the error message: “cannot allocate vector of size X Gb” can be the increasing of the memory limit available to R. First, let’s check the memory limit that is currently specified in R. For this, we simply have to call the memory.limit function as shown below:
memory.limit() # Check currently set limit # [1] 16267
The RStudio console shows that our current memory limit is 16267.
We can also use the memory.limit function to increase (or decrease) memory limits in R. Let’s increase our memory limit to 35000:
memory.limit(size = 35000) # Increase limit # [1] 35000
Now, we can run the rnorm function again:
x <- rnorm(4000000000) # Successfully running rnorm function
Nice, this time it worked (even though it took a very long time to compute)!
Video, Further Resources & Summary
Do you want to learn more about adjusting memory limits? Then you may want to have a look at the following video of my YouTube channel. In the video, I’m explaining the R syntax of this tutorial.
The YouTube video will be added soon.
Note that the examples shown in this tutorial are only two out of many possible solutions for the error message “cannot allocate vector of size X Gb”. Have a look at this thread on Stack Overflow to find additional tips and detailed instructions.
Furthermore, you may want to have a look at the other tutorials on Statistics Globe.
- Clear RStudio Console
- Remove All Objects But One from Workspace in R
- Difference between rm() & rm(list=ls())
- Clean Up Memory in R
- Determine Memory Usage of Data Objects
- What’s the Difference Between the rm & gc Functions?
- R Programming Tutorials
In summary: At this point you should know how to increase or decrease the memory limit in R programming. If you have additional questions, please let me know in the comments. Besides that, don’t forget to subscribe to my email newsletter in order to get updates on the newest tutorials.
I’m Joachim Schork. On this website, I provide statistics tutorials as well as code in Python and R programming.
24 Comments. Leave new
PLEASE IT IS STILL GIVING ME A POP-UP OF CANNOT ALLOCATE VECTOR OF SIZE 53.1Gb
what should I do
Hey Pamela,
It’s impossible for me to help you without further information. Could you tell me some more details?
Regards,
Joachim
Hello,
I have the same problem as Pamela. I’m still new to analyzing data in this way, so I’ll start with some detail, and can provide more if needed. I’m working through a large clustering analysis with hclust() using a matrix of 24000 x 24000. Initially, I’d used the code above to increase the memory limit to 15000 which enabled me to calculate the adjacency in an earlier step, but that limit generated a warning of “memory overload” when I reached the hclust step. I increased the memory limit to 40000 and then to 60000, but continue to get “Error: cannot allocate vector of size 2.2 Gb” (2.2 for both 40000 and 60000). I’ve used the collect.garbage() function to try to free up memory, but that hasn’t worked either.
Hi Hank,
Thanks for the comment and the detailed explanations.
Is there a way to clean up your data to decrease its size before analyzing it?
For example:
– Remove rows and columns that contain only NA values: https://statisticsglobe.com/r-remove-data-frame-rows-with-some-or-all-na
– Remove duplicate rows and columns from your data: https://statisticsglobe.com/remove-duplicated-rows-from-data-frame-in-r
– Remove rows or columns that do not contribute any value to the analysis.
Generally speaking: I assume that at some size R will simply not be able to handle your data anymore. For that reason, I’d try to decrease the size of the data as much as possible before trying to increase memory limits etc.
I hope that helps!
Joachim
Thank you so much, this worked perfectly for me.
That’s great to hear, thank you for the nice comment!
I would also suggest you sample the data as suggested in
https://stackoverflow.com/questions/40989003/hclust-in-r-on-large-datasets
Hey Ross,
Thank you for the additional resource!
Regards
Joachim
Hi Joachim
I tried to apply the example you gave but I still facing the problem. I did all the steps correctly but the problem still there.
Hey Omar,
Unfortunately, this error message can have many different causes, so it’s not possible to give a general solution for all cases.
You may have a look at this thread on Stack Overflow for more possible solutions.
I hope that helps!
Joachim
I had the follow message on my R studio R version 4.2.1:
Warning message:
‘memory.limit()’ is no longer supported
Is there another way? Thank you.
Hello Carlos,
From the error message, it seems like the function has been deprecated or is no longer supported in the R version you’re using (4.2.1).
If you’re encountering memory limitations with your R code, consider the following steps:
Data Size Reduction: Reduce the size of your data if possible. You can do this by removing unnecessary columns, sampling the data, or using data.table or dplyr packages for efficient data manipulation.
Memory-efficient Data Structures: Use more memory-efficient data structures, for example, use matrices instead of data frames if the data allows it.
Efficient Coding Practices: Use efficient coding practices. Avoid loops where possible, instead, use vectorized functions and operations, which are much more efficient in R.
Clear Memory: Clear memory that is no longer needed by removing objects that are no longer used (rm() function) and forcing a garbage collection (gc() function).
Big Data Tools: Use big data tools like bigmemory, ff, disk.frame or daskr packages if you’re dealing with really big data. These tools can help handle datasets that are too large to fit in memory.
Best,
Cansu
Hi Cansu.
Thanks for the prompt answer. I’m trying to filter the dataset as suggested, however, I’m connecting to the database via ODBC, where:
My_df <- dbReadTable(con, Id(schema = "my_schema", table = "my_table"))
By the way, do you know guys if it’s possible to filter the dataset by the ODBC dbReadTable function? Cause, via the function mentioned above, the “dbReadTable” function from the DBI package, reads the entire dataset.
Thanks in advance.
Hello Carlos,
Based on a short research, I found out that you could use the dbGetQuery function instead. This function is supposed to allow you to write SQL queries, and you can include a WHERE clause to filter the data. See the documentation here. But I am not an expert on this topic myself. If you still encounter some problems, you can consult our Facebook discussion group, where people can ask questions about R programming and statistics.
Best,
Cansu
I will take a look. I found nothing in my research. Thanks so much.
Perfect!
Have a good one,
Cansu
Hi ever body,
I cannot fix Error with Error: cannot allocate vector of size 5.5 Gb on order above
are someone to guid me?
Hello Maryam,
I was on holiday. Do you still need help?
Best,
Cansu
I tried the memory.limit(),but it gave me a warning message:
‘memory.limit()’ is no longer supported
Hello Flower,
If you’re getting a warning that memory.limit() is no longer supported, it’s likely that this function has been deprecated in a more recent version of R that you are using. This means the R development team has decided to phase out this function, possibly because modern operating systems manage memory differently, and setting a limit within R may not be as necessary or effective as it once was.
Here are a few alternative approaches you might consider to manage memory usage in R:
Optimize Your Data:
Use more memory-efficient data structures (e.g., use data.table instead of data.frame, or store your data in a matrix if all the elements are of the same type).
Only load the columns you need, rather than the entire dataset.
Use the rm() function to remove objects that are no longer needed, and then call gc() to free up memory.
Consider working with a sample of your data if that is appropriate for your analysis.
Increase Your Available Memory:
If possible, close other applications to free up more memory for R.
Consider adding more RAM to your computer.
Use Disk as Memory:
For very large datasets, consider using on-disk data tables (e.g., with the ff or bigmemory packages) that allow you to work with datasets that are larger than your available RAM.
Use Distributed Computing:
For very large-scale data processing tasks, you might consider using a distributed computing system like Spark (with the sparklyr R package) or H2O, which allows you to distribute your data and computations across multiple machines.
Check System Memory Information:
On Linux, you can use system commands like free -m (to check free memory in MB) or top (for a real-time view of system status, including memory) from within R using the system function. On Windows, you might be able to use the systeminfo command to get similar information, and on macOS, you can use the vm_stat command.
Please note that these solutions are somewhat general and may need to be adapted to your specific use case and operating system.
Best,
Cansu
H ello I recieve error of “Cannot allocate vector of size 289.6 TB “
Hello,
The size of “289.6 TB” (terabytes) is extremely large, and it’s almost certainly not a typical amount of memory you’d have on a standard machine. This suggests that something might be going wrong in your code or data processing.
Best,
Cansu
Hello, I tried to run memory.limit(), but retrieved: warning:’memory.limit()’ is no longer supported[1] Inf. What doest it stand for, and how to solve it? Thanks
Hello Patricia,
The function memory.limit() is no longer supported. This means that its use is now discouraged, possibly because the developers believe there are better ways to handle memory management or that the function isn’t as relevant anymore.
The value [1] Inf typically means that there’s no effective limit on the memory R can use. The value Inf stands for “Infinity”.
I found a discussion on Posit Community about this topic. It may be helpful.
Best,
Cansu