The limit of data size

[waynelapierre]

I am writing a data I am retrieving from an remote Postgresql server to MonetDBLite. I got the error message: Error: cannot allocate vector of size 712.8 Mb. I thought as long as the data is smaller than my disk, MonetDBLite will be able to handle it. Is there any limit for the size of a single data to be able to fit into MonetDBLite?

[hannes]

Some more details please. What is the script you use, can you provide a reproducible example? And yes, as long as the data is smaller than your disk, it should be fine. Did you monitor disk use while running your export?

[waynelapierre]

My script is as below, I am sorry I cannot post my password and username here.

library(tidyverse); library(zoo)
require(data.table); library(lubridate)

### WRDS CONNECTION ###

library(RPostgres)
wrds <- dbConnect(Postgres(),
                  host='wrds-pgdata.wharton.upenn.edu',
                  port=9737,
                  user='user',
                  password='password',
                  dbname='wrds',
                  sslmode='require')

res <- dbSendQuery(wrds, "select DATE, PERMNO, PERMCO, SHROUT, PRC, RET, VOL
                   from CRSP.DSF")
#                   where PRC is not null")
crsp.dsf <- dbFetch(res, n = -1) # too large too fit in memory

library(DBI)
library(MonetDBLite)

dbdir <- tempdir() # creates a temporary directory
con <- dbConnect(MonetDBLite(), dbdir)


dbWriteTable(con, "crsp.dsf", dbFetch(res, n = -1))

[hannes]

Is this on Windows? Either way, I would probably fetch /append the data in chunks of 10000 or so instead of in one go. If on Windows, consider increasing the memory limit using memory.limit