computer becomes unresponsive with MonetDBLite working hard

[tslumley]

On my laptop (MacBook Air, four years old) when I execute a query in MonetDBLite (from R.app) that creates a table much larger than memory, the computer becomes unresponsive while the query executes. About half the time there's the spinning beachball cursor, the other half there's a normal cursor but other applications are extremely slow to respond (eg, typing takes multiple seconds per character).

The behaviour looks similar to what you get with memory overuse and heavy paging, but it isn't: there's still free RAM. The 'Activity Monitor' still indicates 20-40% idle CPU time, but it shows that most of the CPU time being used is in the system, not in user space.

Is there some sort of checking for interrupts that MonetDBLite could do more often, or something?

[hannes]

I have to check interrupt handling again, its tricky since R and MonetDB (standalone) both mess with them. I think I disabled them for MonetDBLite. My suspicion though is that the slowdown comes from heavy IO traffic / contention with the disk, which is required when the new table is persistent (as is the case here I think).

Something you could try though is to run the table creation in single-threaded mode
(options(monetdb.sequential=T) before startup). Does this help?

[tslumley]

I think it's a temporary table -- and the same thing happens when the additional table only exists as the inner part of a SELECT. Stripped down, that second case is like
select sum(u) from select x*(y-mu) as u from select x,y, (1-1/exp(A+x*B)) as mu from data
where A and B are explicit numbers.

A possible argument against it being disk contention is that the Activity Monitor only shows intermittent disk activity. But I'll try the single-threaded option.

[hannes]

Did the single-thread option change anything?

[tslumley]

?Yes, it helped. Also, for reasons I don't understand at all, the whole thing got faster even without that. -thomas Thomas Lumley Professor of Biostatistics University of Auckland

…

________________________________ From: Hannes Mühleisen <notifications@github.com> Sent: Monday, March 19, 2018 8:17 PM To: hannesmuehleisen/MonetDBLite-R Cc: Thomas Lumley; Author Subject: Re: [hannesmuehleisen/MonetDBLite-R] computer becomes unresponsive with MonetDBLite working hard (#20) Did the single-thread option change anything? - You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#20 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AH_fvb409kUdWZL2VaqlYU-6lbwoam43ks5tf1t8gaJpZM4Sdvdw>.