Adding entry and exit hooks for functions

[programLyrique]

It makes it possible to look at the environment (and the return value) of a function at entry and exit.
Eventually, we can reproduce the instrumentation performed with R-dyntrace (but without having to modify the R codebase), or with inject (but much faster).

It now uses the new plugin stencils.

The plugin stencils take a data parameter but that must be a pointer that always points to the same address.
The type tracer needs to store an unknown number of elements in its trace so the design uses a manually allocated growable array with malloc to store them, and this is then put into a R external pointer type.

Limitations

• The type tracing is only performed at the exit hook, as more of the arguments will have been forced by then. It means that we might incorrectly find out the type of an argument if this one is reassigned to another type in the body of the function. It would be possible to peek at the promises at entry and then check if the type changes but it becomes more involved.
• Function whose name is <unknown> are not traced for types, as we would mix the types of many different functions.

Some outputs

There are 2 helpers to get the types at the end:

• rcp_get_types_df : returns a data frame with the argument names as columns, plus the number of arguments in dots, and the return value for a given function
• rcp_get_types: returns an environment where keys are function names and the value is a list of type results per calls. A type result itself is a named lists with 3 elements, arguments (a named list of the argument types), dots_count, and ret

Order of arguments does not matter

p <- function(x, y) {
  cat(x, y, "\n")
  y
}
p <- rcp::rcp_cmpfun(p, list(name = "p"))
p(1, "hello")
p(y=3, x="world")
print(rcp::rcp_get_types_df("p"))

Output:

          x         y dots_count       ret
1    double character          0 character
2 character    double          0    double

Dot argument is correctly handled

library(rcp)
h <- function(a, ...) {
  cat(a, ..., "\n")
}
h = rcp::rcp_cmpfun(h, list(name = "h"))
h(1, "hello")
h("world", 4, "three")
h(4L, t=89)
print(rcp::rcp_get_types_df("h"))

Output:

          a       ..1       ..2   t..1 dots_count  ret
1    double character      <NA>   <NA>          1 NULL
2 character    double character   <NA>          2 NULL
3   integer      <NA>      <NA> double          1 NULL

dots_count is the number of arguments in ... for the call. For those arguments, the displayed name is its tag name if it exists, and then its position (with the ..i convention).

TODO

• basic entry and exit hooks
• support collecting types
  • normal parameters
  • dots parameters (expand them to get the actual parameters inside?)
• test it
• benchmark it: with and without hooks, vs R-dyntrace, vs injectr

[Copilot]

Pull request overview

Copilot reviewed 57 out of 64 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (4)

rcp/tests/smoketest/basic.R:1

• These assertions are incorrect and will fail: (1) test_null() returns NULL but is compared to 42; use is.null(test_null()) (or change the function to return 42 if that’s intended). (2) After defining test_call, the compiled check asserts test_add instead of test_call. (3) stopifnot(test_call(2) == c(2, 1)) evaluates to a logical vector (and stopifnot() expects a single TRUE); use identical(test_call(2), c(2, 1)) or all(...).
  rcp/tests/smoketest/basic.R:1
• These assertions are incorrect and will fail: (1) test_null() returns NULL but is compared to 42; use is.null(test_null()) (or change the function to return 42 if that’s intended). (2) After defining test_call, the compiled check asserts test_add instead of test_call. (3) stopifnot(test_call(2) == c(2, 1)) evaluates to a logical vector (and stopifnot() expects a single TRUE); use identical(test_call(2), c(2, 1)) or all(...).
  rcp/src/stencils/stencils.c:1
• The realloc() result is assigned directly to trace->types without checking for allocation failure. On failure, this will set trace->types to NULL and immediately lead to memory corruption/crashes when writing rec. Use a temporary pointer, validate it, and only then update trace->types/capacity (and consider how to handle OOM—e.g., disable tracing for that call).
  rcp/tests/smoketest/ggplot2-unwinding.R:1
• This smoketest now hard-depends on ggplot2 being installed; library(ggplot2) will error and fail the whole suite in minimal environments. If ggplot2 isn’t guaranteed by the test image, gate the test with if (!requireNamespace('ggplot2', quietly=TRUE)) quit(status=0) (or similar) or ensure the CI/docker images install ggplot2 explicitly.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

@fikovnik I've opened a new pull request, #15, to work on those changes. Once the pull request is ready, I'll request review from you.

[MatejKocourek]

Looks good