knitr gets tricked by data.table `:=` assignment

by

Update Oct 2014. Now in data.table v1.9.5 :

:= no longer prints in knitr for consistency with behaviour at the prompt, #505. Output of a test knit("knitr.Rmd") is now in data.table’s unit tests.

and related :

if (TRUE) DT[,LHS:=RHS] now doesn’t print (thanks to Jureiss, #869). Test added. To get this to work we’ve had to live with one downside: if a := is used inside a function with no DT[] before the end of the function, then the next time DT is typed at the prompt, nothing will be printed. A repeated DT will print. To avoid this: include a DT[] after the last := in your function. If that is not possible (e.g., it’s not a function you can change) then print(DT) and DT[] at the prompt are guaranteed to print. As before, adding an extra [] on the end of a := query is a recommended idiom to update and then print; e.g. > DT[,foo:=3L][]

Previous answer kept for posterity (the global$depthtrigger business is no longer done as from data.table v1.9.5 so this is no longer true) …

Just to be clear I understand then: knitr is printing when you don’t want it to.

Try increasing data.table:::.global$depthtrigger a little bit at the start of the script.

This will be 3 for you currently :

data.table:::.global$depthtrigger
[1] 3

I don’t know how much eval depth knitr adds to the stack. But try changing the trigger to 4 first; i.e.

assign("depthtrigger", 4, data.table:::.global)

and at the end of the knitr script ensure to set it back to 3. If 4 doesn’t work, try 5, then 6. If you get to 10 give up and I’ll think again. ;-P

Why might this work?

See NEWS from v1.8.4 :

DT[,LHS:=RHS,...] no longer prints DT. This implements #2128 “Try
again to get DT[i,j:=value] to return invisibly”. Thanks to discussions here :
how to suppress output when using `:=` in R {data.table}, prior to v1.8.3?
http://r.789695.n4.nabble.com/Avoiding-print-when-using-tp4643076.html
FAQs 2.21 and 2.22 have been updated.

FAQ 2.21 Why does DT[i,col:=value] return the whole of DT? I expected either no visible value (consistent with <-), or a message or return
value containing how many rows were updated. It isn’t obvious that the
data has indeed been updated by reference.
This has changed in v1.8.3
to meet your expectations. Please upgrade. The whole of DT is returned
(now invisibly) so that compound syntax can work; e.g.,
DT[i,done:=TRUE][,sum(done)]. The number of rows updated is returned
when verbosity is on, either on a per query basis or globally using
options(datatable.verbose=TRUE).

FAQ 2.22 Ok, thanks. What was so difficult about the result of DT[i,col:=value] being returned invisibly?
R internally forces
visibility on for [. The value of FunTab’s eval column (see
src/main/names.c) for [ is 0 meaning force R_Visible on (see
R-Internals section 1.6). Therefore, when we tried invisible() or
setting R_Visible to 0 directly ourselves, eval in src/main/eval.c
would force it on again. To solve this problem, the key was to stop
trying to stop the print method running after a :=. Instead, inside :=
we now (from v1.8.3) set a global flag which the print method uses to
know whether to actually print or not.

That global flag is data.table:::.global$print. At the top of data.table:::print.data.table you’ll see it looking at it. That’s because there is no known way to suppress printing from [ (as FAQ 2.22 explains).

So, inside := inside [.data.table it looks to see how “deep” this call is :

if (Cstack_info()[["eval_depth"]] <= .global$depthtrigger) {
    suppPrint = function(x) { .global$print=FALSE; x }
    # Suppress print when returns ok not on error, bug #2376.
    # Thanks to: https://stackoverflow.com/a/13606880/403310
    # All appropriate returns following this point are
    # wrapped i.e. return(suppPrint(x)).
}

Essential that’s just saying: if DT[,x:=y] is running at the prompt, then I know the REPL is going to call the print method on my result, beyond my control. Ok, so given print method is going to run, I’m going to suppress it inside that print method by setting a flag (since the print method that runs (i.e. print.data.table) is something I can control).

In knitr‘s case it’s simulating the REPL in a clever way. It isn’t really a script, iiuc, otherwise DT[,x:=y] wouldn’t print anyway for that reason. But because it’s simulating REPL via an eval there is an extra level of eval depth for code run from knitr. Or something similar (I don’t know knitr).

Which is why I’m thinking increasing the depthtrigger might do the trick.

Hacky/crufty, I agree. But if it works, and you let me know which value works, I can change data.table to be knitr aware and change the depthtrigger automatically. Or any better solutions are most welcome.

More Related Contents:

  1. Overlap join with start and end positions
  2. Why does data.table update names(DT) by reference, even if I assign to another variable?
  3. How to get week numbers from dates?
  4. Filtering out duplicated/non-unique rows in data.table
  5. Which data.table syntax for left join (one column) to prefer
  6. data.table with two string columns of set elements, extract unique rows with each row unsorted
  7. Evaluating both column name and the target value within `j` expression within `data.table`
  8. knitr/rmarkdown/Latex: How to cross-reference figures and tables?
  9. Fill in missing values by group in data.table
  10. dplyr on data.table, am I really using data.table?
  11. Generate Dynamic R Markdown Blocks
  12. data.table and parallel computing
  13. Elegantly assigning multiple columns in data.table with lapply()
  14. Sort a data.table fast by Ascending/Descending order
  15. Align multiple tables side by side
  16. Join R data.tables where key values are not exactly equal–combine rows with closest times
  17. how to show a legend on dual y-axis ggplot
  18. Code chunk font size in Rmarkdown with knitr and latex
  19. Rolling Sum by Another Variable in R
  20. Non-equi join using data.table: column missing from the output
  21. Create a loop that includes both a code chunk and text
  22. Use of lapply .SD in data.table R
  23. Cleaning `Inf` values from an R dataframe
  24. can the value.var in dcast be a list or have multiple value variables?
  25. grouped operations that result in length not equal to 1 or length of group in dplyr
  26. `dcast` with empty RHS [duplicate]
  27. Plot size and resolution with R markdown, knitr, pandoc, beamer
  28. Any way to force fread() of data.table not to stop on empty lines?
  29. Index unique values in data.table
  30. R fast single item lookup from list vs data.table vs hash

Leave a Comment