Why is using update on a lm inside a grouped data.table losing its model data?

by

This is not an answer, but is too long for a comment

The .Environment for the terms component is identical for each resulting model

e1 <- attr(fit[['V1']][[1]]$terms, '.Environment')
e2 <- attr(fit[['V1']][[2]]$terms, '.Environment')
e3 <- attr(fit[['V1']][[3]]$terms, '.Environment')
identical(e1,e2)
## TRUE
identical(e2, e3)
## TRUE

It appears that data.table is using the same bit of memory (my non-technical term) for
each evaluation of j by group (which is efficient). However when update is called, it is using this to refit the model. This will contain the values from the last group.

So, if you fudge this, it will work

fit = DT[, { xx <-list2env(copy(.SD))

             mymodel <-lm(Sepal.Length ~ Sepal.Width + Petal.Length)
             attr(mymodel$terms, '.Environment') <- xx
             list(list(mymodel))}, by= 'Species']





lfit2 <- fit[, list(list(update(V1[[1]], ~.-Sepal.Width))), by = Species]
lfit2[,lapply(V1,nobs)]
V1 V2 V3
1: 41 39 42
# using your exact diagnostic coding.
lfit2[,nobs(V1[[1]]),by = Species]
      Species V1
1:     setosa 41
2: versicolor 39
3:  virginica 42

not a long term solution, but at least a workaround.

More Related Contents:

  1. Select / assign to data.table when variable names are stored in a character vector
  2. Is there a dplyr equivalent to data.table::rleid?
  3. Assign multiple columns using := in data.table, by group
  4. Efficient way to rbind data.frames with different columns
  5. How to replace NA values in a table for selected columns
  6. data.table “key indices” or “group counter”
  7. Select subset of columns in data.table R [duplicate]
  8. How to extract the first n rows per group?
  9. R: data.table cross-join not working
  10. Why is allow.cartesian required at times when when joining data.tables with duplicate keys?
  11. Complex non-equi merge in R
  12. ‘Embedded nul in string’ error when importing csv with fread
  13. eval and quote in data.table
  14. Efficiently computing a linear combination of data.table columns
  15. R fuzzy string match to return specific column based on matched string
  16. When should I use the := operator in data.table?
  17. Subsetting data.table by 2nd column only of a 2 column key, using binary search not vector scan
  18. Using dynamic column names in `data.table`
  19. Conditional binary join and update by reference using the data.table package
  20. predict.lm() with an unknown factor level in test data
  21. Set one or more of coefficients to a specific integer
  22. How do you delete a column by name in data.table?
  23. How does one change the levels of a factor column in a data.table
  24. When should I use setDT() instead of data.table() to create a data.table?
  25. DT[!(x == .)] and DT[x != .] treat NA in x inconsistently
  26. In R data.table, how do I pass variable parameters to an expression?
  27. Row sums over columns with a certain pattern in their name
  28. paste two data.table columns
  29. Removing multiple columns from R data.table with parameter for columns to remove
  30. Fill option for fread

Leave a Comment