Mostly because it can be more efficient — values don’t need to be computed if they’re not going to be used. For example, I may pass three values into a function, but depending on the sequence of conditional expressions, only a subset may actually be used. In a language like C, all three values would … Read more
The object returned by range() (or xrange() in Python2.x) is known as a lazy iterable. Instead of storing the entire range, [0,1,2,..,9], in memory, the generator stores a definition for (i=0; i<10; i+=1) and computes the next value only when needed (AKA lazy-evaluation). Essentially, a generator allows you to return a list like structure, but … Read more
One (arguably more “proper”) way to approach this would be to write your own iterator for iterators that @BenBolker suggested (pdf on writing extensions is here). Lacking something more formal, here is a poor-man’s iterator, similar to expand.grid but manually-advancing. (Note: this will suffice given that the computation on each iteration is “more expensive” than … Read more
As for fusion. Let’s imagine here’s a map operation: .map(x -> x.squash()) It’s stateless and it just transforms any input according to the specified algorithm (in our case squashes them). Now the filter operation: .filter(x -> x.getColor() != YELLOW) It’s also stateless and it just removes some elements (in our case yellow ones). Now let’s … Read more
Here’s a lazy-list Hamming numbers in Prolog using a “generator idiom”. The more I think about it the more it seems to me the lazy lists of Haskell are just open-ended (a.k.a. “difference”) lists of Prolog, and corecursion just a fancy name for the top-down list building of tail recursion modulo cons. Also, Haskell is … Read more
From the Scala docs: Note: the difference between c filter p and c withFilter p is that the former creates a new collection, whereas the latter only restricts the domain of subsequent map, flatMap, foreach, and withFilter operations. So filter will take the original collection and produce a new collection, but withFilter will non-strictly (i.e. … Read more
It’s a strictness declaration. Basically, it means that it must be evaluated to what’s called “weak head normal form” when the data structure value is created. Let’s look at an example, so that we can see just what this means: data Foo = Foo Int Int !Int !(Maybe Int) f = Foo (2+2) (3+3) (4+4) … Read more
I Couldn’t get the above to work, but here is my implementation, i’ve tested it on c:\users on a “Win7” box, because if has all these “nasty” dirs: SafeWalk.EnumerateFiles(@”C:\users”, “*.jpg”, SearchOption.AllDirectories).Take(10) Class: public static class SafeWalk { public static IEnumerable<string> EnumerateFiles(string path, string searchPattern, SearchOption searchOpt) { try { var dirFiles = Enumerable.Empty<string>(); if(searchOpt == … Read more
I’m wondering if it is possible to implement lazy evaluation in C++ in a reasonable manner. If yes, how would you do it? Yes, this is possible and quite often done, e.g. for matrix calculations. The main mechanism to facilitate this is operator overloading. Consider the case of matrix addition. The signature of the function … Read more
For transformations, Spark adds them to a DAG of computation and only when driver requests some data, does this DAG actually gets executed. One advantage of this is that Spark can make many optimization decisions after it had a chance to look at the DAG in entirety. This would not be possible if it executed … Read more