It’s easy to forget that a trait may extend a class. If you use a trait, you can postpone the decision of which constructor to call, like this: trait Extended extends Base { … } object Extended { def apply(arg1: Int) = new Base(arg1) with Extended def apply(arg2: String) = new Base(arg2) with Extended def … Read more
The TLDR: .config(“spark.master”, “local”) a list of the options for spark.master in spark 2.2.1 I ended up on this page after trying to run a simple Spark SQL java program in local mode. To do this, I found that I could set spark.master using: SparkSession spark = SparkSession .builder() .appName(“Java Spark SQL basic example”) .config(“spark.master”, … Read more
Scala 2 version of this answer is here: How can I run generated code during script runtime? In Scala 3: For example you can use Li Haoyi‘s Ammonite ammonite.Main(verboseOutput = false).runCode(“””println(“Hello, World!”)”””) // Hello, World! build.sbt scalaVersion := “3.1.3” libraryDependencies += “com.lihaoyi” % “ammonite” % “2.5.4-22-4a9e6989” cross CrossVersion.full excludeDependencies ++= Seq( ExclusionRule(“com.lihaoyi”, “sourcecode_2.13”), ExclusionRule(“com.lihaoyi”, “fansi_2.13”), … Read more
There are just a couple of simple syntax rules. The appendix of the spec is worth perusing. A function literal or anonymous function (6.23) will look like x => Expr or x => Block depending on whether the context is an Expr or a ResultExpr, respectively. A function application (6.6) will look like f(Expr, Expr) … Read more
sc._jsc.hadoopConfiguration().set(‘my.mapreduce.setting’, ‘someVal’) should work
You don’t need any 3rd party library to do this: sealed trait MyTrait case object SubClass1 extends MyTrait case object SubClass2 extends MyTrait import scala.reflect.runtime.{universe => ru} val tpe = ru.typeOf[MyTrait] val clazz = tpe.typeSymbol.asClass // if you want to ensure the type is a sealed trait, // then you can use clazz.isSealed and clazz.isTrait … Read more
Example#Foo is called a type projection and will match any type Foo of any enclosing instance of type Example. If you write a type Example.Foo, the compiler will look for the value (and not type) called Example and will refer to its enclosing Foo type only. This is often used in the context of singleton … Read more
Most probably you’re compiling your code locally with Scala 2.12 but at the server it’s running with Scala 2.13 or 2.11. Try to recompile your code with the version of Scala at the server. Scala 2.11, 2.12, 2.13 are binary incompatible. The signature of refArrayOps is different (in binary incompatible way) in Scala 2.13 def … Read more
It is not covered in 2.2, but can be easily addressed. You can add required Encoder using ExpressionEncoder, either explicitly: import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder import org.apache.spark.sql.Encoder spark .createDataset(Seq(Map(1 -> 2)))(ExpressionEncoder(): Encoder[Map[Int, Int]]) or implicitly: implicit def mapIntIntEncoder: Encoder[Map[Int, Int]] = ExpressionEncoder() spark.createDataset(Seq(Map(1 -> 2)))
The collections library does backflips to accommodate subtyping: when you use map on a specific collection type (list, map, etc.), you’ll (usually) get the same type back. It manages this through the use of an extremely complex inheritance hierarchy together with type classes like CanBuildFrom. It gets the job done (at least arguably), but the … Read more