Questions tagged [scala]

Scala is a general-purpose programming language principally targeting the Java Virtual Machine. Designed to express common programming patterns in a concise, elegant, and type-safe way, it fuses both imperative and functional programming styles. Its key features are: an advanced static type system with type inference; function types; pattern-matching; implicit parameters and conversions; operator overloading; full interoperability with Java; concurrency

0
votes
0answers
13 views

How to define a type infinite callable function?

How to define a type InfiniteFunction, which is a function and when called returns another InfiniteFunction The type looks like: () => () => () => ... // infinite or recursive: type ...
0
votes
2answers
19 views

Need a Regex that includes all char after expression

I am trying to figure out a regex. That includes all characters after it but if another patterns occurs it does not overlap This is my current regex [a-zA-Z]{2}\d{1}\s?\w? The pattern is always 2 ...
-1
votes
1answer
16 views

Iterate boolean comparison over two DataFrames?

I have created two DF sets, one with a generic number list and another with a specific number list. I want to iterate over the first list and compare it to the second list; if GenericList[X] is equal ...
-1
votes
0answers
15 views

Storing, accessing and Modifying custom Object in RDD

I have just started learning Spark+Scala and this is my first attempt to develop scala application. I have created a class which holds few variables; values of which will be initialized randomly. I am ...
-1
votes
1answer
17 views

How to create an interface using scala

I am developing a spark application in eclipse ide with scala language and with maven dependencies. I would like to know: Is it possible to create an interface that has a text area and a button with ...
0
votes
2answers
27 views

Transforming a scala list based on condition

I have a sorted list of tuples (sorted by last element which is linenumber) val x = List( ("taskENTER_CRITICAL", 1443), ("taskEXIT_CRITICAL", 1492), ("...
3
votes
2answers
41 views

Ambiguous overload even one is more specific

I have two definition of foo, and one of them is more supposedly specific def foo(f: java.util.function.ToIntFunction[String]) = println("foo1") def foo[T](f: String=>T) = println("foo2") //def ...
0
votes
0answers
27 views

Why IntelliJ does not recognize circe library?

My library dependencies looks as follows: lazy val root = (project in file(".")) .enablePlugins(DockerPlugin, JavaServerAppPackaging, GitVersioning) .settings( name := "plugger", ...
3
votes
1answer
39 views

Stop for-comprehension mid-flow when using stacked monads of State and IO

In this Scala example I need to stop when the result is StopNow, I need to do this after calling decisionStep. How can I do that? case class BusinessState() trait BusinessResult case object ...
0
votes
0answers
40 views

java.sql.SQLException: Unrecognized SQL type -102 while connecting to Oracle Database from Apache Spark

I'm trying to load a remote Oracle Database table on to Apache Spark shell. This is how I started the spark-shell. ./spark-shell --driver-class-path ../jars/ojdbc6.jar --jars ../jars/ojdbc6.jar --...
2
votes
1answer
35 views

How to parameterize table name in Slick

class MyTable(tag: Tag) extends Table[MyEntity](tag, "1970Table") { def id = column[Int]("id") override def * = ( id ) <> (MyEntity.tupled, MyEntity.unapply) } val myTable = ...
2
votes
1answer
53 views

Processing sequence with duplicates concurrently

Suppose I've got a function fab: A => B , a sequence of A and need to get a sequence of pairs (A, B) like this: def foo(fab: A => B, as: Seq[A]): Seq[(A, B)] = as.zip(as.map(fab)) Now I want ...
0
votes
2answers
53 views

How to split a list by Regex

I have a very large amount of list and I need to split by the last occurrence of a 3char sequence. EX(AE3,SH7,SH8). How would I split to make all the characters following it a separate list? I've ...
0
votes
2answers
32 views

How to assign variable in pattern matching

I have a method that takes a year (int) as a parameter. Depending on the passed in year, I need to assign a variable and then use that variable in a query: def findByYear (year: Int): Future[Int] = { ...
0
votes
0answers
25 views

The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached

I try to analyze a dataset of 500Mb in Databricks. These data are stored in Excel file. The first thing that I did was to install Spark Excel package com.crealytics.spark.excel from Maven (last ...
2
votes
2answers
40 views

Convert String expression to actual working instance expression

I am trying to convert an expression in Scala that is saved in database as String back to working code. I have tried Reflect Toolbox, Groovy, etc. But I can't seem to achieve what I require. Here'...
2
votes
2answers
54 views

How To iterate List[Result] and return one Boolean/Result?

I have a Future[List[Result]] I need to run a logic for example if all items in the list are Result.Ok then return Result.Ok (or true), else return Result.BadRequest (or false) I've tried: ...
-6
votes
0answers
45 views

Difference between MutableList and Array

What is the difference between Array and MutableList in Scala? Which one is faster? Which one is closer to List(scala.collection.immutable) with respect to performance? When would you use one vs the ...
1
vote
1answer
28 views

Akka: Serializable Props

Base on Akka documentation: If you want to verify that your Props are serializable you can enable the following config option: akka { actor { serialize-creators = on } } My question is ...
0
votes
1answer
21 views

Spark Sql sort operation is not working properly

I am trying to sort a dataframe with sort function but it dosen't sort properly and seems it is sorting in chunks. Maybe it is sorting the individual partitions and not combing the sort.However where ...
0
votes
1answer
17 views

Running a spark structured streaming application scala code written in a file using nohup

I have a spark structured streaming scala code written to run in a batch mode . I am trying to run it using nohup spark2-shell -i /home/sandeep/spark_test.scala --master yarn --deploy-mode client ...
2
votes
1answer
33 views

How to decode missing json array as empty List with circe

For example, we have some case class case class Foo(a: Int, b: List[String]) And we want to deserialize instance of Foo from json {"a": 1} replacing missing b array with Nil We can create custom ...
0
votes
1answer
28 views

Play Framework 2.7.x Compile Time DI Error

Create project from https://github.com/playframework/play-socket.io/tree/master/samples/scala/chat and get compilation error overriding method applicationLifecycle in trait MyApplication of type =>...
-3
votes
1answer
42 views

How to find the nearest date to the current date

I am working on a problem where I want to display nearest date/time from the current date in Scala. I have a data like below: FlightTo Date -FT1 17-06-2019 -FT2 19-06-2019 -FT3 23-...
2
votes
1answer
49 views

Case class immutable still able to change the parameter values

I reviewed some code from a colleague and I came across a case class which is by default immutable. the below case class can be changed so my question is how is this possible since case classes are ...
1
vote
1answer
20 views

How to parse a Json array within a field while only getting certain fields of the object

I'm having some trouble using circe for a more complex extraction. If I have the below Json: { "data": [ {"a": "a-string", "b": "a-string", "c": "a-string"}, {"a": "a-string", "b": "a-...
3
votes
1answer
47 views

What happens with values that are not returned or assigned?

I've encounter a function: def open(partitionId: Long,version: Long): Boolean = { Class.forName("com.mysql.jdbc.Driver") connection = DriverManager.getConnection(url, user, pwd) statement ...
0
votes
0answers
31 views

What is the best way to read from existing Elasticsearch instances in Pyspark preventing mapping/parsing errors?

I have a lot of Elasticsearch indices with many fields that I want to use in Spark. As those indices don't have any mapping specified, using the data gets multiple parsing errors. What is the best way ...
-3
votes
1answer
33 views

Spark: Error while populating counts separately [on hold]

I have a file having 3 columns i.e. userID, productID, action where action would be one of these - browse,click,addtocart,purchase,logout A product is considered as viewed if the action contains "...
0
votes
1answer
35 views

Spark SQL join really lazy?

I'm performing an inner join between, say, 8 dataframes, all coming from the same parent. Sample code: // read parquet val readDF = session.read.parquet(...) // multiple expensive transformations ...
3
votes
2answers
56 views

How to stack the State and IO monads

In a game of Connect4: we start with an empty grid two players place pieces x and o on the grid the first player to achieve 4 pieces in a line wins! this is a text based console game My thinking is ...
3
votes
0answers
44 views

Doobie: create constant table with values

Postgres allows creation of constant tables for use in queries with this syntax: WITH names (id, name) AS (VALUES (1, 'ABC'), (2, 'BCD')) SELECT id FROM names or the inlined version: SELECT id ...
0
votes
2answers
50 views

Why do I have to pass new keyword?

I have the following code: val fsm = TestFSMRef(new SenderCollectorFsm) And do not understand, why do I have to pass to TestFSMRef an instance. Lets look at the definition of TestFSMRef: object ...
-2
votes
0answers
15 views

Unable to load class in scala shell when using URLClassLoader

I embedded scala shell in my app, and I create scala shell via URLClassLoader (This URLClassLoader will load my custom defined class and bind it into scala shell). The weird issue I meet is that I can ...
-1
votes
0answers
54 views

How to replace if with higher order method

In Scala, can i replace if statement with some higher order method when making comparison between to variables. Code explaining question is def fun(num: Double ): Unit ={ if(num < math.random){ ...
5
votes
3answers
85 views

Eliminating identity wrapper types from Scala APIs

Suppose I am trying to "abstract over execution": import scala.language.higherKinds class Operator[W[_]]( f : Int => W[Int] ) { def operate( i : Int ) : W[Int] = f(i) } Now I can define an ...
-1
votes
2answers
28 views

How to clear terminal/screen in scala

I need to clear console screen in Scala I've tried standard ANSI Clear screen which was suggested as "Terminal control/Clear the screen" by rosettacode.org here object Cls extends App {print("\033[...
0
votes
1answer
64 views

Null pointer exception occured only when running the jar file - Scala Spark

I have a Big-data analysis project. I used Spark for it and wrote using Scala. When I ran the project using sbt run it worked perfectly and gave the results what I want. After that I built the jar ...
-2
votes
1answer
36 views

Where is StateT in Cats?

I have in my sbt: "org.typelevel" %% "cats-core" % "2.0.0-M1", "org.typelevel" %% "cats-effect" % "1.3.1", "org.typelevel" %% "cats-free" % "2.0.0-M3" But can't import StateT in code. In what ...
1
vote
1answer
55 views

Cannot write a stream into a parquet sink

I work on Databricks, a data processing platform based on Spark with an HDFS-like file system, so basically I beleive those of you who are familiar with Spark and HDFS will be able to help me without ...
0
votes
0answers
18 views

Scalaz |-> counterpart in Cats

I noticed that the Scalaz |-> operator is not implemented in Cats. Is there a function offering similar semantics?
1
vote
2answers
71 views

How to use literal type in Scala 2.13

I'm trying Literal Types from Scala 2.13 and I encounter the following error : scala> def double[A <: Singleton] = valueOf[A] ^ error: No ...
0
votes
0answers
19 views

Play framework2.4 can not add key store value to .scala build sbt file

I'am trying to upgrade to play framework 2.4 from play 2.3, but when refreshing SBT I got the following error: build/*:resourceDirectory from build/*:privateKeystore ((project.Build).scala:10) [error]...
0
votes
1answer
18 views

Force alpakka kafka consumer show error message on deserialization error

Alpakka kafka consumer processes records until encounters record which it fails to deserialize and silently dies without leaving error message. How to force it to report error message?
0
votes
1answer
27 views

How to sort column of means with Spark SQL?

I am finding mean values by using groupBy and mean functions, like this. x.groupBy("Col1", "Col2" "Col3").mean("Col4") However, I want to later also sort the column with means. How can I do that? I ...
1
vote
0answers
24 views

How to trace class field accesses of certain classes only in Scala?

I would like to use variable tracer for Scala programs which traces reads and writes of certain class fields. For example, if I have a custom actor class like: class Server extends Actor { var ...
2
votes
2answers
34 views

Conditional filtering of JSON before deserialisation to case class model

How to parse a JSON conditionally before deserialisation to the following case class: case class UserInfo(id: String, startDate: String, endDate: String) I have an implicit reads object UserInfo { ...
0
votes
1answer
17 views

View make request to update with scala play framework [on hold]

I have a timer that goes off every 60 seconds to insert new data into my database, and I display that data in an HTML table in my view, however, new data keeps coming in the database yet my view does ...
1
vote
1answer
39 views

How to read scala documentation using reflection

Is there anyway we can read scala doc comments using reflection. My requirement is to read the @group tag value and use it for counting how many functions are there for each group
-1
votes
0answers
46 views

Can anyone tell me why we are using case int the below code [on hold]

I have some data and code in Scala to process the data. But I am not able to understand the code. Thus, can someone please explain me the following piece of code. I have tried the code, I got the ...