Skip to content

Migration from Scala to Python#

Migrating a history Scala project to Python, I find some tips that can help me forget the type system in scala. Feel good!!! 😄

dataclass vs case class#

You have to create a case class for each data model in Scala, whiledataclass is your alternative in python

1
2
3
4
@dataclass()
class Event:
    event_id: int
    event_name: str

Create Dataframe from dataclass#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
spark = (
    SparkSession.builder.master("local[*]")
    .appName("test")
    .getOrCreate()
)
d = [
    Event(1, "abc"),
    Event(2, "ddd"),
]

# Row object
df = spark.createDataFrame(Row(**e.__dict__) for e in d)
df.show()
# +--------+----------+
# |event_id|event_name|
# +--------+----------+
# |       1|       abc|
# |       2|       ddd|
# +--------+----------+