Kotlin Scripting

Written on by Alexander Lindner

Table of Contents

Introduction

Shell scripts, predominantly written in Bash, are the de facto standard for server maintenance, executing continuous integration/continuous deployment (CI/CD) pipelines, automating build processes, and similar tasks. Nearly every Linux-based server includes a shell environment, which allows the execution of shell scripts, with a few exceptions such as certain Docker base images (like distroless).

One of Bash’s most advantageous features is its ability to chain commands using pipes. However, this functionality can sometimes result in complex and difficult-to-maintain command sequences. For example:

ps aux | grep -v "^USER" | sort -rk 3,3 | awk '{printf "%-10s %-8s %-8s %-5s\n", $1, $2, $3, $11}' | column -t | head -n 10

(See explainshell for an explanation).

While Bash provides powerful tools for process control, it lacks some of the basic functionality found in modern programming languages, such as more intuitive string manipulation, regular expression support, and advanced control structures like foreach, for, do-while, and so on. These features often require non-standard workarounds to implement. Additionally, since a shell script’s capabilities are directly tied to the availability of external tools, managing dependencies (e.g., the need for jq for JSON processing) can become cumbersome. I haven’t yet discussed topics such as parallel execution, cross-platform compatibility, easy logging, and similar aspects.

For many users, python is seen as a viable alternative, offering the flexibility of a scripting language while being more powerful and easier to learn than shell scripting. Also, it provides a huge number of packages that can be easily installed. However, Python presents challenges in managing versions and dependencies, often leading to conflicts or complicated setup processes (python 2 vs 3 or pip fuck-up).

Moreover, Python’s a dynamic typed language and I prefer a language with stronger, more explicit typing support.

my first attempt: java

Some years ago, I experimented with java for a replacement of bash scripts

#!/usr/bin/env -S java --source 11
public class script {
    public static void main(String[] args) {
        System.out.println("Hello, World!");
    }
}

try it using

cat <<EOF >>myJvmScript
#!/usr/bin/env -S java --source 11
public class script {
    public static void main(String[] args) {
        System.out.println("Hello, World!");
    }
}
EOF
chmod +x myJvmScript && ./myJvmScript

This shebang magic works quite well, and one can play with #!/usr/bin/env -S java -jar to execute a full jar file, nevertheless, it is very limited.

First of all, Java without external dependencies, is often impractical for many tasks. While the --classpath argument allows the inclusion of external libraries, it necessitates the prior manual downloading of these dependencies. Attempting to handle this dynamically within a script introduces significant complexity, resulting in convoluted code that undermines the primary goal of providing an “easy-to-use” solution.

Additionally, this approach lacks the ability to natively split script-like code across multiple files without creating a formal project structure, which further complicates its use in simple scenarios. Java’s structure requires more rigid organization, making it cumbersome for smaller or less formal tasks, where quick and easy division of code is essential for maintainability and clarity.

Moreover, Java’s syntax is far from simple and intuitive. Its verbosity and complexity, especially for tasks that could be handled more simply in scripting languages, create a barrier to efficiency and readability, making it a less desirable option for lightweight or rapid development tasks.

Current experimentation: Kotlin

At work, where we primarily use Kotlin, I created a small script to gather information across all our microservice repositories. Initially, I placed the script in an IntelliJ project without committing it. However, I decided to share the script by adding it to our scripts repository, which mainly contains Bash scripts. This led me to look for a simple way to execute the Kotlin file directly. That’s when I discovered Kotlin Script (Kotlin’s .kts files).

Kotlin Script allows you to use file annotations that are evaluated before execution to resolve dependencies and include other files. With Kotlin’s concise and elegant syntax, many of Java’s shortcomings are effectively addressed. Additionally, Kotlin Script caches the compilation of scripts, improving resource efficiency and execution speed.

Structure

The structure of a Kotlin Script file is similar to a regular Kotlin file but without the need for a fun main() function. Instead, the file is typically named in the format NAME.main.kts.

@file:DependsOn("org.apache.commons:commons-compress:1.24.0")

import java.nio.file.Files
import java.nio.file.Paths
import java.util.zip.ZipEntry
import java.util.zip.ZipOutputStream

fun createZip(
    outputFilePath: String,
    sourceDir: String
) {
    ZipOutputStream(BufferedOutputStream(FileOutputStream(outputFilePath))).use { zipOut ->
        val sourcePath = Paths.get(sourceDir)
        Files
            .walk(sourcePath)
            .filter { path -> Files.isRegularFile(path) }
            .forEach { path ->
                val zipEntry = ZipEntry(sourcePath.relativize(path).toString())
                zipOut.putNextEntry(zipEntry)
                Files.copy(path, zipOut)
                zipOut.closeEntry()
            }
    }
}

createZip("test.zip", "~/test")
println("hello world")

Don’t forget to make the file executable (e.g., using chmod +x NAME.main.kts) and then run it directly, like this:

./script.main.kts

Annotations Summary

1. @file:DependsOn

@file:DependsOn("org.jetbrains.kotlin:kotlin-stdlib:1.5.31")

2. @file:Repository

@file:Repository("https://maven.example.com/repository")

3. @file:Import

@file:Import("otherScript.kts")

4. @file:CompilerOptions

@file:CompilerOptions("-opt-in=kotlin.Experimental")

Deep dive: @file:Import vs @file:DependsOn

The @file:Import annotation allows you to include uncompiled files in your Kotlin Script, whereas @file:DependsOn is used to add compiled dependencies.

As of IntelliJ IDEA 2024.3, only files added via @file:DependsOn are properly indexed. Despite this, scripts using @file:Import run without any issues.

includeMe.kts 
#!/usr/bin/env kotlin
val name = "world"
test.main.kts 
#!/usr/bin/env kotlin
@file:Import("includeMe.kts")
println("Hello $name")

In IntelliJ IDEA, however, this results in unresolved variables and similar issues, as shown in the example image: intellij error showcase

This essentially undermines the primary advantage of using an IDE. While this might be acceptable if you’re using a text editor like Vim or Nano, for me, it’s a significant problem.

This brings us to a challenge: if you want to share code between multiple scripts, you would need to create a JAR file and publish it to a Maven repository. Fortunately, there’s some good news: the @file:DependsOn annotation also supports specifying a local path to a JAR file. For example:

#!/usr/bin/env kotlin

@file:DependsOn("/home/alindner/projects/alexander-lindner/kscript-std/build/libs/kscript-std-1.0.0.jar")

import io.github.oshai.kotlinlogging.KotlinLogging.logger

val logger = logger {}
logger.info { "Hello world!" }

kotlin script std lib

When we put all the pieces together, we now have a powerful programming language with modern syntax, the ability to easily import dependencies through Maven, and the option to load local .kts and .jar files. This allows us to fully leverage the extensive Java and Kotlin ecosystem.

However, one missing piece is that Kotlin, Java, and their ecosystems are not inherently designed for scripting purposes. For example, to recursively list all .zip files, you would need to write something like this:

#!/usr/bin/env kotlin

import java.io.File

File("/home")
    .walkTopDown()
    .filter { it.path.endsWith(".zip") }
    .forEach { println(it) }

That’s great, but downloading a file from an HTTPS website in Kotlin is still quite complicated compared to using a simple curl command in Bash. Similarly, there are many examples where straightforward Bash commands or tools become far more complex when implemented in Kotlin.

This highlights the need for a standard library tailored to Kotlin scripting. To address this, I created a prototype library that serves as both a utility for Kotlin scripting and a collection of elegant, easy-to-use code snippets.

#!/usr/bin/env kotlin

@file:DependsOn("/home/alindner/projects/alexander-lindner/kscript-std/build/libs/kscript-std-0.0.1.jar")

import io.github.oshai.kotlinlogging.KotlinLogging.logger
import org.alindner.kscript.std.*
import org.alindner.kscript.std.Algorithm.TAR_GZ
import org.alindner.kscript.std.Options.FILES_ONLY
import org.alindner.kscript.std.Options.RECURSIVE


val logger = logger {}

fun createBackup() {
    logger.info { "Executing backup command..." }
    createDirectory("/mnt/backup/40_homeserver/")
    compress("/mnt/backup/40_homeserver/k8s.${rightNow()}.tar.gz", "~/k8s", TAR_GZ)
    list("/mnt/backup/40_homeserver/", FILES_ONLY, RECURSIVE)
        .allBut(5)
        .onEach(logger::info)
        .delete()
    logger.info { "Backup created." }
}

fun remoteBackup() {
    val config = SSHConfig(
        username = "backupuser",
        host = "HOST",
        privateKeyPath = "/home/backupuser/.ssh/id_rsa"
    )
    val backupDir = "/opt/server/backups/${rightNow()}/"
    val localDir = "/mnt/backup/20_HOST/${rightNow()}/"
    config.useSSH { ssh ->
        ssh.createDirectories(backupDir)
        ssh.compress("${backupDir}data.tar.gz", "/opt/server/docker", TAR_GZ)
        ssh.exec("cd /opt/server/docker && git bundle create ${backupDir}git.bundle --all")
        ssh.list(backupDir, FILES_ONLY, RECURSIVE)
            .moveTo(localDir)
            .onEach { logger.info { "Path: ${it.path}" } }
    }
}

parseArguments(args)
    .arg("backup", ::createBackup)
    .arg("remote-backup", ::remoteBackup)
    .default { availableCommands -> logger.info { "No Command was provided. Available commands: $availableCommands" } }

finish("Backup finished")

This library aims to bridge the gap between the simplicity of Bash and the power of Kotlin for scripting.

Conclusion

I am now able to write Kotlin scripts effortlessly, and they have the potential to replace many of my existing Bash scripts. While it’s true that a functional Kotlin/Java environment is required, for me, this is only a minor disadvantage.

Moving forward, I plan to replace some of my current scripts with Kotlin scripts and observe how this change impacts my workflow.