KEYDB.CRON: schedule Lua scripts with your database

January 25, 2021 · 8 min read

Ben Schermel

Introducing the power of Cron to your database#

When you're setting up your database, Cron is often not far away... so for low latency calls, why not have the database handle this?

We recently introduced this tool with the KEYDB.CRON command that enables the user to execute Lua scripts via a scheduler at a specific time and/or interval. This functionality persists and is locally stored, bringing the call into the database instead of through a 3rd party tool.

For those who don't know, KeyDB is a faster, multithreaded alternative to Redis. We strive to introduce powerful tools & functionality that reduces complexity for our users. KEYDB.CRON is one of many features KeyDB has developed.

Should Cron be run by the database?#

When setting up a database server, you want to minimize the amount of processes to monitor and maintain - KEYDB.CRON helps do just that. Having to set up Cron externally is yet another tool that needs to be configured and validated. Clocks need to be synchronized and time zones set. Having a built-in tool whose functionality is automatically passed to new replica nodes and migrated with the database reduces potential for error and simplifies setup.

That being said a long debate could be had about running Lua scripts from within KeyDB/Redis and where such code should reside. If you are running long or unpredictable scripts on this blocking call it can become problematic and effect performance. However this argument is more for 'when to use Lua', not necessarily if executing these tasks with an internal scheduler is a good idea. These arguments often boil down to engineering, but an additional tool used in the right way can have a lot of benefits including avoiding the latency and network hit of interacting with a remote client.

KEYDB.CRON stores Lua scripts as a KEY and persists to the database so it's there if the server reboots, a replica node is made, or your data is migrated. The command schedules based off Unix time for consistency and can schedule single tasks or recurring tasks down to the millisecond.

When should I use Cron with my database?#

If you have ever used Lua with KeyDB or Redis, KEYDB.CRON may be a powerful tool at your disposal. A lot of scripts are triggered by an event or request, however there are a lot of tasks that need to be performed regularly or at a specific time/interval. Using KEYDB.CRON brings the scheduler into the database instead of on the client side or as a 3rd party scheduler. The function of KEYDB.CRON is similar to using EVAL, however with the additional functionality of a built in scheduler.

You may run recurring tasks down to the millisecond or execute a one time scheduled task against your database, however what you do will likely be very specific to your use case. Some people use Lua scripts + Cron to do things such as:

scheduling expires, batch cleanup & garbage collection
regular maintenance and housekeeping of the database
maintaining/resetting timers, counters
popping items off lists, checking for timeouts, etc.
part of task queues
data processing, consolidation
performing tasks such as application specific eviction
ensure a one time task executes at an exact time
... and whatever else may be applicable (we would love to hear your use case: https://community.keydb.dev)

Using the KEYDB.CRON command:#

You can refer to our documentation here as well for reference on using the command.

The premise is quite simple:

KEYDB.CRON name [single/repeat] [optional: start] delay script numkeys [key N] [arg N]

Where:

name is the name of the KEY. This will be visible in the keyspace, can be searched, and deleted with DEL. Each Cron task will have its own name.
[single/repeat] specifies if the script will run only once, or if it will be repeated at the specified interval
[optional: start] is an integer specified in milliseconds since Epoch. If specified, the script will not execute until this Unix time has been reached. If the delay is greater than zero, this delay time will need to elapse prior to the script executing (timer begins to elapse at start time). If a start time is specified, the delay will always remain in reference intervals to that start time.
delay is an integer specified in milliseconds used as the initial delay. If repeat is specified, this will also be the length of the repeating timer which will execute the script each time the delay elapses (will continue to execute indefinitely).
script is the Lua script to execute. This should be the Lua script itself and NOT the SHA1 digest of a loaded script (not yet supported).
numkeys [key N] [arg N] are the number of keys, keys, and arguments for the script, similar to usage with EVAL

Persistence:#

Unlike traditional Lua scripts that may be loaded (cached only), the KEYDB.CRON task will persist across server boots if saved. It can be seen as a KEY in the keyspace and deleted or modified.

Examples:#

keydb-cli> KEYDB.CRON mytestname REPEAT 1610941618000 60000 "redis.call('INCR',KEYS[1])" 1 mytestcounter
OK

The example above increments the value of 'mytestcounter" every 60 seconds starting at the Unix Timestamp of 1610941618000 milliseconds.

keydb-cli> KEYDB.CRON mytestname2 SINGLE 1610941618000 1 "redis.call('set',KEYS[1],'0')" 1 mytestcounter
OK

The above command sets "mytestcounter" to zero at Unix Timestamp 1610941618000 milliseconds. This will occur only once. Note that we must specify a delay time. Once the task has completed the KEY will be removed. and the name "mytestname2" will no longer exist.

If you want to delete the Cron task, simply delete the KEY which is the name of your Cron task and that functionality will no longer exist.

keydb-cli> DEL mytestname
(integer) 1

Loading Script from file#

For more lengthy tasks, we may want to call an existing Lua script with specified inputs.

Example#

Lets start by creating a simple Lua script, myscript.lua. This script will look at the bottom 3 ranking members of a set when the set is greater than 100. In this example we want to remove any members if they have been seen in the bottom 3 rankings for more than 10 counts. As such we will keep tabs on any low ranking members (also for client screening). Once a member is removed we will expire its counter in 24 hours to reset access.

myscript.lua#

if redis.call("ZCARD", KEYS[1]) > 100 then
        --find lowest 3 ranking members
        local lowest_three = redis.call("ZRANGE", KEYS[1], "0", "2")
        for i in ipairs(lowest_three)
        do
                --if in the bottom 3, increment the members specific counter
                local counter = "low_count:" .. lowest_three[i]
                local new_count = redis.call("INCR", counter)
                --expire the counter in 24 hours
                redis.call("EXPIRE", counter, "86400")
                if new_count >= 10 then
                        redis.call("ZREM", KEYS[1], lowest_three[i])
                end
        end
end

We now want to run this script say every minute to probe for the low ranking members. Lets call the Cron task "mycrontask" and pass it the name of the sorted set, "rankings", as will be referred by our script as KEY[1]

Now we load the Cron task with the Lua script through the client per below command:

keydb-cli -p 6379 KEYDB.CRON mycrontask repeat 60000 "$(cat myscript.lua)" 1 rankings

Important considerations:#

Below are some considerations when using KEYDB.CRON:

Lua scripts are blocking, so it's important to make sure they are not waiting on anything, nor are they long or error prone. It is ideal to keep execution sub-millisecond.
Depending on your use and application of Lua and Cron, you should run benchmark tests to ensure there is no adverse effect on performance
There is no simple way of identifying which stored keys are scripts and the definition of those scripts, so it is important to label them well and document.

Can I use Cron with Redis?#

This functionality is not part of Redis. However KeyDB is a drop in alternative to Redis, fully compatible with Redis modules, API, and protocol. This includes Lua scripts, and Redis clients. KeyDB can be thought of as a superset of functionality, adding in additional features and configuration options, as well as commands. Because of KeyDB's multithreaded architecture, we are able to provide powerful tools that might otherwise be too heavy for a single threaded application.

Keep up to date with KEYDB#

We have some really cool features in the works. To keep up to date with what we are doing follow us on one of the following channels:

Recent posts