Players sometimes not reconnecting

[PatatjeMC]

Sometimes players don't reconnect but get stuck with "Server is restarting... Please wait..." on their screen. When looking in console with debug mode on I can see that it tries to ping the server a few times and then spits out the following error:

[14:29:05 WARN] [io.netty.util.concurrent.DefaultPromise]: An exception was thrown by com.velocitypowered.proxy.server.VelocityRegisteredServer$$Lambda/0x00007f5ec8faebb0.operationComplete()
java.lang.NullPointerException: Cannot invoke "com.velocitypowered.proxy.connection.MinecraftConnection.setActiveSessionHandler(com.velocitypowered.proxy.protocol.StateRegistry, com.velocitypowered.proxy.connection.MinecraftSessionHandler)" because "conn" is null
        at com.velocitypowered.proxy.server.VelocityRegisteredServer.lambda$ping$0(VelocityRegisteredServer.java:133) ~[proxy.jar:3.4.0-SNAPSHOT (git-f986eb51-b472)]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[proxy.jar:3.4.0-SNAPSHOT (git-f986eb51-b472)]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[proxy.jar:3.4.0-SNAPSHOT (git-f986eb51-b472)]
        at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35) ~[proxy.jar:3.4.0-SNAPSHOT (git-f986eb51-b472)]
        at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:503) ~[proxy.jar:3.4.0-SNAPSHOT (git-f986eb51-b472)]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[proxy.jar:3.4.0-SNAPSHOT (git-f986eb51-b472)]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[proxy.jar:3.4.0-SNAPSHOT (git-f986eb51-b472)]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[proxy.jar:3.4.0-SNAPSHOT (git-f986eb51-b472)]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405) ~[proxy.jar:3.4.0-SNAPSHOT (git-f986eb51-b472)]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[proxy.jar:3.4.0-SNAPSHOT (git-f986eb51-b472)]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[proxy.jar:3.4.0-SNAPSHOT (git-f986eb51-b472)]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[proxy.jar:3.4.0-SNAPSHOT (git-f986eb51-b472)]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

The amount of ping attempts before this happens varies, the error also shows up before the server has finished restarting. If you need any more information, let me know.