s3d - no more hosts available

[stealthx]

Testing the rclone s3d connected to sia.storage.

Files test-2000MB.bin and test-3000MB.bin were uploaded without problem.

Generating 100 x 5MB files resulted in the error message : No more hosts available.

ERROR backend.sia failed to finalize upload {"error": "failed to upload slab: shard upload failed: no more hosts available"}

WARN sdk failed to refresh hosts {"error": "failed to fetch hosts from indexer: Get \"https://sia.storage/hosts?limit=100&offset=0&sc=...\": dial tcp: lookup sia.storage on 127.0.0.11:53: server misbehaving"}

[Didrole]

Your logs indicate that a DNS resolution issue made the request to get a hosts list from the indexer fail.
Maybe be the DNS server you're using had an issue.
Is it always reproducible?

[chris124567]

Thank you for bringing this to our attention. I think this might have to do with failing to fetch the host list as the above commenter said. 1 failure should not be a big deal because if I remember correctly we only update the list when it succeeds and it's refreshed every 10 minutes. But there could be a bug somewhere so I will take a look and also see if I can reproduce this.

[chris124567]

I was able to reproduce this. We have an open PR in the Go SDK (which s3d depends on) that fixes this in my testing: SiaFoundation/sia-storage-go#42

I will update here when it is merged and when we release new versions. Thank you again

[stealthx]

Good to see you have a fix.

I deleted Docker and started over from scratch.

I have succesfully uploaded :
test-30MB.bin
test-40MB.bin
test-50MB.bin
folder with 10 x 5MB files

When I tried with a folder containing 20 x 5 MB files, I encountered the again.

[chris124567]

We need to review and merge the PR then release a new version to fix the issue for end users. So it may be a few days

[chris124567]

This should be fixed in v0.1.2

[stealthx]

S3d Version 0.1.2, Commit: 9004737

It looks like I still have No more hosts available error:

[chris124567]

Is this from uploading 100 x 5MB files again?

I couldn't reproduce this on 0.1.2 but I will try again tomorrow

[chris124567]

OK, I was able to reproduce this if I introduced a decent amount of network instability. There are a few changes I can think of for sia-storage-go to make this less likely to happen so I will do that this week.