Add optimization where-in each task completion amounts to one shard individual shard write.
Currently, there is one shard file per worker, that gets CREATE only once (RPC), now there is local copy - even then there is OpenCache (no TestAuth RPC here), and then WriteAppend (local: no RPC), finally STORE (RPC) but as tasks increase the STORE gets incrementally heavy pushes to the server - So 1 RPC here per task but payload increases - hypothesis: this is affecting the performance since server need to do a AppendLog the same increasingly large shard write per task, per worker to each of the replica servers.
Now, with per task, one shard -> 2 RPCs per task - CREATE, Write(local), STORE
We can name shard with task names for easier parsing for the coordinator upon ListDirectoryInfo (should be fine since this happens only once or per failure-restart of coordinator)
Add optimization where-in each task completion amounts to one shard individual shard write.
Currently, there is one shard file per worker, that gets CREATE only once (RPC), now there is local copy - even then there is OpenCache (no TestAuth RPC here), and then WriteAppend (local: no RPC), finally STORE (RPC) but as tasks increase the STORE gets incrementally heavy pushes to the server - So 1 RPC here per task but payload increases - hypothesis: this is affecting the performance since server need to do a AppendLog the same increasingly large shard write per task, per worker to each of the replica servers.
Now, with per task, one shard -> 2 RPCs per task - CREATE, Write(local), STORE
We can name shard with task names for easier parsing for the coordinator upon ListDirectoryInfo (should be fine since this happens only once or per failure-restart of coordinator)