Concurrency issues if two software definitions have the same source directory

[albertvaka]

Description

We have two software definitions that are built from the code in our repo, for which we use source path: '..'. When workers > 1, we see ENOENT failures due to having two FileSyncer trying to sync the same file.

Omnibus Version

master

Platform Version

Any Linux, on both Ruby 2.7 and 2.5.

Replication Case

See description.

Build Output

[PathFetcher: <FIRST RECIPE NAME>] I | 2021-01-04T10:11:06+00:00 | Copying from `..'
[PathFetcher: <SECOND RECIPE NAME>] I | 2021-01-04T10:11:06+00:00 | Copying from `..'
/usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/file_syncer.rb:142:in `stat': No such file or directory @ rb_file_s_stat - ../<FILE PATH HERE> (Errno::ENOENT)
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/file_syncer.rb:142:in `block in sync'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/file_syncer.rb:125:in `each'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/file_syncer.rb:125:in `sync'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/fetchers/path_fetcher.rb:64:in `fetch'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/software.rb:1075:in `fetch'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/project.rb:1306:in `block (3 levels) in download'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/thread_pool.rb:61:in `block (4 levels) in initialize'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/thread_pool.rb:59:in `loop'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/thread_pool.rb:59:in `block (3 levels) in initialize'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/thread_pool.rb:58:in `catch'
	from /usr/local/rvm/gems/ruby-2.7.0/bundler/gems/omnibus-ruby-1763439a90d1/lib/omnibus/thread_pool.rb:58:in `block (2 levels) in initialize'

---

[lamont-granquist]

Omnibus doesn't have "recipes", so this confused me quite a bit as to what you actually meant. You mean two different software definitions in the same project? I mentally mapped "recipes" onto "projects" first and not "software" and nearly reflexively closed this as having two different projects running concurrently trying to use the same source directory is massively not supported. From your output though and the comment about "workers > 1" it sounds like you're talking about software definitions though.

[albertvaka]

Thanks for looking at this @lamont-granquist 🙇 Indeed, I meant software definitions, sorry. I've updated the issue now.