mirror of
https://github.com/blender/blender.git
synced 2026-01-17 14:23:15 +00:00
e47400c33f
Make it safe(r) for multiple Blender processes to download to the same file path. - Ensure downloading happens to uniquely-named temporary files. - Use atomic moves (except on Windows) to move these files to the final location. - Use locks to ensure the HTTP metadata cache files are valid. AFAIK there is no simple API to use on Windows to atomically rename a file when the target filename already exists. The last process to finish its download will "win", in the sense that the final state of the file will be determined by the last-finished download. The file-based HTTP metadata cache storage has also been updated, as there it is also possible for multiple processes to access the same file. Here a simpler approach is used, locking the JSON file before reading/writing it. Since it's always a small amount of data, there is a simple retry loop that attempts an open+lock on the file every 0.1 seconds. Only if that fails 20x the operation will fail. ------------ There still is a race condition in the current code, when there's multiple processes (`A` and `B`) downloading the same URL to the same file path: - `A` downloads file version v1 to a temp path - The file is updated to v2 on the server while `A` is downloading. - `B` downloads file v2 to a temp path - `A` finishes the download, and moves v1 temp → final path - `B` finishes the download, and moves v2 temp → final path - `B` writes the v2 HTTP metadata to the cache - `A` writes the v1 HTTP metadata to the cache Now the metadata cache contains v1, while the file on disk is v2. As a result, the next time the URL is fetched, it will be downloaded again. Note that even when solving this particular race condition, it is still possible for v2 to be smaller than v1, which causes the download to be finished faster. So then you end up with v1, even though v2 was also downloaded. I feel that these situations are getting more and more niche, and not worth solving right now. And to solve it, I would rather avoid more and more locking, and go for a smarter solution that prevents downloading the same thing multiple times simultanously. Pull Request: https://projects.blender.org/blender/blender/pulls/151671