fix: improve handling of overflows when there are more pack than we can hold.

Internally, there is a statically allocated vec which holds opened packs and indices. When reconciling the disk-state with what's currently loaded, it was possible to get into a situation where there were more files than we could fit into the slotmap and got into an invalid state. The system now generously trashes existing slots to be able to load more rencent ones, which seems to help in this situation. It's probably still leading to strange situations where not all objects can be read.

Sebastian Thiel committed 1 year ago

Commit 724601d1 parent af8f2012 files · +10 −4Browse files

Unified · Split

2 files

gix-odb/src/lib.rs +1 −1
gix-odb/src/store_impls/dynamic/load_index.rs +9 −3

gix-odb/src/lib.rs+1−1

@@ -119,7 +119,7 @@ pub struct Store { · 118 unmodified lines
119		119
120	/// The below state acts like a slot-map with each slot is mutable when the write lock is held, but readable independently of it.	120	/// The below state acts like a slot-map with each slot is mutable when the write lock is held, but readable independently of it.
121	/// This allows multiple file to be loaded concurrently if there is multiple handles requesting to load packs or additional indices.	121	/// This allows multiple file to be loaded concurrently if there is multiple handles requesting to load packs or additional indices.
122	/// The map is static and cannot typically change.	122	/// The map is static and cannot change.
123	/// It's read often and changed rarely.	123	/// It's read often and changed rarely.
124	pub(crate) files: Vec<types::MutableIndexAndPack>,	124	pub(crate) files: Vec<types::MutableIndexAndPack>,
125		125

gix-odb/src/store_impls/dynamic/load_index.rs+9−3

                         Option::as_ref(&files_guard).expect("slot is set or we wouldn't know it points to this file");
                     if index_info.is_multi_index() && files.mtime() != mtime {
                         // we have a changed multi-pack index. We can't just change the existing slot as it may alter slot indices
-                        // that are currently available. Instead we have to move what's there into a new slot, along with the changes,
+                        // that are currently available. Instead, we have to move what's there into a new slot, along with the changes,
                         // and later free the slot or dispose of the index in the slot (like we do for removed/missing files).
                         index_paths_to_add.push_back((index_info, mtime, Some(slot_idx)));
                         // If the current slot is loaded, the soon-to-be copied multi-index path will be loaded as well.
                         needed: index_paths_to_add.len() + 1, /*the one currently popped off*/
                     });
                 }
+                // Don't allow duplicate indicates, we need a 1:1 mapping.
+                if new_slot_map_indices.contains(&next_possibly_free_index) {
+                    next_possibly_free_index = (next_possibly_free_index + 1) % self.files.len();
+                    num_indices_checked += 1;
+                    continue 'increment_slot_index;
+                }
                 let slot_index = next_possibly_free_index;
                 let slot = &self.files[slot_index];
                 next_possibly_free_index = (next_possibly_free_index + 1) % self.files.len();
         }
         // Unlike libgit2, do not sort by modification date, but by size and put the biggest indices first. That way
         // the chance to hit an object should be higher. We leave it to the handle to sort by LRU.
-        // Git itself doesn't change the order which may safe time, but we want it to be stable which also helps some tests.
+        // Git itself doesn't change the order which may save time, but we want it to be stable which also helps some tests.
         // NOTE: this will work well for well-packed repos or those using geometric repacking, but force us to open a lot
         //       of files when dealing with new objects, as there is no notion of recency here as would be with unmaintained
         //       repositories. Different algorithms should be provided, like newest packs first, and possibly a mix of both
         Ok(indices_by_modification_time)
     }
-    /// returns Ok<dest slot was empty> if the copy could happen because dest-slot was actually free or disposable , and Some(true) if it was empty
+    /// returns `Ok(dest_slot_was_empty)` if the copy could happen because dest-slot was actually free or disposable.
     #[allow(clippy::too_many_arguments)]
     fn try_set_index_slot(
         lock: &parking_lot::MutexGuard<'_, ()>,