Unified · Split
| @@ -307,48 +307,47 @@ was created yet, i.e. during `git add …`. · 306 unmodified lines | |||
| 307 | 307 | ||
| 308 | `gitoxide` neither implements it nor it is used in concurrent environments, which is why we exclude it from this discovery. | 308 | `gitoxide` neither implements it nor it is used in concurrent environments, which is why we exclude it from this discovery. |
| 309 | 309 | ||
| 310 | ## Understanding changes to repositories | 310 | ## Applying changes to repositories while maintaining consistency |
| 311 | 311 | ||
| 312 | A change to any file in a git repository has the potential to affect the process operating on it. For the sake of practicality and from experience, we focus exclusively | 312 | A change to any file in a git repository has the potential to affect the process operating on it, hence it's important to perform them so these |
| 313 | on changes to the object database and the reference database along with typical interactions. | 313 | observe only valid repository states. For the sake of simplicity, we assume atomicity for all operations, but these still have to be ordered correctly |
| 314 | to assure a consistent state. | ||
| 314 | 315 | ||
| 315 | Related changes to multiple files are never atomic, and can be observed | 316 | ### Adding objects to the database |
| 316 | in an in-between state. | ||
| 317 | 317 | ||
| 318 | **TBD** | 318 | 1. add objects |
| 319 | 319 | * this is safe as these objects aren't reachable yet. | |
| 320 | ## Values | 320 | * object databases are append, only, so no existing state is ever affected. |
| 321 | 321 | 2. adjust references | |
| 322 | - We value _highly_ to scale object access performance with cores. | 322 | * point related references to the entry-points of the newly added objects (typically commits or tags) |
| 323 | - We value _more_ to offer a choice of trade-offs than to aim for a one-size-fits-all solution, unless the latter has no shortcomings. | 323 | * now the new objects can be discovered |
| 324 | |||
| 325 | - We don't value the handling of out of memory situations differently than panicking. This might change if `gitoxide` should fly to Mars or land in the linux kernel though. | ||
| 326 | - We don't value enabling 32 bit applications to deal with pack files greater than 4GB and leave this field entirely to the other implementations. | ||
| 327 | 324 | ||
| 328 | ## Known technical problems and their solutions | 325 | ## Known technical problems and their solutions |
| 329 | 326 | ||
| 327 | From what we know, changing repositories can have the following problems, many of which have decent solutions. | ||
| 330 | Solutions aren't always mutually exclusive despite the form of presentation suggesting it. | 328 | Solutions aren't always mutually exclusive despite the form of presentation suggesting it. |
| 331 | 329 | ||
| 332 | | **Database** | **Problem** | **Solution** | **Benefits** | **shortcomings** | **Example Implementation** | | 330 | | **Database** | **Problem** | **Solution** | **Benefits** | **shortcomings** | **Example Implementation** | |
| 333 | |-------------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------| | 331 | |-------------------|----------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------| |
| 334 | | **pack** | **1. initialization** | 1. map all packs at once | read-only possible; same latency for all objects | worst case init delay, highest resource usage, some packs may never be read | gitoxide | | 332 | | **pack** | **1. initialization** | 1. map all packs at once | read-only possible; same latency for all objects | worst case init delay, highest resource usage, some packs may never be read | gitoxide | |
| 335 | | | | 2. map packs later on object access miss | nearly no init delay, no unnecessary work, resource usage as needed | needs mutability; first access of some objects may be slow | libgit2, git | | 333 | | | | 2. map packs later on object access miss | nearly no init delay, no unnecessary work, resource usage as needed | needs mutability; first access of some objects may be slow | libgit2, git | |
| 336 | | **pack** | **2. file limit hit** | 1. fail | read-only possible | | gitoxide | | 334 | | **pack** | **2. file limit hit** | 1. fail | read-only possible | | gitoxide | |
| 337 | | | | 2. free resources and retry, then possibly fail | higher reliability | needs mutability | libgit2 (only on self-imposed limit) | | 335 | | | | 2. free resources and retry, then possibly fail | higher reliability | needs mutability | libgit2 (only on self-imposed limit) | |
| 338 | | **pack** | **3. file handle reduction/avoid hitting file limit** | 1. do not exceed internal handle count | some control over file handles | the entire application needs to respect count, needs sync with actual OS imposed limit, no sharing across multiple in-process pack databases | libgit2 (only within pack database) | | 336 | | **pack** | **3. file handle reduction/avoid hitting file limit** | 1. do not exceed internal handle count | some control over file handles | the entire application needs to respect count, needs sync with actual OS imposed limit, no sharing across multiple in-process pack databases | libgit2 (only within pack database) | |
| 339 | | | | 2. process-wide pooling of memory maps | share packs across multiple repositories instances | pack databases aren't self-contained anymore | | | 337 | | | | 2. process-wide pooling of memory maps | share packs across multiple repositories instances | pack databases aren't self-contained anymore | | |
| 340 | | | | 3. map whole packs at once (instead of non-overlapping parts of them) | Only consume one file handle per pack (as opposed to one per region) | cannot handle packs larger than 4GB on 32bit systems | gitoxide | | 338 | | | | 3. map whole packs at once (instead of non-overlapping parts of them) | Only consume one file handle per pack (as opposed to one per region) | cannot handle packs larger than 4GB on 32bit systems | gitoxide | |
| 341 | | | | 4. Multi-pack index files | A single index can be used N packs, saving N-1 packs for N>1 packs | | | | 339 | | | | 4. Multi-pack index files | A single index can be used N packs, saving N-1 packs for N>1 packs | | | |
| 342 | | **pack** | **4. object miss** | 1. fail | fast if object misses are expected | incorrect or a burden in user code if miss is due to changed packs | gitoxide | | 340 | | **pack** | **4. object miss** | 1. fail | fast if object misses are expected | incorrect or a burden in user code if miss is due to changed packs | gitoxide | |
| 343 | | | | 2. lazy-load more packs, retry, refresh known packs, retry, then fail | always correct even in the light of writers | can cause huge amount of extra work if object misses are expected; does not handle problem 5. | libgit2 | | 341 | | | | 2. lazy-load more packs, retry, refresh known packs, retry, then fail | always correct even in the light of writers | can cause huge amount of extra work if object misses are expected; does not handle problem 5. | libgit2 | |
| 344 | | **pack** | ~~5. race when creating/altering more than a pack at a time~~ | 1. ignore | | a chance for occasional object misses | all of them | | 342 | | **pack** | ~~5. race when creating/altering more than a pack at a time~~ | 1. ignore | | a chance for occasional object misses | all of them | |
| 345 | | | | 2. retry more than one time | greatly reduced likelyhood of object misses | | | | 343 | | | | 2. retry more than one time | greatly reduced likelyhood of object misses | | | |
| 346 | | **pack** | **6.too many (small) packs (i.e. due to pack-receive) reduce lookup performance** | 1. explode pack into loose objects (and deal with them separately) | can run in parallel (but is typically bound by max IOP/s) | might take a while if many objects are contained in the pack due to file IOP/s; needs recompresssion and looses delta compression; risk of too many small objects | | | 344 | | **pack** | **6.too many (small) packs (i.e. due to pack-receive) reduce lookup performance** | 1. explode pack into loose objects (and deal with them separately) | can run in parallel (but is typically bound by max IOP/s) | might take a while if many objects are contained in the pack due to file IOP/s; needs recompresssion and looses delta compression; risk of too many small objects | | |
| 347 | | | | 2. combine multiple packs into one | keep all benefits of packs; very fast if pack-to-pack copy is used; can run in parallel (but is typically bound by max IOP/s) | combining with big packs takes has to write a lot of data; can be costly if pack delta compression is used | | | 345 | | | | 2. combine multiple packs into one | keep all benefits of packs; very fast if pack-to-pack copy is used; can run in parallel (but is typically bound by max IOP/s) | combining with big packs takes has to write a lot of data; can be costly if pack delta compression is used | | |
| 348 | | **loose refs** | **7. readers observe in-between states of transactions** | 1. switch to ref-table database | | switches to shortcomings of ref-table | | | 346 | | **loose refs** | **7. readers observe in-between states of transactions** | 1. switch to ref-table database | | switches to shortcomings of ref-table | | |
| 349 | | **ref-table** | **8. contention of writers around unfair lock** | 1. implement process-internal queuing mechanism | minimal cost and high throughput with minimal, in-process synchronization | doesn't help with contention caused by multiple processes | | | 347 | | **ref-table** | **8. contention of writers around unfair lock** | 1. implement process-internal queuing mechanism | minimal cost and high throughput with minimal, in-process synchronization | doesn't help with contention caused by multiple processes | | |
| 350 | | | | 2. use ref-table partitions | works for in-process and multi-process case, even though it might be slower than queues. | I can't find information about it anymore | | | 348 | | | | 2. use ref-table partitions | works for in-process and multi-process case, even though it might be slower than queues. | I can't find information about it anymore | | |
| 351 | | **loose objects** | **9. too many loose objects reduce overall performance** | 1. use packs | | needs scheduled or otherwise managed maintenance, and winning strategies depend on the size and business of the repository | | | 349 | | **loose objects** | **9. too many loose objects reduce overall performance** | 1. use packs | | needs scheduled or otherwise managed maintenance, and winning strategies depend on the size and business of the repository | | |
| 350 | | **all** | **10. disk full/write failure/OOM** | 1. write temporary files first, with robust auto-removal, move into place when completed; partial transactions are robustly rolled back or stray files aren't discoverable or are valid on their own | | | gitoxide; git (not libgit2, it leaves partial packs on receive for example) | | ||
| 352 | 351 | ||
| 353 | ### Amendum problem 5. | 352 | ### Amendum problem 5. |
| 354 | 353 | ||
| @@ -359,9 +358,18 @@ and loose objects, and then removing the packs and loose objects they replace, t · 4 unmodified lines | |||
| 359 | 358 | ||
| 360 | ## Use-Cases | 359 | ## Use-Cases |
| 361 | 360 | ||
| 362 | We will look at typical access patterns holistically based on various use-cases, look at the problems they would run into and pick their preferred solution. | 361 | We will look at typical access patterns holistically based on various use-cases, look at the problems they would run into and pick their preferred solution that optimizes |
| 362 | for efficiency/performance/reduction of waste. | ||
| 363 | 363 | ||
| 364 | ## Learnings | 364 | ## Learnings and Actions Items |
| 365 | |||
| 366 | Please note that these are based on the following value system: | ||
| 367 | |||
| 368 | - We value _highly_ to scale object access performance with cores. | ||
| 369 | - We value _more_ to offer a choice of trade-offs than to aim for a one-size-fits-all solution, unless the latter has no shortcomings. | ||
| 370 | |||
| 371 | - We don't value the handling of out of memory situations differently than panicking. This might change if `gitoxide` should fly to Mars or land in the linux kernel though. | ||
| 372 | - We don't value enabling 32 bit applications to deal with pack files greater than 4GB and leave this field entirely to the other implementations. | ||
| 365 | 373 | ||
| 366 | ### Loose references database | 374 | ### Loose references database |
| 367 | 375 | ||