Unified · Split
| @@ -307,48 +307,47 @@ was created yet, i.e. during `git add …`. · 306 unmodified lines | ||
| 307 | 307 | |
| 308 | 308 | `gitoxide` neither implements it nor it is used in concurrent environments, which is why we exclude it from this discovery. |
| 309 | 309 | |
| 310 | −## Understanding changes to repositories | |
| 310 | +## Applying changes to repositories while maintaining consistency | |
| 311 | 311 | |
| 312 | −A change to any file in a git repository has the potential to affect the process operating on it. For the sake of practicality and from experience, we focus exclusively | |
| 313 | −on changes to the object database and the reference database along with typical interactions. | |
| 312 | +A change to any file in a git repository has the potential to affect the process operating on it, hence it's important to perform them so these | |
| 313 | +observe only valid repository states. For the sake of simplicity, we assume atomicity for all operations, but these still have to be ordered correctly | |
| 314 | +to assure a consistent state. | |
| 314 | 315 | |
| 315 | −Related changes to multiple files are never atomic, and can be observed | |
| 316 | −in an in-between state. | |
| 316 | +### Adding objects to the database | |
| 317 | 317 | |
| 318 | −**TBD** | |
| 319 | − | |
| 320 | −## Values | |
| 321 | − | |
| 322 | −- We value _highly_ to scale object access performance with cores. | |
| 323 | −- We value _more_ to offer a choice of trade-offs than to aim for a one-size-fits-all solution, unless the latter has no shortcomings. | |
| 324 | − | |
| 325 | −- We don't value the handling of out of memory situations differently than panicking. This might change if `gitoxide` should fly to Mars or land in the linux kernel though. | |
| 326 | −- We don't value enabling 32 bit applications to deal with pack files greater than 4GB and leave this field entirely to the other implementations. | |
| 318 | +1. add objects | |
| 319 | + * this is safe as these objects aren't reachable yet. | |
| 320 | + * object databases are append, only, so no existing state is ever affected. | |
| 321 | +2. adjust references | |
| 322 | + * point related references to the entry-points of the newly added objects (typically commits or tags) | |
| 323 | + * now the new objects can be discovered | |
| 327 | 324 | |
| 328 | 325 | ## Known technical problems and their solutions |
| 329 | 326 | |
| 327 | +From what we know, changing repositories can have the following problems, many of which have decent solutions. | |
| 330 | 328 | Solutions aren't always mutually exclusive despite the form of presentation suggesting it. |
| 331 | 329 | |
| 332 | −| **Database** | **Problem** | **Solution** | **Benefits** | **shortcomings** | **Example Implementation** | | |
| 333 | −|-------------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------| | |
| 334 | −| **pack** | **1. initialization** | 1. map all packs at once | read-only possible; same latency for all objects | worst case init delay, highest resource usage, some packs may never be read | gitoxide | | |
| 335 | −| | | 2. map packs later on object access miss | nearly no init delay, no unnecessary work, resource usage as needed | needs mutability; first access of some objects may be slow | libgit2, git | | |
| 336 | −| **pack** | **2. file limit hit** | 1. fail | read-only possible | | gitoxide | | |
| 337 | −| | | 2. free resources and retry, then possibly fail | higher reliability | needs mutability | libgit2 (only on self-imposed limit) | | |
| 338 | −| **pack** | **3. file handle reduction/avoid hitting file limit** | 1. do not exceed internal handle count | some control over file handles | the entire application needs to respect count, needs sync with actual OS imposed limit, no sharing across multiple in-process pack databases | libgit2 (only within pack database) | | |
| 339 | −| | | 2. process-wide pooling of memory maps | share packs across multiple repositories instances | pack databases aren't self-contained anymore | | | |
| 340 | −| | | 3. map whole packs at once (instead of non-overlapping parts of them) | Only consume one file handle per pack (as opposed to one per region) | cannot handle packs larger than 4GB on 32bit systems | gitoxide | | |
| 341 | −| | | 4. Multi-pack index files | A single index can be used N packs, saving N-1 packs for N>1 packs | | | | |
| 342 | −| **pack** | **4. object miss** | 1. fail | fast if object misses are expected | incorrect or a burden in user code if miss is due to changed packs | gitoxide | | |
| 343 | −| | | 2. lazy-load more packs, retry, refresh known packs, retry, then fail | always correct even in the light of writers | can cause huge amount of extra work if object misses are expected; does not handle problem 5. | libgit2 | | |
| 344 | −| **pack** | ~~5. race when creating/altering more than a pack at a time~~ | 1. ignore | | a chance for occasional object misses | all of them | | |
| 345 | −| | | 2. retry more than one time | greatly reduced likelyhood of object misses | | | | |
| 346 | −| **pack** | **6.too many (small) packs (i.e. due to pack-receive) reduce lookup performance** | 1. explode pack into loose objects (and deal with them separately) | can run in parallel (but is typically bound by max IOP/s) | might take a while if many objects are contained in the pack due to file IOP/s; needs recompresssion and looses delta compression; risk of too many small objects | | | |
| 347 | −| | | 2. combine multiple packs into one | keep all benefits of packs; very fast if pack-to-pack copy is used; can run in parallel (but is typically bound by max IOP/s) | combining with big packs takes has to write a lot of data; can be costly if pack delta compression is used | | | |
| 348 | −| **loose refs** | **7. readers observe in-between states of transactions** | 1. switch to ref-table database | | switches to shortcomings of ref-table | | | |
| 349 | −| **ref-table** | **8. contention of writers around unfair lock** | 1. implement process-internal queuing mechanism | minimal cost and high throughput with minimal, in-process synchronization | doesn't help with contention caused by multiple processes | | | |
| 350 | −| | | 2. use ref-table partitions | works for in-process and multi-process case, even though it might be slower than queues. | I can't find information about it anymore | | | |
| 351 | −| **loose objects** | **9. too many loose objects reduce overall performance** | 1. use packs | | needs scheduled or otherwise managed maintenance, and winning strategies depend on the size and business of the repository | | | |
| 330 | +| **Database** | **Problem** | **Solution** | **Benefits** | **shortcomings** | **Example Implementation** | | |
| 331 | +|-------------------|----------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------| | |
| 332 | +| **pack** | **1. initialization** | 1. map all packs at once | read-only possible; same latency for all objects | worst case init delay, highest resource usage, some packs may never be read | gitoxide | | |
| 333 | +| | | 2. map packs later on object access miss | nearly no init delay, no unnecessary work, resource usage as needed | needs mutability; first access of some objects may be slow | libgit2, git | | |
| 334 | +| **pack** | **2. file limit hit** | 1. fail | read-only possible | | gitoxide | | |
| 335 | +| | | 2. free resources and retry, then possibly fail | higher reliability | needs mutability | libgit2 (only on self-imposed limit) | | |
| 336 | +| **pack** | **3. file handle reduction/avoid hitting file limit** | 1. do not exceed internal handle count | some control over file handles | the entire application needs to respect count, needs sync with actual OS imposed limit, no sharing across multiple in-process pack databases | libgit2 (only within pack database) | | |
| 337 | +| | | 2. process-wide pooling of memory maps | share packs across multiple repositories instances | pack databases aren't self-contained anymore | | | |
| 338 | +| | | 3. map whole packs at once (instead of non-overlapping parts of them) | Only consume one file handle per pack (as opposed to one per region) | cannot handle packs larger than 4GB on 32bit systems | gitoxide | | |
| 339 | +| | | 4. Multi-pack index files | A single index can be used N packs, saving N-1 packs for N>1 packs | | | | |
| 340 | +| **pack** | **4. object miss** | 1. fail | fast if object misses are expected | incorrect or a burden in user code if miss is due to changed packs | gitoxide | | |
| 341 | +| | | 2. lazy-load more packs, retry, refresh known packs, retry, then fail | always correct even in the light of writers | can cause huge amount of extra work if object misses are expected; does not handle problem 5. | libgit2 | | |
| 342 | +| **pack** | ~~5. race when creating/altering more than a pack at a time~~ | 1. ignore | | a chance for occasional object misses | all of them | | |
| 343 | +| | | 2. retry more than one time | greatly reduced likelyhood of object misses | | | | |
| 344 | +| **pack** | **6.too many (small) packs (i.e. due to pack-receive) reduce lookup performance** | 1. explode pack into loose objects (and deal with them separately) | can run in parallel (but is typically bound by max IOP/s) | might take a while if many objects are contained in the pack due to file IOP/s; needs recompresssion and looses delta compression; risk of too many small objects | | | |
| 345 | +| | | 2. combine multiple packs into one | keep all benefits of packs; very fast if pack-to-pack copy is used; can run in parallel (but is typically bound by max IOP/s) | combining with big packs takes has to write a lot of data; can be costly if pack delta compression is used | | | |
| 346 | +| **loose refs** | **7. readers observe in-between states of transactions** | 1. switch to ref-table database | | switches to shortcomings of ref-table | | | |
| 347 | +| **ref-table** | **8. contention of writers around unfair lock** | 1. implement process-internal queuing mechanism | minimal cost and high throughput with minimal, in-process synchronization | doesn't help with contention caused by multiple processes | | | |
| 348 | +| | | 2. use ref-table partitions | works for in-process and multi-process case, even though it might be slower than queues. | I can't find information about it anymore | | | |
| 349 | +| **loose objects** | **9. too many loose objects reduce overall performance** | 1. use packs | | needs scheduled or otherwise managed maintenance, and winning strategies depend on the size and business of the repository | | | |
| 350 | +| **all** | **10. disk full/write failure/OOM** | 1. write temporary files first, with robust auto-removal, move into place when completed; partial transactions are robustly rolled back or stray files aren't discoverable or are valid on their own | | | gitoxide; git (not libgit2, it leaves partial packs on receive for example) | | |
| 352 | 351 | |
| 353 | 352 | ### Amendum problem 5. |
| 354 | 353 | |
| @@ -359,9 +358,18 @@ and loose objects, and then removing the packs and loose objects they replace, t · 4 unmodified lines | ||
| 359 | 358 | |
| 360 | 359 | ## Use-Cases |
| 361 | 360 | |
| 362 | −We will look at typical access patterns holistically based on various use-cases, look at the problems they would run into and pick their preferred solution. | |
| 361 | +We will look at typical access patterns holistically based on various use-cases, look at the problems they would run into and pick their preferred solution that optimizes | |
| 362 | +for efficiency/performance/reduction of waste. | |
| 363 | 363 | |
| 364 | −## Learnings | |
| 364 | +## Learnings and Actions Items | |
| 365 | + | |
| 366 | +Please note that these are based on the following value system: | |
| 367 | + | |
| 368 | +- We value _highly_ to scale object access performance with cores. | |
| 369 | +- We value _more_ to offer a choice of trade-offs than to aim for a one-size-fits-all solution, unless the latter has no shortcomings. | |
| 370 | + | |
| 371 | +- We don't value the handling of out of memory situations differently than panicking. This might change if `gitoxide` should fly to Mars or land in the linux kernel though. | |
| 372 | +- We don't value enabling 32 bit applications to deal with pack files greater than 4GB and leave this field entirely to the other implementations. | |
| 365 | 373 | |
| 366 | 374 | ### Loose references database |
| 367 | 375 | |