Skip to content

Conversation

MineCake147E
Copy link

@MineCake147E MineCake147E commented Jun 11, 2025

  • Separated watcher update from StorageService.updateCachedStacks().
    • Optimized the now separate watcher update code.
    • Watcher updates now only runs in StorageService.onServerEndTick()
  • Replaced the ItemStack hash calculation in AEItemKey with a new one with more entropy, dramatically reducing the chance of the slow lookup in hash table, and ItemStack.isSameItemSameComponents calls.
    • Added AEKey.hashCode64() for improved-entropy hashes.
    • FuzzySearch.KeyComparator now uses AEKey.hashCode64() for finer tiebreaker.
  • Replaced the FluidStack hash calculation in AEFluidKey with a new one with more entropy, reducing the chance of the slow lookup in hash table, and FluidStack.isSameFluidSameComponents calls.
  • Optimized ME IO Port Fill/Empty operations (~10x faster in some scenario).
    • Added several filtering APIs for improved filling performance.

Resolves #8539

  * Optimized the now separate watcher update code.
  * Watcher updates now only runs in `StorageService.onServerEndTick()`
… with more entropy, dramatically reducing the chance of the slow lookup in hash table, and `ItemStack.isSameItemSameComponents` calls.

  * Added `AEKey.hashCode64()` for improved-entropy hashes.
  * `FuzzySearch.KeyComparator` now uses `AEKey.hashCode64()` for finer tiebreaker.
…ne with more entropy, dramatically reducing the chance of the slow lookup in hash table, and `FluidStack.isSameFluidSameComponents` calls.

* Renamed `ItemHashHelper` to `HashHelper`
* Hash calculation now captures more entropy sources.
* `NetworkStorage` now uses `Int2ObjectAVLTreeMap` instead of `NavigableMap`
* `StorageService.notifyWatchers()` now tries to avoid frequent rehashes.
* `BasicCellInventory.extract()` no longer calls `getCellItems()` more than once.
* `IOPortBlockEntity` no longer calls `KeyCounter.iterator()` more than necessary.
  * `IOPortBlockEntity.fillCells` now prioritizes partitioned cells.
  * `IOPortBlockEntity.fillCells` now filters available items from storage with cell's configurations.
  * Added several filtering APIs for improved filling performance.
Copy link
Member

@Technici4n Technici4n left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for sending the PR. Some notes:

  • The IO port changes should be split off to a separate PR. But the patch is too long... I wouldn't accept anything that increases the code size so much.
  • Widening hashCode to a long should not make a noticeable difference.
  • The new hash code computation sounds like it could make a big difference, but besides its size, do you have some examples of problematic cases that this fixes? I am thinking of say an example where two stacks are different but have the same hash code.

@MineCake147E MineCake147E changed the title Optimized watcher update and ME IO Port Fill/Empty operations Optimized watcher update Jun 12, 2025
@MineCake147E
Copy link
Author

MineCake147E commented Jun 12, 2025

  • The new hash code computation sounds like it could make a big difference, but besides its size, do you have some examples of problematic cases that this fixes? I am thinking of say an example where two stacks are different but have the same hash code.

For example, with old algorithm, {id:"minecraft:netherite_hoe",components:{"minecraft:damage":643,"minecraft:repair_cost":644}} and {id:"minecraft:netherite_hoe",components:{"minecraft:damage":472,"minecraft:repair_cost":449}} have the same hash code 0x6116_8C69.
This is just one example of thousands of different colliding combinations of damage and repair_cost among minecraft:netherite_hoe.
Another example of hash collision is 0x6CCF_3935 from {id:"minecraft:bow",components:{"minecraft:damage":311,"minecraft:enchantments":{levels:{"minecraft:power":3,"minecraft:unbreaking":3}}}} and {id:"minecraft:bow",components:{"minecraft:damage":340,"minecraft:enchantments":{levels:{"minecraft:power":4,"minecraft:unbreaking":3}}}}, both being survival-obtainable item via fishing.
Even with the same item with different damage, enchantments, and repair cost, the hash collides so easily. That alone signifies how frequently the items' hash code collide each other.
But nearly all the collisions are gone with the new hash algorithm.

  • The IO port changes should be split off to a separate PR. But the patch is too long... I wouldn't accept anything that increases the code size so much.

I updated the code accordingly.
I stress-tested the update logic using IO port, but IO port was hiding the performance difference of watcher update logic.
So I had to optimize that for a better stress-testing experience.
I will post a separate PR for that matter, after this patch gets merged.

@MineCake147E MineCake147E requested a review from Technici4n June 12, 2025 07:21
@Technici4n
Copy link
Member

OK I see. Is there maybe a way that you can provide the list of items in the system? I would like to get this fixed in NeoForge if possible, with a targeted fix. A /ae2 grids export of the big grid should contain all the items and be a useful reference. 🤔

@MineCake147E
Copy link
Author

Is there maybe a way that you can provide the list of items in the system? I would like to get this fixed in NeoForge if possible, with a targeted fix. A /ae2 grids export of the big grid should contain all the items and be a useful reference. 🤔

Here is the one with many colliding Netherite hoes. I hope that helps.
ae2_grid_6_20250613_1832.zip

@Technici4n
Copy link
Member

Thanks a lot for this data. Thanks to it, I managed to find a strategy for a simpler hashCode fix. The goal is to implement it in NeoForge directly, see: neoforged/NeoForge#2332.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Frequent Hash collision and redundant lookups in StorageService.updateCachedStacks()
2 participants