getAllKeys doesnt return all keys

[Kapsonfire]

I compiled v3.0.3 (igbinary support) for my since php7 branch (3.01b?) didnt gave results for getAllKeys() (CLIENT ERROR)

now it works, but i dont get all keys

https://stackoverflow.com/questions/54625489/memcached-not-storing-array-of-complex-items-with-specific-keys

array(1) { ["localhost:11212"]=> array(79) { ["pid"]=> int(31376) ["uptime"]=> int(13852) ["time"]=> int(1549883801) ["version"]=> string(6) "1.5.12" ["libevent"]=> string(13) "2.0.21-stable" ["pointer_size"]=> int(64) ["rusage_user"]=> float(4.856) ["rusage_system"]=> float(3.02) ["max_connections"]=> int(1024) ["curr_connections"]=> int(1) ["total_connections"]=> int(2158) ["rejected_connections"]=> int(0) ["connection_structures"]=> int(4) ["reserved_fds"]=> int(20) ["cmd_get"]=> int(4835) ["cmd_set"]=> int(2432) ["cmd_flush"]=> int(0) ["cmd_touch"]=> int(10) ["get_hits"]=> int(4669) ["get_misses"]=> int(166) ["get_expired"]=> int(0) ["get_flushed"]=> int(0) ["delete_misses"]=> int(1836) ["delete_hits"]=> int(3) ["incr_misses"]=> int(0) ["incr_hits"]=> int(0) ["decr_misses"]=> int(0) ["decr_hits"]=> int(0) ["cas_misses"]=> int(0) ["cas_hits"]=> int(0) ["cas_badval"]=> int(0) ["touch_hits"]=> int(10) ["touch_misses"]=> int(0) ["auth_cmds"]=> int(0) ["auth_errors"]=> int(0) ["bytes_read"]=> int(3063149) ["bytes_written"]=> int(3450961) ["limit_maxbytes"]=> int(1073741824) ["accepting_conns"]=> int(1) ["listen_disabled_num"]=> int(0) ["time_in_listen_disabled_us"]=> int(0) ["threads"]=> int(4) ["conn_yields"]=> int(0) ["hash_power_level"]=> int(16) ["hash_bytes"]=> int(524288) ["hash_is_expanding"]=> int(0) ["slab_reassign_rescues"]=> int(0) ["slab_reassign_chunk_rescues"]=> int(0) ["slab_reassign_evictions_nomem"]=> int(0) ["slab_reassign_inline_reclaim"]=> int(0) ["slab_reassign_busy_items"]=> int(0) ["slab_reassign_busy_deletes"]=> int(0) ["slab_reassign_running"]=> int(0) ["slabs_moved"]=> int(0) ["lru_crawler_running"]=> int(0) ["lru_crawler_starts"]=> int(5390) ["lru_maintainer_juggles"]=> int(174542) ["malloc_fails"]=> int(0) ["log_worker_dropped"]=> int(0) ["log_worker_written"]=> int(0) ["log_watcher_skipped"]=> int(0) ["log_watcher_sent"]=> int(0) ["bytes"]=> int(34906) ["curr_items"]=> int(92) ["total_items"]=> int(2432) ["slab_global_page_pool"]=> int(0) ["expired_unfetched"]=> int(55) ["evicted_unfetched"]=> int(0) ["evicted_active"]=> int(0) ["evictions"]=> int(0) ["reclaimed"]=> int(55) ["crawler_reclaimed"]=> int(1) ["crawler_items_checked"]=> int(1999) ["lrutail_reflocked"]=> int(652) ["moves_to_cold"]=> int(3916) ["moves_to_warm"]=> int(1687) ["moves_within_lru"]=> int(71) ["direct_reclaims"]=> int(0) ["lru_bumps_dropped"]=> int(0) } }

result of get stats

result of getAllKeys return not the 92 items in stats

[Kapsonfire]

installed php7.3 with pecl install memcached (3.1.3)

still not working

[sodabrew]

This is a known limitation, getAllKeys can only get some keys. See #367, #315, #203 for the history of how this came to be.

[lexo-mfleuti]

This makes the whole software absolutely useless. It means that we basically need to clear the whole cache on every change any customer makes on any virtual host. So basically there's no caching at all since the cache needs to be recreated all the time. It worked well with the old version. But doesn't with the new version? That does not make any sense. The whole idea of the cache breaks down.

[sodabrew]

You should not use getAllKeys except for debugging. It's a very slow call to a memcached server and causes problems at scale.

[lexo-mfleuti]

We are using Memcached on a host with like 100 different websites. Each website stores the objects into Memcached. We need to check if the object is still in the cache or not. If it's in the Cache => Memcached shall deliver. Otherwise we regenerate the page and store it in the cache. What other solution is there to make this work otherwise? We need to be able to get the information if an object is stored in the cache or not.

[sodabrew]

You should be using one of the get family of functions:

php-memcached/memcached-api.php

Lines 264 to 274 in 7443d16

	public function get( $key, callable $cache_cb = null, $flags = 0) {}
	public function getByKey( $server_key, $key, callable $cache_cb = null, $flags = 0 ) {}
	public function getMulti( array $keys, $flags = 0) {}
	public function getMultiByKey( $server_key, array $keys, $flags = 0) {}
	public function getDelayed( array $keys, $with_cas = null, $value_cb = null ) {}
	public function getDelayedByKey( $server_key, array $keys, $with_cas = null, $value_cb = null ) {}

When you set a key-value into memcached, the key should be set such that on a GET request to your website, you can get("this page key exactly") without fetching extra results that aren't relevant to the page you're about to return.

This issue is about the getAllKeys function. getAllKeys is for diagnostics only. If you are calling getAllKeys and then in your PHP code you are iterating over the list of all keys on the entire memcached server in order to find the one or more keys specific to the page you are about to return, you're being very inefficient and I strongly advise adjusting your code as described above.

[lexo-mfleuti]

Thank you very much for your swift and precise return on this issue. And sorry for my little outbreak. As it seems we were doing it all wrong and then did not understand why the stuff we were doing wrong doesn't work as expected ;)
I'll look into this with our developers and optimize the code as suggested. Thanks again for your time and everything!

[lexo-mfleuti]

There's one problem though: When we want to clear the cache for a whole website (all keys stored in the cache for one cached website - which can be like 100 objects (pages) or more), we need to be able to easily tell the cache to do so. Our developer tells me, that this is why we used the getAllKeys function. The only solution we see now is to store every key of every page generated in a file (as JSON array) and when it comes to deleting we'd overgive this array to the Memcached deleteMulti() function. Or is there a simpler way to do that? I imagine a function like (pseudocode)
Memcached::delete("_website_key_.*")
Meaning: Delete all keys which start with "website_key". Something like that would be lovely ;)

[sodabrew]

What you’re looking for is generally called a “cache tag” to invalidate a group of keys. Memcached doesn’t support this.

Using getAllKeys is a bad plan again because of the performance cost I mentioned in the previous post. You won’t see it until you’re at major scale, and then it’ll take down the memcached server performance for a few seconds. Remember memcached is designed for scale where this would be really bad.

There are other strategies for this type of cache invalidation, google around for ideas!

[Maikuolan]

This issue is about the getAllKeys function. getAllKeys is for diagnostics only.

Any chance anyone might be willing to drop a note about that into the relevant php.net documentation? Currently, I don't see any mention about that in said documentation, which lends to one, upon reading said documentation, to assume that the function could, in fact, be appropriate for uses beyond just diagnostics.