Use cases for PHP generators
Despite being available since PHP 5.5.0, generators are still largely underused. In fact, it appears than most of the developers I know understand how generators work but don’t seem to see when they could be useful in real-world cases.
Yeah, generators definitely look great but you know… except for computing the Fibonacci sequence, I don’t see how it could be useful to me.
And they’re not wrong, even the examples in PHP’s documentation about generators
are pretty simplistic. They only show how to efficiently implement range
or iterate over the lines of a file.
But even from these simple examples we can understand the core aspects of
generators: they are just simplified Iterators.
A generator allows you to write code that uses
foreachto iterate over a set of data without needing to build an array in memory
With that in mind, I’ll try to explain why generators are awesome using some use-cases taken from applications I work on in my company.
Some context
I currently work for TEA. Basically, we develop an e-book reading ecosystem. It goes all the way from getting the e-book files from the publishers to present them in an e-commerce website and allowing the final customer to read them online (using a web-reader written by @johanpoirier) or on an e-reader.
In order to be able to sell these e-books and display relevant information to our customers, we need to have a lot of metadata for our products (title, format, price, publisher, author(s), …).
For most of the code samples that will be shown below, I’ll refer to these metadata under the name of e-book.
So here we go!
Iterating through large data-sets
For this first use-case, let’s assume that I have a large collection of e-books and I want to filter those which can be read in a web-reader.
Traditionally, I would write something like this:
1private function getEbooksEligibleToWebReader($ebooks)
2{
3 $rule = 'format = "EPUB" AND protection != "Adobe DRM"';
4 $filteredEbooks = [];
5
6 foreach ($ebooks as $ebook) {
7 if ($this->rulerz->satisfies($ebook, $rule)) {
8 $filteredEbooks[] = $ebook;
9 }
10 }
11
12 return $filteredEbooks;
13}
The problem here is easy to see: the more free e-books I have, the more
$filteredEbooks will consume memory.
A solution could be to create an iterator that would iterate through the
$ebooks and return the ones that are eligible. But we would have to create a
new class just for that and iterators are a bit tedious to write… Lucky us,
since PHP 5.5.0 we can use generators!
1private function getEbooksEligibleToWebReader($ebooks)
2{
3 $rule = 'format = "EPUB" AND protection != "Adobe DRM"';
4
5 foreach ($ebooks as $ebook) {
6 if ($this->rulerz->satisfies($ebook, $rule)) {
7 yield $ebook;
8 }
9 }
10}
Yep, refactoring the getEbooksEligibleToWebReader method to use generators is
as simple as replacing the assignation to $filteredEbooks by a yield
statement.
Assuming that $ebooks isn’t an array with all the e-books but an iterator or
a generator (even better!), the memory consumption will now be constant, no
matter the number of eligible books and we are sure to find these books only if
and when we need them.
Bonus: RulerZ internally uses generators so we could rewrite the method like this and be as efficient in terms of memory.
1private function getEbooksEligibleToWebReader($ebooks)
2{
3 $rule = 'format = "EPUB" AND protection != "Adobe DRM"';
4
5 return $this->rulerz->filter($ebooks, $rule);
6}
Aggregating several data-sources
Now, let’s consider the $ebooks retrieval part. I didn’t tell you, but these
e-books come in fact from different data-sources: a relational database and
Elasticsearch.
We can then write a simple method to aggregate these two sources:
1private function getEbooks()
2{
3 $ebooks = [];
4
5 // fetch from the DB
6 $stmt = $this->db->prepare("SELECT * FROM ebook_catalog");
7 $stmt->execute();
8 $stmt->setFetchMode(\PDO::FETCH_ASSOC);
9
10 foreach ($stmt as $data) {
11 $ebooks[] = $this->hydrateEbook($data);
12 }
13
14 // and from Elasticsearch (findAll uses ES scan/scroll)
15 $cursor = $this->esClient->findAll();
16
17 foreach ($cursor as $data) {
18 $ebooks[] = $this->hydrateEbook($data);
19 }
20
21 return $ebooks;
22}
But once again, the amount of memory used by this method depends too much on the number of e-books we have in the database and in Elasticsearch.
We could start by using generators to return the results:
1private function getEbooks()
2{
3 // fetch from the DB
4 $stmt = $this->db->prepare("SELECT * FROM ebook_catalog");
5 $stmt->execute();
6 $stmt->setFetchMode(\PDO::FETCH_ASSOC);
7
8 foreach ($stmt as $data) {
9 yield $this->hydrateEbook($data);
10 }
11
12 // and from Elasticsearch (findAll uses ES scan/scroll)
13 $cursor = $this->esClient->findAll();
14
15 foreach ($cursor as $data) {
16 yield $this->hydrateEbook($data);
17 }
18}
That’s better, but we still have a problem: our getBooks method does too much
work! We should split the two responsibilities (reading in the database and
calling Elasticsearch) in two separate methods:
1private function getEbooks()
2{
3 yield from $this->getEbooksFromDatabase();
4 yield from $this->getEbooksFromEs();
5}
6
7private function getEbooksFromDatabase()
8{
9 $stmt = $this->db->prepare("SELECT * FROM ebook_catalog");
10 $stmt->execute();
11 $stmt->setFetchMode(\PDO::FETCH_ASSOC);
12
13 foreach ($stmt as $data) {
14 yield $this->hydrateEbook($data);
15 }
16}
17
18private function getEbooksFromEs()
19{
20 // and from Elasticsearch (findAll uses ES scan/scroll)
21 $cursor = $this->esClient->findAll();
22
23 foreach ($cursor as $data) {
24 yield $this->hydrateEbook($data);
25 }
26}
You will notice the use of the yield from operator (available since PHP 7.0)
that allows to delegate the use of generators. That’s perfect, for instance, to
aggregate several data-sources that use generators.
In fact, the yield from keyword works with any Traversable object, so arrays
or iterators can also be used with this delegation operator.
Using this keyword, we could aggregate several data-sources with only a few lines of code:
1private function getEbooks()
2{
3 yield new Ebook(…);
4 yield from [new Ebook(…), new Ebook(…)];
5 yield from new ArrayIterator([new Ebook(…), new Ebook(…)]);
6 yield from $this->getEbooksFromCSV();
7 yield from $this->getEbooksFromDatabase();
8}
Complex, on-demand hydration for database rows
Another use-case I found for generators was the implementation of an on-demand hydration that could handle relationships.
I had to import hundreds of thousands of orders from a legacy database into our system, each order having several order lines.
Having both the order and the order lines was a pre-requisite for what we needed to do, so I had to write a method that would be able to return hydrated orders without being too slow or eating a lot of memory.
The idea is quite naive: join the order and the matching lines, and group order and order lines together in a loop.
1public function loadOrdersWithItems()
2{
3 $oracleQuery = <<<SQL
4SELECT o.*, item.*
5FROM order_history o
6INNER JOIN ORDER_ITEM item ON item.order_id = o.id
7ORDER BY order.id
8SQL;
9
10 if (($stmt = oci_parse($oracleDb, $oracleQuery)) === false) {
11 throw new \RuntimeException('Prepare fail in ');
12 }
13 if (oci_execute($stmt) === false) {
14 throw new \RuntimeException('Execute fail in ');
15 }
16
17 $currentOrderId = null;
18 $currentOrder = null;
19 while (($row = oci_fetch_assoc($stmt)) !== false) {
20 // did we move to the next order?
21 if ($row['ID'] !== $currentOrderId) {
22 if ($currentOrderId !== null) {
23 yield $currentOrder;
24 }
25
26 $currentOrderId = $row['ID'];
27
28 $currentOrder = $row;
29 $currentOrder['lines'] = [];
30 }
31
32 $currentOrder['lines'][] = $row;
33 }
34
35 yield $currentOrder;
36}
Using a generator, I managed to implement a method that could fetch orders from the database and join the corresponding order lines. All of this while consuming a stable amount of memory. The generator removed the need to keep track of all the orders with their order lines: the current order was all I needed to aggregate all the data.
Simulating async tasks
Last but not least, generators can also be used to simulate asynchronous tasks. While writing this post, I stumbled upon @nikita_ppv’s post about the same subject, and as he is the one who originally implemented generators in PHP, I’ll just link to his post.
He quickly explains what are generators and explains (in depth) how we can take advantage of the fact that they can both be interrupted and send/receive data to implement coroutines and even cooperative multitasking.
Wrapping up
Generators…
- … are simplified Iterators ;
- … can send an unlimited amount of data, without saturating the memory ;
- … can be aggregated using generators delegation ;
- … can be used to implement cooperative multitasking ;
- … are awesome!