The Problem: Duplicated Messages in the Queue

Our users often generate activities that trigger multiple events. These events, such as syncing user data with third-party services, get queued for processing. The challenge arises when multiple events with the same key (such as user ID) are added to the queue in quick succession. Without a proper mechanism in place, older events could overwrite newer data, or worse, important data could be lost in third-party systems.

To address this, we initially utilized the WithoutOverlapping feature in Laravel. This ensures that two jobs with the same key do not execute simultaneously. However, we still encountered problems where older jobs were being processed after newer, more relevant ones.

Iteration 1: Increasing the Retry Count

Our first attempt to fix the issue involved increasing the number of retries on the jobs. We thought that by changing the retry count from the default of 3 to 25, we could mitigate the problem, allowing the jobs more chances to succeed.

// Increasing retries in the job class
public $tries = 25;

During testing in our local and development environments, things looked fine. We even conducted load testing, and the system performed well under small loads. However, after deploying the changes to production, the problem returned within a few days. The increased retries only delayed the inevitable and did not solve the root issue.

Iteration 2: Leveraging $maxExceptions

Next, we tried using the $maxExceptions parameter. This parameter controls how many times a job can fail before it is marked as failed. We set it to 1, meaning that if a job failed once, it would be immediately retried until there was no other job with the same key currently in progress.

// Setting maximum exceptions in the job class
public $maxExceptions = 1;

While unconventional at first glance, this approach worked remarkably well. The message was re-queued until there were no other conflicting jobs, ensuring we didn't overwrite new data with older data. However, this came with a downside.

Pros and Cons of $maxExceptions

By using $maxExceptions, we lost Laravel's native auto-retry functionality. If a message failed, it needed to be manually re-run. Fortunately, this wasn’t a big issue for us because our users generate many such events. Each subsequent message in the queue essentially acts as a retry, ensuring that the most recent data is always processed.

Handling Duplicated Events

A key challenge for us was managing the large volume of duplicated events that we didn’t want to process. Laravel provides a Rate Limiting feature that allows you to throttle the execution of jobs. However, we couldn’t simply rate-limit all events because we couldn’t afford to skip the final message in the queue. The last event typically signifies that the user has completed all updates, and the latest state needs to be synchronized with third-party services.

The Solution: Marking Outdated Messages

To overcome this challenge, we implemented a custom middleware: MarkOutdatedMiddleware. This middleware checks for the presence of newer messages with the same key in the queue. If such a message exists, the current message is marked as outdated and skipped, ensuring only the latest message is processed.

Here’s the implementation of MarkOutdatedMiddleware:

class MarkOutdatedMiddleware
{
    public function __construct(readonly protected string $uniqueId, readonly protected int $dispatchedAt) {}

    public function handle($job, $next)
    {
        $lastAddedTime = Cache::driver('redis')->get($this->uniqueId);
        if ($lastAddedTime && $lastAddedTime > $this->dispatchedAt) {
            return;
        }

        return $next($job);
    }
}

This middleware ensures that only the most recent message for a given key is processed, while older, outdated messages are skipped. This solution fit well with our business logic, where the final state update is the most critical.

Also, below you can see an example of Message Handler class:

class SomeJob implements ShouldQueue
{
    use Dispatchable;
    use InteractsWithQueue;
    use Queueable;
    use SerializesModels;

    public $tries = 0;

    public $maxExceptions = 1;

    private int $userId;
    private int $dispatchedAt;

    public function __construct(int $userId, ?int $dispatchedAt = null)
    {
        $this->userId = $userId;
        $this->onQueue('user-queue');
        $this->dispatchedAt = $dispatchedAt ?? time();
        Cache::driver('redis')->put($this->uniqueId(), time(), 600);
    }

    public function middleware(): array
    {
        return [
            new MarkOutdatedMiddleware($this->uniqueId(), $this->dispatchedAt),
            (new WithoutOverlapping($this->uniqueId()))->releaseAfter(5)
                ->expireAfter(30),
        ];
    }

    public function handle(User $user): void
    {
        //Some business logic
    }

    public function uniqueId(): string
    {
        return sprintf('unique-job-for-user-id-%s', $this->getUserId());
    }
}

Conclusion

By combining Laravel's WithoutOverlapping feature with a custom middleware solution, we were able to effectively handle duplicated events and avoid processing outdated messages. The result was a more efficient queue system, free from the problems of overwriting or losing data.

Final Thoughts

This approach may not be suitable for every use case, but it worked well for our high-frequency, user-generated events. If you face similar challenges in managing queues, consider using custom middleware alongside Laravel's existing features.

If you have any questions, feedback, or better solutions, feel free to share them in the comments or reach out directly!

Thank you for reading!

P.S.: Of course, this story doesn't apply to 100% or maybe even 90% of developers and companies. However, based on my experience conducting hundreds of interviews and working on around 10 projects, the situation is very similar if not 1:1.

Today, I propose to explore an example of how to write code that adheres to the following best practices (or at least I would like it to):

1. Open-Closed Principle

2. Dependency Inversion + Dependency Injection

3. A method must not have more than 5-7 dependencies in the constructor(link1, link2).

4. Single Responsibility

If you know of any other practices, please add them.

Let's consider this question based on a typical task that developers often face, using the example of payment methods.

The problem statement

You're assigned to maintain a project with payment functionality, such as generating reports based on a parameter that comes in the GET request, like https://pay.com?provider={paypal}/. Let's call this parameter "provider," and the implemented payment methods are provider1, provider2, provider3, and provider4.

The current code is implemented, at best, like this:

<?php
class PaymentReportGenerator{
    function __construct($provider1, $provider2, $provider3, $provider4){
    }

    public function generateReport(string $targetProviderName){    
        switch($targetProviderName):
        case $provider1::NAME;
            return $provider1->getReport();
        case $provider2::NAME;
            return $provider2->getReport();
        case $provider3::NAME;
            return $provider3->getReport();
        case $provider4::NAME;
            return $provider4->getReport();
        default:
            throw new \PaymentReportServiceException("Wrong provider name given");
    }
}

The code is quite good, and often programmers consider it to comply with all SOLID principles.

Now, you receive a new task: the company is expanding to new markets, and you need to add 6 more payment methods, with the possibility of adding more soon.

The developer now faces a challenging choice of how to extend the existing code. I'll try to list typical ways to solve this problem:

1) Straightforward approach

Increase the number of parameters in the constructor and add more cases to the switch statement.
This approach might seem like a quick solution, but it tends to violate the Open-Closed Principle, as every time a new provider is added, you need to modify the generateReport function. Additionally, it increases the cyclomatic complexity and makes the code harder to maintain.

2) Abandon Dependency Injection

Dynamically create an instance of the required service based on the incoming value. This typically looks like this:

public function generateReport(string $targetProviderName){
$className = '/some/namespace/' . targetProviderName . 'PaymentReportService';
$methodName = 'getReport';
$paymentReportService = new $className();

if (class_exists($className)) {
    $instance = new $className();

    if (method_exists($instance, $methodName)) {
        return $instance->$methodName("John");
    } else {
        echo "Method $methodName not found in class $className.";
    }
} else {
    echo "Class $className not found.";
}

The drawbacks of this approach are likely apparent to everyone, but I'll mention a few of the most glaring issues:

1. All dependencies, if they exist, must be manually passed, and where to get them? Right, again shove them all into the constructor.

2. If (or when) the signature of the final classes changes, you'll have to come back here and make changes each time.

3. Not the worst but the most unpleasant - code written this way won't be easily discoverable by an IDE, leading to increased time spent on maintaining this code in the future. According to statistics, programmers spend about 90% of their time reading code and only about 10% writing it. Thus, such code will incur significant costs for the business due to developer salaries not being insignificant.

4. Lastly, there's no talk of any inversion of dependencies here at all.

Essentially, we've broken all the benefits that the excellent Symfony framework provides.

3) Manual retrieval from the container

Digging into the dependency container object and doing something similar to the previous approach. I won't detail how exactly to do this, but I'll just say that this approach is very close to what we're aiming for. However, we are still deviating from the original idea of how the framework should be used. Also, for this trick, we have to mark our classes as public in services.yaml or directly in the class files, or else they cannot be easily requested from the container. Framework developers consider this action risky, as it can lead to vulnerabilities in your code (for example, if you use third-party libraries with vulnerabilities, an attacker can gain access to the internals of your objects).

Perhaps you know or have encountered other ways to solve this problem; please write about them. I'm very interested.

4) Proposed solution

I should clarify that the example below is just a specific case of using the functionality of a compiler pass. This example shows that you can do very cool things without writing a multitude of configs and delving into the depths of the framework documentation.

1. We need to create an interface that will unify all our services, for example: IPaymentReportProvider.

2. Specify that each of our classes implements this interface. It's worth noting that you don't necessarily have to use an interface to mark the necessary set of classes; you can also use tags.

3. Next, go to services.yaml and add the following lines:

    services:
   
        # ...other config....
   
        ### Describe that our strategy must got list of all objects that implements our target interface
        App\Service\Payments\PaymentReportGenerator:
          arguments: [!tagged_iterator { tag: 'app.payment_report.provider', index_by: 'key' , default_index_method: 'getProviderName'}]
      #############
        _instanceof:
          App\Service\Payments\ReportProviders\IPaymentReportProvider:
            lazy: true
            tags: ['app.payment_report.provider']
   

The second block automatically creates, registers, and tags all classes implementing the IPaymentReportProvider interface with the tag - app.payment_report.provider. The lazy: true argument indicates that the classes will be initialized only when explicitly called, and a lightweight proxy object will be used when provided as a list.

The first part declares a service and specifies that one of the parameters to be provided to the object upon creation is a collection of objects tagged with -app.payment_report.provider. Two additional parameters indicate that we want to use a custom key as an index and specify which static method to invoke to get the key value for each element.

In essence, the strategy itself is already implemented using Symfony's features. Now, we need to implement/modify our context class (naming got from here).

class PaymentReportGenerator
{
    /**
     * @var IPaymentReportProvider[]
     */
    protected array $providers = [];

    public function __construct(iterable $providers)
    {
        $providers = $providers instanceof Traversable ? iterator_to_array($providers) : $providers;

        $this->providers = $providers;
    }

    public function generate(string $paymentProviderName, array $reportOptions):array
    {
        return $this->findPaymentProvider($paymentProviderName)->generateReport($reportOptions);
    }

    private function findPaymentProvider(string $paymentProviderName): IPaymentReportProvider
    {
        if (!isset($this->providers[$paymentProviderName])) {
            throw new InvalidArgumentException(sprintf(' Payment provider for "%s" not found', $paymentProviderName));
        }
        return $this->providers[$paymentProviderName];
    }
}

At the moment of object creation, Symfony gathers all classes with the specified tag and injects the collection into the constructor as the variable $providers.

Then, when the client method generate is called, we easily locate the desired provider and initiate the report generation (the object will only be initialized at the moment of calling the generateReport method).

Conclusion:

In this exploration of expanding payment methods in a Symfony project, we delved into various approaches to maintain code flexibility, adhere to SOLID principles, and facilitate future scalability. From direct parameter injection to dynamic service creation, we dissected the pros and cons of each strategy.

The introduction of Symfony features like compiler passes, tags, and lazy loading allowed us to implement a robust solution with minimal code modification, creating a system that dynamically adapts to the inclusion of new payment providers. The chosen strategy not only maintains the integrity of the code but also aligns with Symfony's design philosophy.

As developers, it's crucial to strike a balance between simplicity and extensibility. The discussed techniques not only address the immediate challenge of incorporating additional payment methods but also contribute to building a maintainable and scalable codebase.

Remember, the key to successful software design lies not only in solving current problems but in anticipating and accommodating future changes. By embracing Symfony's features intelligently, developers can create systems that evolve seamlessly, ensuring their projects stay resilient and adaptable in the face of ever-changing requirements.

1. Server with installed some Linux OS(I'm using Debian 11)

2. Installed docker + docker-compose

3. Installed Nginx

4. Domain registered on Cloudflare

Steps to set up:

1. Cloudflare - almost nothing

2. Create docker-compose.yaml on your server and copy content from here - https://hub.docker.com/_/ghost/ (docker-compose.yaml block)

version: '3.1'

services:

  ghost:
    image: ghost:5-alpine
    restart: always
    ports:
      - 2368:2368
    environment:
      database__client: mysql
      database__connection__host: db
      database__connection__user: root
      database__connection__password: example
      database__connection__database: ghost
      url: https://blog.vladverpeta.com
    volumes:
      - ghost:/var/lib/ghost/content

  db:
    image: mysql:8.0
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: example
    volumes:
      - db:/var/lib/mysql

volumes:
  ghost:
  db:

PS: Dont forgot setup firewall that will restrict any connections to your server except: 80, 22, 433 ports

1. Run docker-compose up -d

2. Edit/create - /etc/nginx/sites-enabled/{your-blog.com}

server {
    listen 80;
    server_name blog.vladverpeta.com;

    location / {
        proxy_pass http://localhost:2368;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
    }
}

Put eyes on lines 7 and 10, without them, you will have TOO_MANY_REDIRECTS error.

1. Run nginx -s reload to apply changes in nginx.

That's all, I hope it works for you!)

Don't hesitate to contact me - I'll try to help you🤓

• a single type of relational database
• a single message broker
• a unified framework in the context of a programming language
• etc.

This article will talk about testing Apache Kafka as a message broker.

Kafka is currently already used in the company and this technology is more scalable than the same RabbitMQ, so we are looking in its direction. Before we start implementing Kafka in our projects, we will test whether this technology covers our needs.

Requirements for a message broker:

• PHP compatibility
• Stable work, fault tolerance, high SLA
• Scalability
• Fast speed of sending/reading messages to the queue
• Confidence in message delivery
• Confidence in receiving and processing a message (ack / nack)

In fact, all this is already there and works great (at small and medium volumes) for RabbitMQ. But as I wrote above, we do not want to breed a zoo of technologies.

1. Choosing a client for PHP

The Kafka official website has several links to the integration repositories:

• https://github.com/EVODelavega/phpkafka
• https://github.com/arnaud-lb/php-rdkafka
• https://github.com/quipo/kafka-php
• https://github.com/michal-harish/kafka-php

Only one repository is "live" at the moment and has a lot of stars on GitHub - arnaud-lb / php-rdkafka.

2. Environment setup

In the 21st year, Docker is like a whip to Indiana Jones for a developer.

1. I went to hub.docker.com and found the most popular official Kafka image.
2. Further search in Google for a ready-made example of a docker-compose file with PHP, possibly even with Kafka. And about a surprise - the first line of the search result is Phillaf / php-kafka-demo. There is already an image from the first paragraph.

   git clone https://github.com/Phillaf/php-kafka-demo
   

3. After cloning the repository, run:

   docker-compose up -d
   

   From the first try, of course, the project did not come together (at least on my MacOs BigSur). In order to fix the error, it was enough to simply remove the well-defined version of Kafka.
4. Everything is ready for the application, but what about without an additional bonus? In order to better understand what is happening with an unknown tool, I decided to immediately look for a visual manager. There were several paid versions that needed to be installed on a PC. But that's not the true way) So I installed opensource admin panel. To do this, add a few lines to our docker-compose.yml

   kafka-ui:
    image: provectuslabs/kafka-ui
    container_name: kafka-ui
    ports:
      - "8080:8080"
    restart: always
    environment:
      - KAFKA_CLUSTERS_0_NAME=local
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092
      - KAFKA_CLUSTERS_0_ZOOKEEPER=zookeeper:2181
      - KAFKA_CLUSTERS_0_READONLY=false
   

5. Ready, set, start ...
   • docker-compose up -d
   • Go to http://localhost - Adds messages to our queue (topic).
   • Go to http://localhost:8080 - open's our admin panel - where we will see our topic and the first messages in it.
   • Go into the container (I personally prefer to use docker in phpStorm), but you can use the docker GUI as well as using the console (I will not dwell on this here, I think that those who are already interested in Kafka - will definitely be able to enter the Docker container) and run consumer:

     php ./public/consumer_low.php
     

     We will see our messages in the console.

Congratulations, you are now Kafka adepts.

3. We adjust the configuration according to our requirements.

As I wrote above, we are interested in full control over messages, our application wants to receive a message exactly 1 time, no more and no less. At the same time, we want to be sure that if in the process of processing a message, the consumer suddenly ends his work (as a result of deployment, an error, or a server crash, ...) then after restarting, this message will be processed, and will not sink into oblivion.

1. To do this, we need to turn off the automatic synchronization of Kafka about the saved offset:

   $conf->set('enable.auto.commit', 'false');
   

2. In order for the server to save offset, we must assign a unique identifier to the consumer so that when reconnecting, Kafka knows where we left off:

   $conf->set('group.id', 'group_1');
   

3. Start listening to the queue:

   $topic->consumeStart($partition, RD_KAFKA_OFFSET_STORED);
   

   1st parameter - number of the partition to be read (by default, the topic has the 1st partition and the value of the variable will be = 0). 2nd parameter - number - offset from which to start reading messages. There are 3 preset modes:

   • RD_KAFKA_OFFSET_BEGINNING - start reading from the first message in the partition
   • RD_KAFKA_OFFSET_END - read-only new messages that will appear in the queue only after the consumer is connected
   • RD_KAFKA_OFFSET_STORED - read from where you left off last time (this flag can only be used if - 'group.id' is specified). Here you can find this and a lot more configuration options.

4. Receiving a message from the queue:

   $msg = $topic->consume($partition, 1000);
   

5. After successful processing of the message (saving to the database or etc.), mark the message as read (increment the offset):

   $topic->offsetStore($partition, $msg->offset);
   

Voila, we have implemented consistent work with messages in the queue using Apache Kafka.

Link to the repository.

I would be glad to hear your questions or comments)