Incorrect webhooks behavior


#1

Hello,

I’m facing a strange behavior of webhooks: I got a project for which there’s only 1 webhook created (I’ve verified this by retrieving all webhooks for this project).
The issue is that I’m receiving wrong events.
I’ve marked one task of my project as completed and received webhook events immediately.
Here’s what’s wrong with these events:

  • there are 19 events arriving simultaneously
  • 18 out of 19 events are absolutely the same (duplicate) with different timestamps (they differ up to 1 sec)
[9] => Array
        (
            [id] => 647963095426456
            [date] => 2018-05-07T14:51:18.045Z
            [action] => changed
            [resource] => task
            [project_id] => 647960414544263
        )

    [10] => Array
        (
            [id] => 647963095426456
            [date] => 2018-05-07T14:51:18.770Z
            [action] => changed
            [resource] => task
            [project_id] => 647960414544263
        )

    [11] => Array
        (
            [id] => 647963095426456
            [date] => 2018-05-07T14:51:19.655Z
            [action] => changed
            [resource] => task
            [project_id] => 647960414544263
        )
  • events are not related to the change that I’ve just made but to the action that was done yesterday (added comment to one of tasks). You can see from timestamp that the date is yesterday and not today.
  • if I uncheck task (mark as not completed) I receive the same 19 events from yesterday’s action

Are there any ongoing problems with webhooks? I didn’t see this kind of issues for last several months.


#2

Hi there @Dmytro_TOLSTYI,

I’m circling back around to this. Is this still happening for you? My best guess as to what happened was that for some reason those particular webhook events didn’t appear on our side as successfully delivered - they will get rescheduled again for up to a week with an exponential backoff.

(“Successfully delivered” in this context means that the response from your webhooks handler is either 200 or 204. Any other response will appear to not have been successful to us.)

In addition to the timed exponential backoff retries, if there is another change that happens, the whole batch of events from after what looked like the last successful delivery will get retried. This means that I would expect changes to that task to trigger an immediate attempt to deliver, but will send more changes than just the most recent one (all the ones that didn’t appear to be successful forward to the most recent changes)

As another note, it’s unfortunately fairly common for one gesture from a user to appear as multiple webhook events. This is because we send a webhook for each property change on a task, so adding a comment might have “You changed the stories on this task” and “the last modification time on this task has changed” events. Since we don’t go into this much detail with the payloads of the webhook events we deliver, the only thing that looks different about these two events are the timestamps.

We might invest some time in improving this in the future, but for now, we generally recommend “debouncing” webhook events by waiting until it has been a few seconds since the last event was delivered, and then fetching the details of the task, rather than getting the task once for every event.


#3

Hi @Matt_Bramlage,

Thank you for your response and explanation of internal processes (this helps us, integrators, to better understand the product and how we should behave on our side).

I’ve just retried the same flow with checking task as “Complete” for the same project that was facing issues and, unfortunately (for your), lucky (for us) the issue is not reproducible anymore. As expected, 2 events arrived: task changed + story added.

Just to give you some more information about that incident here are my thoughts: it doesn’t really look like the situation you described (“exponential backoff retries”) because:

  1. I’m sure that I’ve been replying with 200 OK status for all these events from Asana (code was not touched and it’s working fine now)
  2. I’ve been triggering “complete” event for task A but I never received any event about this task (among the bunch of 19 events I was receiving there was nothing about the one I triggered)
  3. The same 19 events were sent no matter what task from this project I was triggering
  4. The issue was only with 1 project. Other projects webhooks were working fine
  5. I know that you send separate even for each change in the system (and it’s absolutely OK for me) but, as you can see from the dump in my initial message, these are not usual series of events: 18 of them are the same ‘task changed’. This didn’t happen before.
  6. For this project, I was not receiving any new notifications at all (I was having the same 19 events from the day before actions)

Anyway, from my side, I’m happy that the issue is not there anymore. I hope my details might help you to investigate this issue if you decide to go further with this.

If you got any additional questions I can try to help you, as far as I still got logs but I’m unable to reproduce the issue with fresh data.