Issue613

Title Commits notifications don't show up on bugs.p.o
Priority critical Status resolved
Superseder Nosy List berker.peksag, brett.cannon, ezio.melotti, maciej.szulik, ncoghlan, ned.deily
Assigned To ezio.melotti Topics

Created on 2017-02-14.14:04:39 by berker.peksag, last changed 2017-03-25.05:41:42 by ncoghlan.

Files
File name Uploaded Type Edit Remove
95193780-0b4b-11e7-8357-5888fe9c2898.headers brett.cannon, 2017-03-17.23:03:21 application/octet-stream
95193780-0b4b-11e7-8357-5888fe9c2898.json brett.cannon, 2017-03-17.23:03:55 application/json
95193780-0b4b-11e7-8357-5888fe9c2898.response brett.cannon, 2017-03-17.23:04:03 application/octet-stream
debug.diff berker.peksag, 2017-03-17.10:16:29 text/plain
debug_with_test.diff berker.peksag, 2017-03-17.11:06:15 text/plain
issue613.patch maciej.szulik, 2017-03-17.22:22:38 text/plain
Messages
msg3246 (view) Author: berker.peksag Date: 2017-02-14.14:04:38
See http://bugs.python.org/issue29438#msg287733 for example.

All PRs have been correctly set bpo-NNNN in the ir commit messages.
msg3247 (view) Author: berker.peksag Date: 2017-02-14.18:58:23
I got an email for http://bugs.python.org/issue29534

Mark Dickinson added the comment:


New changeset 6d1dece06d13a7d40637e07b2c79f34aab368766 by Mark Dickinson in branch 'master':
Fixed #29534 - _decimal difference with _pydecimal (#65)
https://github.com/python/cpython/commit/6d1dece06d13a7d40637e07b2c79f34aab368766


----------
_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue29534>

But it doesn't show up at http://bugs.python.org/issue29534

Note that "Mark Dickinson added the comment" should be "Roundup Robot added the comment".
msg3248 (view) Author: berker.peksag Date: 2017-02-14.21:13:47
> Note that "Mark Dickinson added the comment" should be "Roundup Robot added the comment".

Ezio told me that this was an intentional change. Sorry for the noise.

Brett, is there an easy way to resend payloads so we can debug this?
msg3249 (view) Author: brett.cannon Date: 2017-02-14.21:18:07
Re-sending a webhook event is easy ... if I can figure out which one to resend. :) There are tons of events going on so I need to know the delivery_id to easily find the event to resend.
msg3250 (view) Author: ezio.melotti Date: 2017-02-14.21:44:39
The commit message includes the pull request id, and it got accidentally captured by the regex that looks for issue numbers.  The code doesn't check if the issue id exists, and that caused the failure.

An easy fix is to add a try/except, but once we reach PR #1000 it will start matching and adding comments on old issues.  This might be good enough once we figure out a better solution.

Is the PR id always automatically added at the end of the commit message, between (...)?  If that's the case it might be easier to strip the end of the commit message, or only match for bpo-NNNNN once everyone starts using the convention.
msg3251 (view) Author: berker.peksag Date: 2017-02-14.21:48:36
> Is the PR id always automatically added at the end of the commit message, between (...)?

Yes, but backport PRs can have titles like

   Fix some sphinx warnings (#9) (#81)

#9 is the original PR number and #81 is the backport PR number.

See: https://github.com/python/cpython/commit/9268855ba6e99bf595e75b75780cf29667371f45
msg3252 (view) Author: ezio.melotti Date: 2017-02-14.22:09:46
I pushed the try/except fix in https://hg.python.org/tracker/roundup/rev/1e58dddad971
I don't have a delivery_id, but the event was pushed almost 3h ago, around 19:18:56 UTC.  See https://mail.python.org/pipermail/python-bugs-list/2017-February/326183.html
If you can find and resend the event, we should see the message popping up in http://bugs.python.org/issue29534 otherwise we can just wait for the next push.
msg3253 (view) Author: brett.cannon Date: 2017-02-14.22:15:09
It's delivery ID 6d377980-f2ea-11e6-8db5-e3e11d869c19 and a re-delivery worked!
msg3256 (view) Author: ezio.melotti Date: 2017-02-15.01:05:49
ISTM that best way to fix this is to just ignore all the (#NNN) at the end of the commit message.  The try/except should also stay to avoid failures in case of typos or other extraneous ids.

If the commit message references other PRs directly, it might cause false positives when we reach 1000+ PRs.  This is probably not going to happen very often, but, if it does, we might want to implement something smarter (e.g. avoid posting messages to issues that have been closed for over a year).
msg3258 (view) Author: maciej.szulik Date: 2017-02-15.16:22:41
Yeah, as soon as I fix the local testing of those things I'll get back to having 4-digit requirement for bpo issues, that should additionally save us from unnecessary errors, like the one above.
msg3278 (view) Author: berker.peksag Date: 2017-02-27.14:04:03
I can reproduce this again, reopening.
msg3279 (view) Author: berker.peksag Date: 2017-03-02.11:15:52
See https://github.com/python/cpython/commit/0dadf56737f591c83d18db5e445960d39448583e for example. The commit message contains bpo-29655 and there should be a comment similar to http://psf.upfronthosting.co.za/roundup/meta/msg3247 at https://bugs.python.org/issue29655

I haven't gotten any email this time.

The last message posted to bpo was http://bugs.python.org/issue29532#msg288347
msg3280 (view) Author: berker.peksag Date: 2017-03-03.13:08:31
Update: It looks like it sometimes works for 3.6 branch:

* http://bugs.python.org/issue28893#msg288853 (2017-03-03 06:16)

But not for master branch:

* https://bugs.python.org/issue9303#msg288877 (2017-03-03 13:11)
msg3281 (view) Author: brett.cannon Date: 2017-03-03.21:05:22
No notifications for two PRs on https://bugs.python.org/issue29455. I checked the webhook payload for the 3.6 commit and GitHub got a 400:

Connection: close
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Fri, 03 Mar 2017 20:58:19 GMT
Server: BaseHTTP/0.3 Python/2.6.6
Transfer-Encoding: chunked
Via: 1.1 psf.upfronthosting.co.za

The request was:

Request URL: https://bugs.python.org/pull_request
Request method: POST
content-type: application/json
Expect: 
User-Agent: GitHub-Hookshot/6b02022
X-GitHub-Delivery: 2076d100-0054-11e7-9584-1096a01ee5f0
X-GitHub-Event: push

{
  "ref": "refs/heads/3.6",
  "before": "21ce65aa67f0dc63002ab0a5fb21ef921cf5279e",
  "after": "9d07aceedabcdc9826489f8b9baffff056283bb3",
  "created": false,
  "deleted": false,
  "forced": false,
  "base_ref": null,
  "compare": "https://github.com/python/cpython/compare/21ce65aa67f0...9d07aceedabc",
  "commits": [
    {
      "id": "9d07aceedabcdc9826489f8b9baffff056283bb3",
      "tree_id": "bad6e533cdf6cdf15f9a6f5cda466e52e54d57b4",
      "distinct": true,
      "message": "bpo-29455: Mention coverage.py in trace module documentation (GH-435)\n\n(cherry picked from commit 5dfccb06dc513ae67fac5fee66356ad58a4de170)",
      "timestamp": "2017-03-03T12:58:17-08:00",
      "url": "https://github.com/python/cpython/commit/9d07aceedabcdc9826489f8b9baffff056283bb3",
      "author": {
        "name": "Brett Cannon",
        "email": "brettcannon@users.noreply.github.com",
        "username": "brettcannon"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "username": "web-flow"
      },
      "added": [

      ],
      "removed": [

      ],
      "modified": [
        "Doc/library/trace.rst"
      ]
    }
  ],
  "head_commit": {
    "id": "9d07aceedabcdc9826489f8b9baffff056283bb3",
    "tree_id": "bad6e533cdf6cdf15f9a6f5cda466e52e54d57b4",
    "distinct": true,
    "message": "bpo-29455: Mention coverage.py in trace module documentation (GH-435)\n\n(cherry picked from commit 5dfccb06dc513ae67fac5fee66356ad58a4de170)",
    "timestamp": "2017-03-03T12:58:17-08:00",
    "url": "https://github.com/python/cpython/commit/9d07aceedabcdc9826489f8b9baffff056283bb3",
    "author": {
      "name": "Brett Cannon",
      "email": "brettcannon@users.noreply.github.com",
      "username": "brettcannon"
    },
    "committer": {
      "name": "GitHub",
      "email": "noreply@github.com",
      "username": "web-flow"
    },
    "added": [

    ],
    "removed": [

    ],
    "modified": [
      "Doc/library/trace.rst"
    ]
  },
  "repository": {
    "id": 81598961,
    "name": "cpython",
    "full_name": "python/cpython",
    "owner": {
      "name": "python",
      "email": "",
      "login": "python",
      "id": 1525981,
      "avatar_url": "https://avatars0.githubusercontent.com/u/1525981?v=3",
      "gravatar_id": "",
      "url": "https://api.github.com/users/python",
      "html_url": "https://github.com/python",
      "followers_url": "https://api.github.com/users/python/followers",
      "following_url": "https://api.github.com/users/python/following{/other_user}",
      "gists_url": "https://api.github.com/users/python/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/python/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/python/subscriptions",
      "organizations_url": "https://api.github.com/users/python/orgs",
      "repos_url": "https://api.github.com/users/python/repos",
      "events_url": "https://api.github.com/users/python/events{/privacy}",
      "received_events_url": "https://api.github.com/users/python/received_events",
      "type": "Organization",
      "site_admin": false
    },
    "private": false,
    "html_url": "https://github.com/python/cpython",
    "description": "The Python programming language",
    "fork": false,
    "url": "https://github.com/python/cpython",
    "forks_url": "https://api.github.com/repos/python/cpython/forks",
    "keys_url": "https://api.github.com/repos/python/cpython/keys{/key_id}",
    "collaborators_url": "https://api.github.com/repos/python/cpython/collaborators{/collaborator}",
    "teams_url": "https://api.github.com/repos/python/cpython/teams",
    "hooks_url": "https://api.github.com/repos/python/cpython/hooks",
    "issue_events_url": "https://api.github.com/repos/python/cpython/issues/events{/number}",
    "events_url": "https://api.github.com/repos/python/cpython/events",
    "assignees_url": "https://api.github.com/repos/python/cpython/assignees{/user}",
    "branches_url": "https://api.github.com/repos/python/cpython/branches{/branch}",
    "tags_url": "https://api.github.com/repos/python/cpython/tags",
    "blobs_url": "https://api.github.com/repos/python/cpython/git/blobs{/sha}",
    "git_tags_url": "https://api.github.com/repos/python/cpython/git/tags{/sha}",
    "git_refs_url": "https://api.github.com/repos/python/cpython/git/refs{/sha}",
    "trees_url": "https://api.github.com/repos/python/cpython/git/trees{/sha}",
    "statuses_url": "https://api.github.com/repos/python/cpython/statuses/{sha}",
    "languages_url": "https://api.github.com/repos/python/cpython/languages",
    "stargazers_url": "https://api.github.com/repos/python/cpython/stargazers",
    "contributors_url": "https://api.github.com/repos/python/cpython/contributors",
    "subscribers_url": "https://api.github.com/repos/python/cpython/subscribers",
    "subscription_url": "https://api.github.com/repos/python/cpython/subscription",
    "commits_url": "https://api.github.com/repos/python/cpython/commits{/sha}",
    "git_commits_url": "https://api.github.com/repos/python/cpython/git/commits{/sha}",
    "comments_url": "https://api.github.com/repos/python/cpython/comments{/number}",
    "issue_comment_url": "https://api.github.com/repos/python/cpython/issues/comments{/number}",
    "contents_url": "https://api.github.com/repos/python/cpython/contents/{+path}",
    "compare_url": "https://api.github.com/repos/python/cpython/compare/{base}...{head}",
    "merges_url": "https://api.github.com/repos/python/cpython/merges",
    "archive_url": "https://api.github.com/repos/python/cpython/{archive_format}{/ref}",
    "downloads_url": "https://api.github.com/repos/python/cpython/downloads",
    "issues_url": "https://api.github.com/repos/python/cpython/issues{/number}",
    "pulls_url": "https://api.github.com/repos/python/cpython/pulls{/number}",
    "milestones_url": "https://api.github.com/repos/python/cpython/milestones{/number}",
    "notifications_url": "https://api.github.com/repos/python/cpython/notifications{?since,all,participating}",
    "labels_url": "https://api.github.com/repos/python/cpython/labels{/name}",
    "releases_url": "https://api.github.com/repos/python/cpython/releases{/id}",
    "deployments_url": "https://api.github.com/repos/python/cpython/deployments",
    "created_at": 1486754631,
    "updated_at": "2017-03-03T19:37:47Z",
    "pushed_at": 1488574698,
    "git_url": "git://github.com/python/cpython.git",
    "ssh_url": "git@github.com:python/cpython.git",
    "clone_url": "https://github.com/python/cpython.git",
    "svn_url": "https://github.com/python/cpython",
    "homepage": "https://www.python.org/",
    "size": 191699,
    "stargazers_count": 6378,
    "watchers_count": 6378,
    "language": "Python",
    "has_issues": false,
    "has_downloads": true,
    "has_wiki": false,
    "has_pages": false,
    "forks_count": 521,
    "mirror_url": null,
    "open_issues_count": 91,
    "forks": 521,
    "open_issues": 91,
    "watchers": 6378,
    "default_branch": "master",
    "stargazers": 6378,
    "master_branch": "master",
    "organization": "python"
  },
  "pusher": {
    "name": "brettcannon",
    "email": "brettcannon@users.noreply.github.com"
  },
  "organization": {
    "login": "python",
    "id": 1525981,
    "url": "https://api.github.com/orgs/python",
    "repos_url": "https://api.github.com/orgs/python/repos",
    "events_url": "https://api.github.com/orgs/python/events",
    "hooks_url": "https://api.github.com/orgs/python/hooks",
    "issues_url": "https://api.github.com/orgs/python/issues",
    "members_url": "https://api.github.com/orgs/python/members{/member}",
    "public_members_url": "https://api.github.com/orgs/python/public_members{/member}",
    "avatar_url": "https://avatars0.githubusercontent.com/u/1525981?v=3",
    "description": "Repositories related to the Python Programming language"
  },
  "sender": {
    "login": "brettcannon",
    "id": 54418,
    "avatar_url": "https://avatars3.githubusercontent.com/u/54418?v=3",
    "gravatar_id": "",
    "url": "https://api.github.com/users/brettcannon",
    "html_url": "https://github.com/brettcannon",
    "followers_url": "https://api.github.com/users/brettcannon/followers",
    "following_url": "https://api.github.com/users/brettcannon/following{/other_user}",
    "gists_url": "https://api.github.com/users/brettcannon/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/brettcannon/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/brettcannon/subscriptions",
    "organizations_url": "https://api.github.com/users/brettcannon/orgs",
    "repos_url": "https://api.github.com/users/brettcannon/repos",
    "events_url": "https://api.github.com/users/brettcannon/events{/privacy}",
    "received_events_url": "https://api.github.com/users/brettcannon/received_events",
    "type": "User",
    "site_admin": false
  },
  "distinct_commits": [
    {
      "id": "9d07aceedabcdc9826489f8b9baffff056283bb3",
      "tree_id": "bad6e533cdf6cdf15f9a6f5cda466e52e54d57b4",
      "distinct": true,
      "message": "bpo-29455: Mention coverage.py in trace module documentation (GH-435)\n\n(cherry picked from commit 5dfccb06dc513ae67fac5fee66356ad58a4de170)",
      "timestamp": "2017-03-03T12:58:17-08:00",
      "url": "https://github.com/python/cpython/commit/9d07aceedabcdc9826489f8b9baffff056283bb3",
      "author": {
        "name": "Brett Cannon",
        "email": "brettcannon@users.noreply.github.com",
        "username": "brettcannon"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "username": "web-flow"
      },
      "added": [

      ],
      "removed": [

      ],
      "modified": [
        "Doc/library/trace.rst"
      ]
    }
  ],
  "ref_name": "3.6"
}
msg3283 (view) Author: ncoghlan Date: 2017-03-08.06:43:00
https://github.com/python/cpython/pull/169 is another case where the "merged" status updated appeared on http://bugs.python.org/issue29537, but there was no notification comment.
msg3284 (view) Author: ned.deily Date: 2017-03-10.21:07:58
Anything we can do to expedite this?  It's a real problem with merge/commit information not making it into the bpo issues (other than the list of PR requests).
msg3286 (view) Author: berker.peksag Date: 2017-03-17.09:19:59
I believe the 400 status code comes from one of the "raise Reject()" lines in roundup/github.py. I will try to add some descriptive messages to them to help debug the issue.
msg3287 (view) Author: berker.peksag Date: 2017-03-17.10:16:29
Here is a patch that tweaks some exception messages and adds logging.error() calls in GitHubHandler class.
msg3288 (view) Author: berker.peksag Date: 2017-03-17.11:06:15
I converted Brett's payload example to a test case and I couldn't be able to create a failing test case. Here is a patch with a test.
msg3291 (view) Author: maciej.szulik Date: 2017-03-17.22:22:38
Brett, the most worrying thing is missing `X-Hub-Signature` header in the request from http://psf.upfronthosting.co.za/roundup/meta/msg3281. I've uploaded tweaked patch from Berker but I've mostly switched to logging, we don't want to simplify hacking our tracker, too much returning exact information what went wrong. Most importantly I've added X-GitHub-Delivery, so it's easier to track the actual requests on GH. Additionally I've fixed author matching if committer is GitHub, we ignore that information (this happens when using GH UI).
msg3292 (view) Author: maciej.szulik Date: 2017-03-17.22:25:30
Brett, can I ask you to send me directly (or attach here) some of those rejected requests? Based on the current one I don't see anything suspicious, but maybe I'm overlooking something...
msg3293 (view) Author: berker.peksag Date: 2017-03-17.22:39:28
Note that there is an open issue for the committer name thing: http://psf.upfronthosting.co.za/roundup/meta/issue619
msg3295 (view) Author: brett.cannon Date: 2017-03-17.23:02:46
First, don't worry about the x-github-signature missing. It's the last line of the headers shown in the GitHub UI and so it's quite possible i just botched my copy-and-paste.

And I will upload the headers, 400 response, and JSON body of the most recently failed message (95193780-0b4b-11e7-8357-5888fe9c2898 on 2017-03-17 12:54:51).
msg3302 (view) Author: maciej.szulik Date: 2017-03-23.22:13:17
I've added more logging: 

https://hg.python.org/tracker/roundup/rev/b5c6f372b227
msg3303 (view) Author: ezio.melotti Date: 2017-03-24.13:18:26
I've been looking at the logs overnight and these are the few errors I got so far (timestamps should be UTC):

[24/Mar/2017 08:23:44] "POST /tracker/pull_request HTTP/1.1" 200 -
ERROR:root:zero messages created for 31bb6800-106b-11e7-91cd-85bcd980655d

[24/Mar/2017 08:26:08] "POST /tracker/pull_request HTTP/1.1" 200 -
ERROR:root:zero messages created for 87901000-106b-11e7-8ffe-9f76236c3a84

[24/Mar/2017 11:20:42] "POST /tracker/pull_request HTTP/1.1" 200 -
ERROR:root:zero messages created for e9f50a80-1083-11e7-96e7-81ffe1ed7d05

[24/Mar/2017 11:21:46] "POST /tracker/pull_request HTTP/1.1" 200 -
ERROR:root:zero messages created for 0bee8d00-1084-11e7-8be1-b103588f1b04

(I've tried to find these on GitHub myself, but either they are well hidden, or I don't have access to it.)
msg3304 (view) Author: berker.peksag Date: 2017-03-24.15:52:22
I think you need to be an administrator on that repo to see the payloads.
msg3307 (view) Author: ezio.melotti Date: 2017-03-24.16:43:32
I asked for access, and looked at the payloads.  The 4 errors mentioned in the previous message are just push messages that don't reference any bpo issue, so they are not really errors (and they were delivered correctly).

I found older payloads that failed with a 400 and tried to redeliver them.
I was then able to get an error from Roundup: the problem only showed up when the author and committer were different people, and the code was trying to format a string using a syntax only supported by 2.7.  The tracker is still running on 2.6, so it was failing.

I now fixed it (https://hg.python.org/tracker/roundup/rev/7d5f2a9f5f34) and tried to redeliver a few payloads, and the deliver was successful.

Should I go back to redeliver all the payloads?  It might cause some noise and confusion, but at least we will have proper references in the issues.
msg3308 (view) Author: ned.deily Date: 2017-03-24.16:57:47
My instinct would be to deliver them.  Having a reasonably accurate record of pushes/commits in a b.p.o has proven to be crucially important to me over the years when I have to go back and investigate the history of some problem.
msg3309 (view) Author: berker.peksag Date: 2017-03-24.18:26:03
I agree with Ned. Thanks, Ezio!
msg3310 (view) Author: ezio.melotti Date: 2017-03-24.20:14:06
While going through the other failed request, I found and fixed another bug: https://hg.python.org/tracker/roundup/rev/f04116acd3d8
While fixing that bug and reading the code I also noticed another unrelated bug -- Maciej is taking care of it.
There's also another bug related to notification that will be fixed next.
msg3311 (view) Author: ezio.melotti Date: 2017-03-24.22:11:46
Other bug fixed: https://hg.python.org/tracker/roundup/rev/1da702d3cf41
Maciej also fixed https://hg.python.org/tracker/roundup/rev/8c49b7b2bc8f and https://hg.python.org/tracker/roundup/rev/e58c841100f8
I'm still going through all the failed requests and sending them, and so far they are all succeeding.
msg3312 (view) Author: ezio.melotti Date: 2017-03-25.01:05:16
I successfully redelivered all the requests, so all known bugs should be fixed now.
msg3313 (view) Author: ned.deily Date: 2017-03-25.01:46:27
Thanks, Ezio!  Having those messages there really makes a big difference!
msg3314 (view) Author: ncoghlan Date: 2017-03-25.05:41:42
Awesome, thank you all!
History
Date User Action Args
2017-03-25 05:41:42ncoghlansetmessages: + msg3314
2017-03-25 01:46:27ned.deilysetmessages: + msg3313
2017-03-25 01:05:17ezio.melottisetstatus: in-progress -> resolved
assignedto: ezio.melotti
messages: + msg3312
2017-03-24 22:11:46ezio.melottisetmessages: + msg3311
2017-03-24 20:14:06ezio.melottisetstatus: chatting -> in-progress
messages: + msg3310
2017-03-24 18:26:03berker.peksagsetmessages: + msg3309
2017-03-24 16:57:47ned.deilysetmessages: + msg3308
2017-03-24 16:43:33ezio.melottisetmessages: + msg3307
2017-03-24 15:52:22berker.peksagsetmessages: + msg3304
2017-03-24 13:18:27ezio.melottisetmessages: + msg3303
2017-03-23 22:13:17maciej.szuliksetmessages: + msg3302
2017-03-17 23:04:03brett.cannonsetfiles: + 95193780-0b4b-11e7-8357-5888fe9c2898.response
2017-03-17 23:03:55brett.cannonsetfiles: + 95193780-0b4b-11e7-8357-5888fe9c2898.json
2017-03-17 23:03:21brett.cannonsetfiles: + 95193780-0b4b-11e7-8357-5888fe9c2898.headers
2017-03-17 23:02:46brett.cannonsetmessages: + msg3295
2017-03-17 22:39:28berker.peksagsetmessages: + msg3293
2017-03-17 22:25:30maciej.szuliksetmessages: + msg3292
2017-03-17 22:22:38maciej.szuliksetfiles: + issue613.patch
messages: + msg3291
2017-03-17 11:06:16berker.peksagsetfiles: + debug_with_test.diff
messages: + msg3288
2017-03-17 10:16:30berker.peksagsetfiles: + debug.diff
messages: + msg3287
2017-03-17 09:20:00berker.peksagsetmessages: + msg3286
2017-03-10 21:07:59ned.deilysetpriority: bug -> critical
nosy: + ned.deily
messages: + msg3284
2017-03-08 06:43:01ncoghlansetnosy: + ncoghlan
messages: + msg3283
2017-03-03 21:05:23brett.cannonsetmessages: + msg3281
2017-03-03 13:08:32berker.peksagsetmessages: + msg3280
2017-03-02 11:15:52berker.peksagsetmessages: + msg3279
2017-02-27 14:04:03berker.peksagsetstatus: done-cbb -> chatting
messages: + msg3278
2017-02-15 16:22:41maciej.szuliksetmessages: + msg3258
2017-02-15 01:05:49ezio.melottisetstatus: chatting -> done-cbb
messages: + msg3256
2017-02-14 22:15:09brett.cannonsetmessages: + msg3253
2017-02-14 22:09:47ezio.melottisetmessages: + msg3252
2017-02-14 21:48:36berker.peksagsetmessages: + msg3251
2017-02-14 21:44:40ezio.melottisetmessages: + msg3250
2017-02-14 21:18:07brett.cannonsetmessages: + msg3249
2017-02-14 21:13:47berker.peksagsetnosy: + brett.cannon
messages: + msg3248
2017-02-14 18:58:23berker.peksagsetstatus: unread -> chatting
messages: + msg3247
2017-02-14 14:04:39berker.peksagcreate