Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
mumble's k5 comment watcher

By mumble in Meta
Sat Aug 06, 2011 at 10:55:38 PM EST
Tags: comment watching, email notification, shell script, free code (all tags)

Welcome to mumble's k5 comment watcher.

I was bored one day, and kept pressing refresh on my comment page (http://www.kuro5hin.org/user/mumble/comments), mentally subtracting the old scores and the reply counts from the current ones. I thought there has to be a better way - let's see what we can do.

And so was born ./comment-watch.sh


$ ./comment-watch.sh Zombie%20Jesus%20Christ
user = Zombie%20Jesus%20Christ
sleeptime = 10m
testing = false

Downloading: http://www.kuro5hin.org/user/Zombie%20Jesus%20Christ/comments
...........
2011-08-01  12:03:20 AM
--------
1)    http://www.kuro5hin.org/comments/2011/7/31/42717/7806/9#9    I registered.    [none / 0] Replies: 0
---
--------
..
2011-08-01  12:23:24 AM
--------
1)    http://www.kuro5hin.org/comments/2011/7/31/42717/7806/9#9    I registered.    [1.50 / 2] Replies: 0
1)    http://www.kuro5hin.org/comments/2011/7/31/42717/7806/9#9    I registered.    [none / 0] Replies: 0
---
--------
.................................
2011-08-01  05:54:55 AM
--------
1)    http://www.kuro5hin.org/comments/2011/7/31/42717/7806/20#20    I worked as a coder for six years without a degree    [none / 0] Replies: 0
---
--------
......
2011-08-01  06:55:08 AM
--------
1)    http://www.kuro5hin.org/comments/2011/7/31/634/57848/5#5    Did you try using a different user agent string?    [none / 0] Replies: 1
---
--------
..................
2011-08-01  09:55:40 AM
--------
2)    http://www.kuro5hin.org/comments/2011/7/31/42717/7806/20#20    I worked as a coder for six years without a degree    [none / 0] Replies: 1
2)    http://www.kuro5hin.org/comments/2011/7/31/42717/7806/20#20    I worked as a coder for six years without a degree    [none / 0] Replies: 0
---
8)    http://www.kuro5hin.org/comments/2011/7/30/104410/939/5#5    I Beta Tested dBase for the Classic Mac OS    [none / 0] Replies: 1
8)    http://www.kuro5hin.org/comments/2011/7/30/104410/939/5#5    I Beta Tested dBase for the Classic Mac OS    [none / 0] Replies: 0
---
--------
.................
2011-08-01  12:46:11 PM
--------
1)    http://www.kuro5hin.org/comments/2011/7/31/634/57848/5#5    Did you try using a different user agent string?    [none / 1] Replies: 1
1)    http://www.kuro5hin.org/comments/2011/7/31/634/57848/5#5    Did you try using a different user agent string?    [none / 0] Replies: 1
---
2)    http://www.kuro5hin.org/comments/2011/7/31/42717/7806/20#20    I worked as a coder for six years without a degree    [none / 1] Replies: 1
2)    http://www.kuro5hin.org/comments/2011/7/31/42717/7806/20#20    I worked as a coder for six years without a degree    [none / 0] Replies: 1
---
3)    http://www.kuro5hin.org/comments/2011/7/31/42717/7806/9#9    I registered.    [1.00 / 3] Replies: 0
3)    http://www.kuro5hin.org/comments/2011/7/31/42717/7806/9#9    I registered.    [1.50 / 2] Replies: 0
---
4)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/13#13    Ok it works now    [1.50 / 2] Replies: 0
4)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/13#13    Ok it works now    [none / 1] Replies: 0
---
5)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/12#12    Fuxor.    [1.50 / 2] Replies: 1
5)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/12#12    Fuxor.    [none / 1] Replies: 1
---
6)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/11#11    I got the UISplitViewController working, but...    [none / 1] Replies: 0
6)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/11#11    I got the UISplitViewController working, but...    [none / 0] Replies: 0
---
7)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/10#10    No.  Tell me about them. $    [none / 1] Replies: 0
7)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/10#10    No.  Tell me about them. $    [none / 0] Replies: 0
---
8)    http://www.kuro5hin.org/comments/2011/7/30/104410/939/5#5    I Beta Tested dBase for the Classic Mac OS    [none / 1] Replies: 1
8)    http://www.kuro5hin.org/comments/2011/7/30/104410/939/5#5    I Beta Tested dBase for the Classic Mac OS    [none / 0] Replies: 1
---
9)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/9#9    The iPad is progressing nicely.    [none / 1] Replies: 1
9)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/9#9    The iPad is progressing nicely.    [none / 0] Replies: 1
---
10)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/8#8    Maybe, if I can get the iPad working again.    [none / 1] Replies: 0
10)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/8#8    Maybe, if I can get the iPad working again.    [none / 0] Replies: 0
---
--------
................................................................................................................................
2011-08-02  10:10:04 AM
--------
1)    http://www.kuro5hin.org/comments/2011/8/1/163728/7830/1#1    What happened to the comment I just posted?    [none / 0] Replies: 0
---
2)    http://www.kuro5hin.org/comments/2011/8/1/172030/5608/1#1    Domain registrar clusterfuck    [none / 0] Replies: 0
---
--------

Let's have a look under the hood:
$ wget -q "http://www.kuro5hin.org/user/Zombie%20Jesus%20Christ/comments" -O wget.mdc
$ ./extract-comment-details.sh wget.mdc | head
1)    http://www.kuro5hin.org/comments/2011/8/2/16439/61945/2#2    Actually I figured out how to disable autocorrect    [1.00 / 2] Replies: 0
2)    http://www.kuro5hin.org/comments/2011/8/2/43117/46315/14#14    I always dress professionally when I work at home    [1.50 / 2] Replies: 1
3)    http://www.kuro5hin.org/comments/2011/8/2/43117/46315/13#13    I'm good with Android Platform Development    [none / 1] Replies: 0
4)    http://www.kuro5hin.org/comments/2011/8/2/131932/9247/3#3    That really sucks.  Here's why:    [none / 1] Replies: 1
5)    http://www.kuro5hin.org/comments/2011/8/2/12263/50986/1#1    Running php must be totally suicidal    [none / 0] Replies: 1
6)    http://www.kuro5hin.org/comments/2011/8/2/41217/72714/13#13    Seagull Networks installed ClamAV...    [none / 0] Replies: 1
7)    http://www.kuro5hin.org/comments/2011/8/2/41217/72714/6#6    I'm happy to take suggestions    [none / 0] Replies: 2
8)    http://www.kuro5hin.org/comments/2011/8/2/41217/72714/2#2    I've filed tons of thunderbird bug reports.    [none / 0] Replies: 1
9)    http://www.kuro5hin.org/comments/2011/7/31/42717/7806/28#28    Go read your own damn comment again.    [none / 0] Replies: 0
10)    http://www.kuro5hin.org/comments/2011/8/1/145121/1905/12#12    Does he know about your Kuro5hin addiction?    [none / 0] Replies: 1
$ ./extract-comment-details.sh wget.mdc > new-stats.mdc
$ ./display-comment-differences.sh new-stats.mdc new-stats.mdc
--------
--------
# OK. Nothing to see yet. Let's make something:
$ tail -27 new-stats.mdc > old-stats.mdc
$ ./display-comment-differences.sh new-stats.mdc old-stats.mdc
--------
1)    Actually I figured out how to disable autocorrect    [1.00 / 2] Replies: 0
---
2)    I always dress professionally when I work at home    [1.50 / 2] Replies: 1
---
3)    I'm good with Android Platform Development    [none / 1] Replies: 0
---
--------
# So we see, new-stats.mdc now has 3 new comments, compared to old-stats.mdc.
# Let's edit new-stats.mdc and change some scores and reply counts.
$ nano new-stats.mdc
$ ./display-comment-differences.sh new-stats.mdc old-stats.mdc
--------
1)    Actually I figured out how to disable autocorrect    [1.00 / 2] Replies: 0
---
2)    I always dress professionally when I work at home    [1.50 / 2] Replies: 1
---
3)    I'm good with Android Platform Development    [none / 1] Replies: 0
---
22)    I worked as a coder for six years without a degree    [none / 1] Replies: 7035
22)    I worked as a coder for six years without a degree    [none / 1] Replies: 1
---
26)    I got the UISplitViewController working, but...    [none / 1] Replies: 13
26)    I got the UISplitViewController working, but...    [none / 1] Replies: 0
---
28)    When will we elect an atheist President?    [2.00 / 50] Replies: 2
28)    When will we elect an atheist President?    [2.00 / 4] Replies: 2
---
--------

Single lines mean it is a new comment in new-stats that isn't in old-stats.
Double lines mean the comment is in both, but the score and or reply has changed. (first line of the pair is new-stats, the second is old-stats).

Also, I almost forgot, there is a display-url option.
$ ./display-comment-differences.sh new-stats.mdc old-stats.mdc u
--------
1)    http://www.kuro5hin.org/comments/2011/8/2/16439/61945/2#2    Actually I figured out how to disable autocorrect    [1.00 / 2] Replies: 0
---
2)    http://www.kuro5hin.org/comments/2011/8/2/43117/46315/14#14    I always dress professionally when I work at home    [1.50 / 2] Replies: 1
---
3)    http://www.kuro5hin.org/comments/2011/8/2/43117/46315/13#13    I'm good with Android Platform Development    [none / 1] Replies: 0
---
22)    http://www.kuro5hin.org/comments/2011/7/31/42717/7806/20#20    I worked as a coder for six years without a degree    [none / 1] Replies: 7035
22)    http://www.kuro5hin.org/comments/2011/7/31/42717/7806/20#20    I worked as a coder for six years without a degree    [none / 1] Replies: 1
---
26)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/11#11    I got the UISplitViewController working, but...    [none / 1] Replies: 13
26)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/11#11    I got the UISplitViewController working, but...    [none / 1] Replies: 0
---
28)    http://www.kuro5hin.org/comments/2011/7/30/174234/559/1#1    When will we elect an atheist President?    [2.00 / 50] Replies: 2
28)    http://www.kuro5hin.org/comments/2011/7/30/174234/559/1#1    When will we elect an atheist President?    [2.00 / 4] Replies: 2
---
--------
I don't know about your terminal, but in mine I can right click on the links, and it opens it in firefox. Quite useful.

An aside, you can make the display more compact if you comment out the two echo "---" lines in ./display-comment-differences.sh
Then your output will look like:
$ ./display-comment-differences.sh new-stats.mdc old-stats.mdc u
--------
1)    http://www.kuro5hin.org/comments/2011/8/2/16439/61945/2#2    Actually I figured out how to disable autocorrect    [1.00 / 2] Replies: 0
2)    http://www.kuro5hin.org/comments/2011/8/2/43117/46315/14#14    I always dress professionally when I work at home    [1.50 / 2] Replies: 1
3)    http://www.kuro5hin.org/comments/2011/8/2/43117/46315/13#13    I'm good with Android Platform Development    [none / 1] Replies: 0
22)    http://www.kuro5hin.org/comments/2011/7/31/42717/7806/20#20    I worked as a coder for six years without a degree    [none / 1] Replies: 7035
22)    http://www.kuro5hin.org/comments/2011/7/31/42717/7806/20#20    I worked as a coder for six years without a degree    [none / 1] Replies: 1
26)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/11#11    I got the UISplitViewController working, but...    [none / 1] Replies: 13
26)    http://www.kuro5hin.org/comments/2011/7/30/15485/3578/11#11    I got the UISplitViewController working, but...    [none / 1] Replies: 0
28)    http://www.kuro5hin.org/comments/2011/7/30/174234/559/1#1    When will we elect an atheist President?    [2.00 / 50] Replies: 2
28)    http://www.kuro5hin.org/comments/2011/7/30/174234/559/1#1    When will we elect an atheist President?    [2.00 / 4] Replies: 2
--------
It's just a personal preference thing.

OK. So I had that all up and working and then thought, what if I want to watch more than one person at once?
My code used new-stats.txt and old-stats.txt so if more than one ran at a time, they would step on each others toes.
So quick change, and now the code uses new-stats.$rawuser and old-stats.$rawuser
$rawuser is just $user with space swapped out to %20.
This is the line of code:
rawuser=`echo "$user" | sed 's/ /%20/g'`

So now I had 5 of these running on my screen, in different terminal windows, and it was getting a little hard to keep track (especially since they look best when the terminal is almost the width of my 21" screen, so the lines aren't wrapped).
Surely I can watch them all from one central location.
And so was born: ./multi-watch.sh
The thing about ./multi-watch.sh is that it doesn't actually download anything from k5. It just makes use of the data already being supplied by multiple comment-watch.sh running in the background. So you start up as many comment-watch.sh as you want, one for each user, then minimize them down. And then watch them all using just multi-watch. And since it is not itself hitting k5 you can make it as fast as you like, eg 15s.

Something like this:
$ ./multi-watch.sh 15s mumble "Del Griffith" N0574 Pnarp "Zombie Jesus Christ" donnalee "king of fools" "horseskin spacesuit"
Multi-watch started.
sleeptime = 15s
Watching: mumble, Del Griffith, N0574, Pnarp, Zombie Jesus Christ, donnalee, king of fools, horseskin spacesuit,

....................................................................................................................................
2011-07-31  06:37:54 AM
Zombie Jesus Christ:
--------
1)    http://www.kuro5hin.org/comments/2011/7/30/11245/6917/1#1    ARE WE NOT MEN?    [none / 1] Replies: 0
1)    http://www.kuro5hin.org/comments/2011/7/30/11245/6917/1#1    ARE WE NOT MEN?    [none / 0] Replies: 0
---
--------
.......
2011-07-31  06:39:41 AM
donnalee:
--------
1)    http://www.kuro5hin.org/comments/2011/7/30/11245/6917/2#2    punk swings    [none / 1] Replies: 0
1)    http://www.kuro5hin.org/comments/2011/7/30/11245/6917/2#2    punk swings    [none / 0] Replies: 0
---
--------
..............
2011-07-31  06:43:15 AM
N0574:
--------
1)    http://www.kuro5hin.org/comments/2011/7/30/114814/523/2#2    I think you mean Robert Silverberg    [none / 0] Replies: 0
---
--------
......................................................................................................................................
2011-07-31  07:16:54 AM
mumble:
--------
3)    http://www.kuro5hin.org/comments/2011/7/29/232743/844/12#12    trolling can be indistinguishable from idiocy. $    [none / 1] Replies: 0
3)    http://www.kuro5hin.org/comments/2011/7/29/232743/844/12#12    trolling can be indistinguishable from idiocy. $    [none / 0] Replies: 0
---
--------
.........................
2011-07-31  07:23:12 AM
N0574:
--------
1)    http://www.kuro5hin.org/comments/2011/7/30/114814/523/2#2    I think you mean Robert Silverberg    [none / 0] Replies: 1
1)    http://www.kuro5hin.org/comments/2011/7/30/114814/523/2#2    I think you mean Robert Silverberg    [none / 0] Replies: 0
---
--------
..........................
2011-07-31  07:29:46 AM
king of fools:
--------
1)    http://www.kuro5hin.org/comments/2011/7/29/174352/039/15#15    yes #    [3.00 / 2] Replies: 0
1)    http://www.kuro5hin.org/comments/2011/7/29/174352/039/15#15    yes #    [none / 1] Replies: 0
---
--------
.
2011-07-31  07:30:02 AM
donnalee:
--------
2)    http://www.kuro5hin.org/comments/2011/7/30/85011/4293/4#4    so, you have to be a rabid capitalist to get help?    [1.50 / 2] Replies: 1
2)    http://www.kuro5hin.org/comments/2011/7/30/85011/4293/4#4    so, you have to be a rabid capitalist to get help?    [none / 1] Replies: 1
---
--------
....
2011-07-31  07:31:04 AM
Del Griffith:
--------
1)    http://www.kuro5hin.org/comments/2011/7/29/232743/844/10#10    no it's in my signature.    [none / 0] Replies: 2
1)    http://www.kuro5hin.org/comments/2011/7/29/232743/844/10#10    no it's in my signature.    [none / 0] Replies: 1
---
--------
........................................................................................................

Note, there is one . for each loop of the while loop - 15s in this example.

OK. So that was all working beautifully, and then I thought, why can't I watch EVERYONE!
That is where this page comes in: http://www.kuro5hin.org/?op=search&offset=0&old_count=30&type=comment&section=&string=&search=Search&count=30
(NB: when I download this page in firefox it is quite slow - I suspect it hits the k5 backend quite hard.)

And so was born ./global-watch.sh

$ ./global-watch.sh 1 10m
pages = 1
sleeptime = 10m

http://www.kuro5hin.org/?op=search&offset=0&old_count=30&type=comment&section=&string=&search=Search&count=30
......Trying again ...
.
2011-08-01  11:25:39 PM
--------
24)    http://www.kuro5hin.org/comments/2011/7/31/2110/22208/2#2      Pnarp:         Actually...    [3.00 / 2] Replies: 0
24)    http://www.kuro5hin.org/comments/2011/7/31/2110/22208/2#2      Pnarp:         Actually...    [none / 1] Replies: 0
---
--------
...Trying again ...

2011-08-01  11:56:42 PM
--------
1)    http://www.kuro5hin.org/comments/2011/7/31/16418/6462/4#4      Zombie Gautama Buddha:         Lead by example. $    [none / 0] Replies: 0
---
28)    http://www.kuro5hin.org/comments/2011/7/31/16418/6462/3#3      Enlarged to Show Texture:         Kill yourself $    [none / 0] Replies: 1
27)    http://www.kuro5hin.org/comments/2011/7/31/16418/6462/3#3      Enlarged to Show Texture:         Kill yourself $    [none / 0] Replies: 0
---
--------
..
2011-08-02  12:17:05 AM
--------
1)    http://www.kuro5hin.org/comments/2011/7/31/21321/0596/7#7      Delirium:         what the fuck, man    [none / 0] Replies: 0
---
--------
.....
2011-08-02  01:07:15 AM
--------
11)    http://www.kuro5hin.org/comments/2011/7/31/21321/0596/6#6      krkrbt:         good to know, think I'm done with that game    [none / 1] Replies: 0
11)    http://www.kuro5hin.org/comments/2011/7/31/21321/0596/6#6      krkrbt:         good to know, think I'm done with that game    [none / 0] Replies: 0
---
--------
....
2011-08-02  01:48:02 AM
--------
11)    http://www.kuro5hin.org/comments/2011/7/31/21321/0596/6#6      krkrbt:         good to know, think I'm done with that game    [3.00 / 2] Replies: 0
11)    http://www.kuro5hin.org/comments/2011/7/31/21321/0596/6#6      krkrbt:         good to know, think I'm done with that game    [none / 1] Replies: 0
---
--------
.
2011-08-02  01:58:22 AM
--------
1)    http://www.kuro5hin.org/comments/2011/7/31/21321/0596/9#9      LilDebbie:         might i recommend    [none / 0] Replies: 0
---
2)    http://www.kuro5hin.org/comments/2011/7/31/21321/0596/8#8      LilDebbie:         with as much fuel ore as they can eat $    [none / 0] Replies: 0
---
3)    http://www.kuro5hin.org/comments/2011/7/31/634/57848/11#11      mumble:         Don't ask my brain ...    [none / 0] Replies: 0
---
4)    http://www.kuro5hin.org/comments/2011/7/31/634/57848/10#10      mumble:         I just now got back to this ...    [none / 0] Replies: 0
---
5)    http://www.kuro5hin.org/comments/2011/7/31/21321/0596/7#7      Delirium:         what the fuck, man    [none / 1] Replies: 1
1)    http://www.kuro5hin.org/comments/2011/7/31/21321/0596/7#7      Delirium:         what the fuck, man    [none / 0] Replies: 0
---
15)    http://www.kuro5hin.org/comments/2011/7/31/21321/0596/6#6      krkrbt:         good to know, think I'm done with that game    [3.00 / 3] Replies: 0
11)    http://www.kuro5hin.org/comments/2011/7/31/21321/0596/6#6      krkrbt:         good to know, think I'm done with that game    [3.00 / 2] Replies: 0
---
19)    http://www.kuro5hin.org/comments/2011/7/31/21321/0596/3#3      GreyGhost:         WTF are you talking about?    [none / 1] Replies: 1
15)    http://www.kuro5hin.org/comments/2011/7/31/21321/0596/3#3      GreyGhost:         WTF are you talking about?    [none / 0] Replies: 1
---
23)    http://www.kuro5hin.org/comments/2011/8/1/11613/32618/1#1      N0574:         wait wait one damn minute,    [none / 1] Replies: 1
19)    http://www.kuro5hin.org/comments/2011/8/1/11613/32618/1#1      N0574:         wait wait one damn minute,    [none / 0] Replies: 1
---
25)    http://www.kuro5hin.org/comments/2011/7/31/21321/0596/2#2      United Fools:         How do you pay your fighters?    [none / 1] Replies: 1
21)    http://www.kuro5hin.org/comments/2011/7/31/21321/0596/2#2      United Fools:         How do you pay your fighters?    [none / 0] Replies: 0
---
--------
...
2011-08-02  02:28:46 AM
--------
1)    http://www.kuro5hin.org/comments/2011/8/1/74324/58498/1#1      mumble:         still cold here!     [none / 0] Replies: 0
---
--------
.
2011-08-02  02:39:05 AM
--------
9)    http://www.kuro5hin.org/comments/2011/7/30/41212/6641/9#9      Zombie Gautama Buddha:         I bet it doesn't weigh as much as my old    [none / 1] Replies: 0
9)    http://www.kuro5hin.org/comments/2011/7/30/41212/6641/9#9      Zombie Gautama Buddha:         I bet it doesn't weigh as much as my old    [none / 0] Replies: 0
---
--------
.

The observant among you may have noticed "Trying again ...". This is due to k5 being a pest. Every now and then it just feeds you back a zero length page. I use an if to retry, but I guess in the case of ./global-watch.sh you could use a while loop. (You can't use a while loop for ./comment-watch.sh since it will be an infinite loop if the user does not exist).
Here is what I use in ./download-global-html.sh
if [ `grep -c "^Replies" wget.g.$rand` = 0 ] ; then
# Try again!

Let's look under the hood of ./global-watch.sh

$ ./download-global-html.sh

Usage: ./download-global-html.sh long.html [pages]

$ ./download-global-html.sh wget.global
http://www.kuro5hin.org/?op=search&offset=0&old_count=30&type=comment&section=&string=&search=Search&count=30
# ie, it defaults to downloading only 1 page of 30 comments.
$ ./download-global-html.sh wget.global 3
http://www.kuro5hin.org/?op=search&offset=0&old_count=30&type=comment&section=&string=&search=Search&count=30
http://www.kuro5hin.org/?op=search&offset=0&old_count=30&type=comment&section=&string=&count=30&next=Next+Page+%3E%3E
http://www.kuro5hin.org/?op=search&offset=30&old_count=30&type=comment&section=&string=&count=30&next=Next+Page+%3E%3E
# So we see it is easy to download multiple pages too.
# Then to extract the details from wget.global, we again use ./extract-comment-details.sh but with an extra parameter (that can be anything).
# So we have:
$ ./extract-comment-details.sh wget.global users | head
1)    http://www.kuro5hin.org/comments/2011/8/3/14628/43453/6#6      horseskin spacesuit:         we'd hear it from the people in the town    [none / 0] Replies: 0
2)    http://www.kuro5hin.org/comments/2011/8/1/145121/1905/18#18      sye:         moral high ground    [none / 0] Replies: 0
3)    http://www.kuro5hin.org/comments/2011/8/2/211536/1126/18#18      claes:         'internal': That's really the key.    [none / 0] Replies: 0
4)    http://www.kuro5hin.org/comments/2011/8/2/234816/2043/4#4      Marvin Suggs:         living the life    [none / 0] Replies: 0
5)    http://www.kuro5hin.org/comments/2011/8/2/211536/1126/17#17      Del Griffith:         all the more reason to do it.    [none / 0] Replies: 0
6)    http://www.kuro5hin.org/comments/2011/8/3/83028/37332/1#1      Del Griffith:         don't worry the day is still young    [none / 1] Replies: 0
7)    http://www.kuro5hin.org/comments/2011/8/2/16734/72444/8#8      Del Griffith:         I don't get teos    [none / 0] Replies: 0
8)    http://www.kuro5hin.org/comments/2011/8/2/16734/72444/7#7      Del Griffith:         Try the computer    [none / 0] Replies: 0
9)    http://www.kuro5hin.org/comments/2011/8/2/16439/61945/4#4      Lexx Core:         Imitation MDC    [none / 0] Replies: 0
10)    http://www.kuro5hin.org/comments/2011/8/3/14628/43453/5#5      schlouse:         Also: Biblical Michael Crawford $    [none / 0] Replies: 0
# Now it also gives us the commenters name. This is important in ./global-watch.sh so we know who posted what.
# This is not needed for ./comment-watch.sh or ./multi-watch.sh
# And without the extra parameter we are back at:
$ ./extract-comment-details.sh wget.global | head -5
1)    http://www.kuro5hin.org/comments/2011/8/3/14628/43453/6#6    we'd hear it from the people in the town    [none / 0] Replies: 0
2)    http://www.kuro5hin.org/comments/2011/8/1/145121/1905/18#18    moral high ground    [none / 0] Replies: 0
3)    http://www.kuro5hin.org/comments/2011/8/2/211536/1126/18#18    'internal': That's really the key.    [none / 0] Replies: 0
4)    http://www.kuro5hin.org/comments/2011/8/2/234816/2043/4#4    living the life    [none / 0] Replies: 0
5)    http://www.kuro5hin.org/comments/2011/8/2/211536/1126/17#17    all the more reason to do it.    [none / 0] Replies: 0

./display-comment-differences.sh works fine with the output in either case.
That is because the URL and user-name are separated by spaces rather than tab, so as far as cut -f2 is concerned, it is only one column.
I guess I can give some examples:
$ ./extract-comment-details.sh wget.global | head -3 | cut -f1
1)
2)
3)
$ ./extract-comment-details.sh wget.global | head -3 | cut -f2
http://www.kuro5hin.org/comments/2011/8/3/14628/43453/6#6
http://www.kuro5hin.org/comments/2011/8/1/145121/1905/18#18
http://www.kuro5hin.org/comments/2011/8/2/211536/1126/18#18
$ ./extract-comment-details.sh wget.global | head -3 | cut -f3
we'd hear it from the people in the town
moral high ground
'internal': That's really the key.
$ ./extract-comment-details.sh wget.global | head -3 | cut -f4
[none / 0] Replies: 0
[none / 0] Replies: 0
[none / 0] Replies: 0
# And now with username display on:
$ ./extract-comment-details.sh wget.global u | head -3 | cut -f1
1)
2)
3)
$ ./extract-comment-details.sh wget.global u | head -3 | cut -f2
http://www.kuro5hin.org/comments/2011/8/3/14628/43453/6#6      horseskin spacesuit:    
http://www.kuro5hin.org/comments/2011/8/1/145121/1905/18#18      sye:    
http://www.kuro5hin.org/comments/2011/8/2/211536/1126/18#18      claes:    
$ ./extract-comment-details.sh wget.global u | head -3 | cut -f3
we'd hear it from the people in the town
moral high ground
'internal': That's really the key.
$ ./extract-comment-details.sh wget.global u | head -3 | cut -f4
[none / 0] Replies: 0
[none / 0] Replies: 0
[none / 0] Replies: 0

Finally, at any time we can look at a posters newest comments by doing head new-stats.$rawuser. eg:
$ head -6 new-stats.Zombie%20Jesus%20Christ
1)    http://www.kuro5hin.org/comments/2011/8/2/16439/61945/2#2    Actually I figured out how to disable autocorrect    [1.00 / 2] Replies: 0
2)    http://www.kuro5hin.org/comments/2011/8/2/43117/46315/14#14    I always dress professionally when I work at home    [1.50 / 2] Replies: 1
3)    http://www.kuro5hin.org/comments/2011/8/2/43117/46315/13#13    I'm good with Android Platform Development    [1.50 / 2] Replies: 0
4)    http://www.kuro5hin.org/comments/2011/8/2/131932/9247/3#3    That really sucks.  Here's why:    [none / 1] Replies: 1
5)    http://www.kuro5hin.org/comments/2011/8/2/12263/50986/1#1    Running php must be totally suicidal    [none / 0] Replies: 1
6)    http://www.kuro5hin.org/comments/2011/8/2/41217/72714/13#13    Seagull Networks installed ClamAV...    [none / 0] Replies: 1

Or, we have ./show-stats.sh if you prefer.
$ ./show-stats.sh

Usage: ./show-stats.sh username [u|.] [count]

$ ./show-stats.sh "Zombie Jesus Christ" . 6
1)    Actually I figured out how to disable autocorrect    [1.00 / 2] Replies: 0
2)    I always dress professionally when I work at home    [1.50 / 2] Replies: 1
3)    I'm good with Android Platform Development    [1.50 / 2] Replies: 0
4)    That really sucks.  Here's why:    [none / 1] Replies: 1
5)    Running php must be totally suicidal    [none / 0] Replies: 1
6)    Seagull Networks installed ClamAV...    [none / 0] Replies: 1
$ ./show-stats.sh "Zombie Jesus Christ" u 6
1)    http://www.kuro5hin.org/comments/2011/8/2/16439/61945/2#2    Actually I figured out how to disable autocorrect    [1.00 / 2] Replies: 0
2)    http://www.kuro5hin.org/comments/2011/8/2/43117/46315/14#14    I always dress professionally when I work at home    [1.50 / 2] Replies: 1
3)    http://www.kuro5hin.org/comments/2011/8/2/43117/46315/13#13    I'm good with Android Platform Development    [1.50 / 2] Replies: 0
4)    http://www.kuro5hin.org/comments/2011/8/2/131932/9247/3#3    That really sucks.  Here's why:    [none / 1] Replies: 1
5)    http://www.kuro5hin.org/comments/2011/8/2/12263/50986/1#1    Running php must be totally suicidal    [none / 0] Replies: 1
6)    http://www.kuro5hin.org/comments/2011/8/2/41217/72714/13#13    Seagull Networks installed ClamAV...    [none / 0] Replies: 1

You can do other things with this data too.
eg: $ head -30 new-stats.Zombie%20Jesus%20Christ | cut -f2
This gives a list of the URL's of MDC's last 30 comments. Then you could feed it to wget, and download these comments (something that might be useful to donnalee too).

Inside all of comment-watch.sh, multi-watch.sh and global-watch.sh there is code roughly equivalent to this:
    if [ -f beep.file ] ; then
       printf "\07"; sleep 1; printf "\07"
    fi

    if [ -f date.file ] ; then
      date +"%Y-%m-%d  %r"
    fi

    # Put in switch for display url+username on and off.
    if [ -f url.file ] ; then
      ./display-comment-differences.sh new-stats.global old-stats.global u
    else
      ./display-comment-differences.sh new-stats.global old-stats.global
    fi

For a start, this means I can switch on and off, beeping, display of date, and url vs non-url display just by doing touch beep.file or rm beep.file and so on, from another terminal.
In particular, I like to leave the code running overnight, so when I go to bed I just type "rm beep.file". When I am back at the computer I re-enable beeping with "touch beep.file"

But, this is just the start. Instead of a system beep, we could play a sound:
eg: aplay -q /usr/share/sounds/purple/receive.wav

Or, if you have commandline email set up, you could send $user, date/time and the output of ./display-comment-differences.sh to your chosen email account.
Instant, "I get sent an email each time someone rates or replies to any of my last 30 comments".
I don't have command line email set up, I don't need it, but lets play along.
By copying from here: http://www.simplehelp.net/2008/12/01/how-to-send-email-from-the-linux-command-line/
I would put something like this inside ./comment-watch.sh

if [ "$user" = "mumble" ] ; then   # So only get notified of changes to me. I don't want to be emailed about others.
  ./display-comment-differences.sh "new-stats.$rawuser" "old-stats.$rawuser" u | mail -s "k5 notification for $user" me@example.com
fi

Or, use your imagination. Any form of notification that you can think of, that can be done on the command line, you can put in here.

And that is about it. I will mention that in practice I use comment-watch.sh, multi-watch.sh and global-watch.sh all at the same time.
They each perform slightly different functions. multi-watch watches n*30 comments, for n users, while global-watch only watches pages*30 of the latest comments.
This means global-watch is great for keeping track of new comments, no matter from whom, but multi-watch is better if you are interested in particular users.
And if you want to look at just one user, then go back to the appropriate comment-watch.sh terminal.

The good news is I am releasing this code under a copyright license of: "do what the f... you want with it, but you are encouraged to share any modifications and improvements".
Bad news is you probably need to understand shell script to get it working, and probably won't work on your system anyway!! It works on mine, and is fairly well bug tested, but only on my linux box (with bash as the shell). If you are on windows, even if you have cygwin, the not very wide terminal is going to wrap the lines and make it look ugly and hard to read, even if by some miracle you get it to run. It may run on other *nix. I did a quick test on FreeBSD and it won't work without tweaking because it needs $RANDOM and seq.
(I guess in theory the code will still work even without $RANDOM (which gives a different random number each time it is invoked), but at the increased risk that if you have more than one watch running at once that they will step on each others working files.)

Here is the code to test them:
-----
$ cat test-rand-and-seq.sh
#!/bin/sh

rand=$RANDOM

if [ "$rand" = "" ] || [ "$rand" = "$RANDOM" ] ; then
  echo "\$RANDOM doesn't seem to work on this box!"
else
  echo "\$RANDOM seems to work on this box."
fi

echo "If seq works, you should see a list of numbers from 3 to 11"
seq 3 11
-----
$RANDOM is used in: extract-comment-details.sh, download-global-html.sh
seq is used in: extract-comment-details.sh, download-global-html.sh, display-comment-differences.sh (actually, in the current code, this is commented out).

FTW:
$ grep "\$RANDOM" *.sh
$ grep "seq " *.sh

To get $RANDOM to work in FeeBSD I used (bash supports the $RANDOM variable, but sh does not):
#!/usr/local/bin/bash
There is no seq, but man jot is the way to go.

Also need wget on your system. If you use something different, you have to manually swap out wget from the code yourself.

Also, non-logged in wget sees slightly different comments from a logged in session. Some of these are because they are hidden comments, but some of them are due to bizzare k5 behaviour.
I decided not to fix this, as it is minor, but you could tack this on the end of all the urls: ?uname=$k5user&pass=$k5pass, or do something with cookies.

Good luck! I hope it works for at least some others. It really should be in perl or python or something, but it is not.
If anyone is brave or stupid enough to try and make it more portable, that would be great!

--------------
Code, and this description page available here: http://www.2shared.com/file/aDZgB1Sw/comment-watch.html

It is a zip file because I couldn't get them to accept a .tar.gz. So you will need to do a chmod +x *.sh first. Enjoy.

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
will you use the comment watcher?
o yes. 25%
o no. 37%
o maybe 37%
o it doesn't work on my platform 0%
o I'm stuck on windows 0%
o WIPO 0%

Votes: 8
Results | Other Polls

Related Links
o Kuro5hin
o Also by mumble


Display: Sort:
mumble's k5 comment watcher | 12 comments (8 topical, 4 editorial, 0 hidden)
Thank You For Your Support! (2.50 / 6) (#1)
by Zombie Jesus Christ on Thu Aug 04, 2011 at 12:51:05 PM EST

But may I suggest that if it's as sunny and beautiful day where you are as it is where I am, maybe some time outside away from the Internet would do you some good.

Who knows, maybe you could meet a nice girl.

--
Mike Crawford for Clark County Commissioner
District 1 North County
mike@communard.org

Paid for by The Communard Party of Washington State


NO YUO (3.00 / 2) (#3)
by Enlarged to Show Texture on Fri Aug 05, 2011 at 11:19:11 AM EST




"Those people who think they know everything are a great annoyance to those of us who do." -- Isaac Asimov
[ Parent ]
Awesome! "Push technology" is back! (3.00 / 3) (#2)
by Blarney on Fri Aug 05, 2011 at 11:17:21 AM EST

Too bad that RSS hasn't been extended to follow comment discussion in some sort of standard-compliant way. I guess you've managed to hack K5 into doing what USENET has been doing for all this time. But nobody posts on USENET anymore except for lunatics, so obviously it'd be unfair to compare it to K5!

This is all Web 1.0ey though. What we need is for K5 to be linked to something for normal people - maybe Facebook would be a good platform - and we could all join the appropriate Facebook group and be notified in realtime every time that someone on K5 made a comment. Probably Rusty has to fix his site to make that work correctly. Unless someone wants to run a server which constantly scrapes K5 and uses the info to update a Facebook mirror group. Or maybe it'd be even easier to just create a Facebook group, call it Kuro6hin or something and we all go there! It'd be awesome. I wonder how tolerant FB is of sodomistic horsecock-powered schizoaffective AIDS-infected copyright undistributable open source content.



Twitter. (none / 1) (#4)
by claes on Fri Aug 05, 2011 at 11:40:33 AM EST

Each and every comment tweeted in its entirety, even if it's a MDC rant that takes 1000 tweets.

[ Parent ]
not a million miles ... (none / 1) (#5)
by mumble on Fri Aug 05, 2011 at 12:04:42 PM EST

Not a million miles from possible, actually.

I just did a google search for "twitter command line", and there seems to be several. And that is about all we need to get the job done.

Inside ./display-comment-differences.sh
There is one place where it has:
echo "$nline"
and another has:
echo "$nline"
echo "$oline"

Then we just have to twitter that info. eg:
14)    http://www.kuro5hin.org/comments/2011/8/5/64556/11920/33#33      Phssthpok:         oh, if i just watch myself    [none / 0] Replies: 1

But in this script we also have $nurl, which is just the URL of the comment. So we can download the comment, then use something to extract the comment text from there, split it into pieces, and then twitter that.

Call the script ./twitter-comment-differences.sh and then put it in the desired places in the *-watch.sh scripts.

-----
stats for a better tomorrow
bitcoin: 1GsfkeggHSqbcVGS3GSJnwaCu6FYwF73fR
"They must know I'm here. The half and half jug is missing" - MDC.
"I've grown weary of googling the solutions to my many problems" - MDC.
[ Parent ]

I can't imagine... (none / 0) (#11)
by jolt rush soon on Tue Aug 09, 2011 at 07:57:42 AM EST

...what that would be like.
--
Subosc — free electronic music.
[ Parent ]
Voting this up (3.00 / 4) (#6)
by Ezra Loomis Pound on Fri Aug 05, 2011 at 01:55:54 PM EST

because it actually features useful content and shows how completely self-obsessed you people are.

:::"Let me tell ya, if she wasn't cut out to handle some fake boy online, well sister, life only gets more difficult, and you only get more emo as you age." --balsamic vinigga :::#_#:::
new code here: (none / 0) (#12)
by mumble on Fri Aug 19, 2011 at 01:54:24 PM EST

I have written some more code along the same lines, but can do arbitrary searches of k5 amongst other things: http://www.kuro5hin.org/story/2011/8/19/134242/189

-----
stats for a better tomorrow
bitcoin: 1GsfkeggHSqbcVGS3GSJnwaCu6FYwF73fR
"They must know I'm here. The half and half jug is missing" - MDC.
"I've grown weary of googling the solutions to my many problems" - MDC.
mumble's k5 comment watcher | 12 comments (8 topical, 4 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!