Posts filed in ‘Work’
Updating Memetracker
One of my favorite sites on the web is Techmeme. It’s a site that shows you the hottest trending stories across online media right now. It’s run by a guy named Gabe Rivera who developed the initial technology, has launched a few similar sites for other topics (baseball, media, politics), has sold some ads, mostly on Techmeme, and hired a few people to boost coverage on the sites. I have a lot of respect for what he’s built.
I’ve wanted to be able to create my own Techmeme-like sites for topics I’m interested in. I’d love to be able to see what’s happening right now across Chicago media, for example. In some ways Windy Citizen started as an attempt at “faking” a solution to that problem via crowdsourcing, since I lacked the ability to create something that could surface the top stories programmatically by itself.
A few years back though, someone built a Drupal module called Memetracker that lets you turn a Drupal install into a memetracker for any topic you choose. It was written by Kyle Matthews as a Google Summer of Code project waaaay back in 2008. You can see Kyle’s initial proposals and outlines here and here.
Kyle released a working version of Memetracker at the end of the summer and graciously helped me get an instance up and running. His code sat on top of Drupal’s FeedAPI module. You’d use that module to pull in feeds and then you’d use memetracker to figure out which feeds you wanted to include in your automated front page. Then it would run a bunch of magic on the feed items and tell you which were the top stories across those feeds.
It worked! And it still does. However, memetracker had some tricky python dependencies, relied on memory hog, cron-job killing FeedAPI, and lacked nuance. There was no way to assign weights to sources, for example. No way to control the sort order of related stories. No way to… you get the idea.
Over the last few years as I’ve worked on Windy Citizen, I’ve made a few attempts to get these problems fixed. I set up a few memetracker instances and have hired contractors to set up and work on memetracker instances.
With the instances I’ve set up myself, the problem has always been that memetracker would cause the server to overload and crash and I could never get cron to run consistently.
So I tried hiring people who know more about that stuff to fix some of the outstanding issues in the code. This has never worked out well. Every developer that I show memetracker to tells me immediately that we need to rewrite it from scratch to run on AppEngine or AWS or something. I understand that it’s easier for developers to work on stuff they built from the ground up, but that doesn’t help me much as the developer’s going to move on eventually and I’ll be left to support the work they did. I don’t have time to learn Python so I can understand the memetracker they built. I know Drupal, so I’d like to solve the outstanding bugs in the Memetracker module rather than have to learn Python from scratch.
The problem is that most of the developers I know who have experience with algorithmic aggregatrion would rather be tied up in a burlap bag full of weasels than touch or have anything to do with Drupal. The minute they start grumbling is the minute I know it’s over. Meanwhile, the Drupal people I know tend to be fellow themers and implementers and I never know if they can actually code or if they’re fellow fakers like me.
So I’ve now had 3 aborted attempts at paying someone to fix the bugs in memetracker and get it working. No successful outcomes. Over the last two years meanwhile, I’ve gotten more comfortable with things like using the command line (rebooting our WC box daily does that to you) and have a better understanding of the LAMP stack and how those pieces fit together.
So this past weekend, I signed up for a Linode account, set up a LAMP stack (took me 6 hours 
), got a Drupal 6 Pressflow up and running (took me 3 hours ), and then got a basic memetracker instance running on it. That last bit took about 12 hours of trial and error and I am indebted to encouragement and help from Chad Paulson. Eventually we found the problem, I was importing the wrong Python module. python-cluster != PyCluster. LOLWTF.
Now I’ve got that running and am going to work through the things that need to be done. In no certain order they are:
- Patch memetracker so it relies on the Feeds module rather than FeedAPI. FeedAPI has been deprecated and replaced by Feeds. We’ve been using feeds over on Windy Citizen and it’s awesome and never causes your box to crash. cron jobs actually run with Feeds. Meanwhile, FeedAPI never felt reliable in any way. This should help with the performance problems I’ve seen before in previous memetracker set ups.
- Patch memetracker so I can have more control over how stories display and are sorted within memes.
- Patch memetracker so it’s tracking and displaying inbound links and factoring those into the rankings (this is going to be ungodly hard).
- Patch memetracker so other external signals can be factored into memerankings, like the ### of FB shares that a story has received.
I’ve started an issue queue over on the Memetracker page on Drupal.org for the first issue. You can read my first update here. I’m hoping someone will come along and help me figure out the stuff I don’t get in that thread, but after two years, I realize I’m going to have to do this on my own so I have no expectations per se.
I will try to post progress updates as I go over here. We shall see…
The Windy Citizen is hiring an ad sales rep
Back in January, I redesigned my community news site, The Windy Citizen, to focus in on one thing and one thing only, a Digg-style reader-picked list of front page stories.
Since that time, our site has grown consistently week to week. New members have joined up. Our submission numbers have been inching up as well. It’s been a real treat to watch the community blossom.
But as the community has grown, our advertising and sponsorships programs have not, due to lack of time on my part to focus on them.
So just today I’ve posted a listing on Craigslist for a part-time advertising sales rep. You can read it here.
.
I also wrote a blog post over on The Windy Citizen explaining why I’m doing this in greater detail and offering up more examples of how well everything’s going. You can read that here.
In the first two hours since posting the listing, I’ve received two resumes. We’ll see how this goes.
The Task: If effort’s not a myth, then let’s give it a go…
Let's see how this goes, eh?
This is the first entry on my new homepage, BradFlora.com, where you will (eventually) be able to learn more about my work, what I’m working on now and what I’m learning as I’m going through it.
About Me
I am a multimedia journalist, web producer and open source web developer. My day job is serving as publisher and founder of WindyCitizen.com, a social news network for Chicago. I also consult on news-related web development projects and develop WordPress and Drupal-based web applications on a contract basis. I live in Chicago’s Old Town neighborhood and am a Midwesterner at heart (i.e. I get antsy when I can’t see the horizon). You can follow me on twitter at http://www.twitter.com/bflora or e-mail me at bradflora at gmail.com
For the last year, this site was an automated clip aggregator, pulling links out of Google News anytime my byline was in them. From time to time it would dredge up a story about one of my doppelgangers – there are at least 4 other out there, my father, a Maryland-based 10-year-old surfer, a photographer and a few years back there was a wrestler in Iowa who’s disappeared off the google radar. As might imagine, it wasn’t the most compelling presentation.
And I wasn’t creating anything there. Just storing links to stuff published elsehwere.
Toward a more efficient day
A recent Seth Godin link on Hacker News got me to rethink what this site could be. In “Is Effort a Myth” he tackles the widespread belief that “success” is based ENTIRELY on factors outside one’s control: luck, looks, connections, health – rather than on the one factor anyone has total control over: effort.
I’ve heard this argument before, often at church on Sunday mornings, and it always registers. I’ve often found myself chalking up other people’s successes and even my own to outside factors. The problem with this, of course, is twofold:
- You’re left feeling discouraged and (falsely) powerless.
- You’re never able to fully accept credit for a job well done.
Godin offers a challenge of sorts in his post, aimed at entrepreneurs, bootstrappers, executives, anyone facing a formidable challenge:
1. Delete 120 minutes a day of ‘spare time’ from your life. This can include TV, reading the newspaper, commuting, wasting time in social networks and meetings. Up to you.
2. Spend the 120 minutes doing this instead:
- Exercise for thirty minutes.
- Read relevant non-fiction (trade magazines, journals, business books, blogs, etc.)
- Send three thank you notes.
- Learn new digital techniques (spreadsheet macros, Firefox shortcuts, productivity tools, graphic design, html coding)
- Volunteer.
- Blog for five minutes about something you learned.
- Give a speech once a month about something you don’t currently know a lot about.
3. Spend at least one weekend day doing absolutely nothing but being with people you love.
4. Only spend money, for one year, on things you absolutely need to get by. Save the rest, relentlessly.
As someone who is in fact self-funding a startup, a diet like this reads like a useful roadmap. So let’s give it a go from now until Christmas.
First let’s see how these points apply to me:
Task 1: Delete 120 minutes of “spare time” from your life”
I think I can just about get there through two steps:
No more ESPN: I watch Sportscenter every day. I do not play football, baseball or basketball and yet I follow these sports from a distance, daily, because the program provides a reliable, never-fail disitraction from whatever code/design/writing/life problem I’m facing. Clearly, this time can be better used elsewhere.
More deliberate newsgrazing: I spend a great deal of time each day simply reading weird news on the web. As a journalist, this is an essential part of my day. However there are a few sites I read that add little to nothing to my knowledge of the world or Chicago that I can do without. For the next 3 months, I’ll steer clear of just a few of them.
Task 2: Fill that 2 hours with useful stuff
1. Daily exercise – I’ve never been big on exercise, but there’s a decent 2-mile circuit here in Old Town that I’ve jogged a few times. I can do this 5-days a week.
2. Read relevant non-fiction – This means being more consistent in reading more obscure local stuff.
3. Send thank-you notes – I get a lot of e-mail. Designating a few minutes each day simply to follow up with people and see how they’re doing would be great.
4. Learn new digital techniques – There are problems with WindyCitizen.com that have vexed me since day 1. Rather than continue to fixate on them, I can instead learn small technical tasks that could eventually lead me toward the solution.
5. Volunteer – I have friends that are very active in service at church and elsewhere. For all my talk about “being a good citizen” I could stand to do something like this every week.
6. Blog for 5 minutes about something you learned – And this is why I’ve rebuilt BradFlora.com, as a place where I can post short updates on something new I’ve learned each day. Some of these updates will be embrassingly simple, but by writing about them I’ll be able to better retain the knowledge.
7. Give a speech once a month about something new you’ve learned – Every now and then I’m asked to speak on work-related topics. I can be more deliberate in seeking these out.
Task 3: Spend one day a week with people you love
This is a secular way of saying “take a sabath.” For those unfamiliar with the concept, the Sabath is a day of rest when no work can be done and when one is meant to be focused in on God. My church attendence (I am a member at the Moody Church) has been pitiful lately, though I have started attending a 1-hour Bible study once a week. I will get my butt out of bed Sunday mornings and stay off the computer on Sundays for the next three months.
Task 4: Only spend money on things you absolutely need for a year. Save the rest, relentlessly.
For the last 6 months, I’ve definitely been following the first part of this edict. However I also realize that I could be more deliberate in cutting out impulse expenditures.
I recently signed up for Mint.com, a web service that tracks your personal finances for you. After a month of using it, it’s showing me some pretty disturbing things, chief among them the fact that I spent over $150 at McDonalds last month! Yikes.
So I will take a look at my spending patterns and also make a commitment to save 10% of every dollar I make, in addition to the 10% that I tithe at church. Before I went off to grad school, I was aggressively socking away 55% of my salary each month. I can’t afford to do that right now, but I can be saving SOMETHING.
Ok. This is a tall order. I’m not going to be able to keep up with most of this stuff, but it’s a start. We’ll give it a go this week and see what comes of it. Wish me luck.