FeedLounge

This post contains some technical information that will be very interesting to other developers, but may not be so interesting to those that don’t work with databases.

For those of you that just want the nutshell:

We’ve switched to new database server software because we think the new software will be faster and store the same data while using less disk space (making our storage costs lower).

Now for the detailed explanation:

Back in August, we switched the FeedLounge database from MySQL’s MyISAM tables to InnoDB tables. This solved a number of problems that we were having with the MyISAM tables, but also introduced a number of other issues including slower performance on a number of queries (counting in particular). As we did more and more research into how to solve these problems, we decided that one promising solution would be to switch to a PostgreSQL database instead of a MySQL database altogether.

That is what we were testing this weekend, and the alpha server is now running live with a PostgreSQL database instead of the MySQL database. This is still in “testing” mode - if we run into any serious issues we might switch back (temporarily or permanently) to the MySQL database (which will have data current through last Friday evening).

Some of the reasons we decided to make the switch to PostgreSQL:

• Database size - when we switched from MyISAM tables to InnoDB tables in MySQL, the size of our database grew from ~1GB to 10+GB! When we made the switch this weekend, the MySQL InnoDB database was using 34 GB, and the same data in a PostgreSQL database is only 9.6 GB - this should keep our hardware costs down a bit.
• Load time - The current MySQL setup takes over a day to restore the current database, the load of the data into the PostgreSQL database took just over 4 hours.

I was once given a simplistic summary of the MySQL and PostgreSQL databases:

MySQL was built with speed as the primary feature, PostgreSQL was built with more robust features like triggers, but lacked the speed of MySQL. As both are maturing, they are moving towards the other. MySQL 5 adds triggers and stored procedures, while PostgreSQL is focused on improving performance.

For an application like FeedLounge, the faster counting and smaller row size of PostgreSQL are compelling reasons to use it instead of MySQL with InnoDB tables. Add that it runs in 1/3 the time, using 1/3 the memory (making it essentially 10x faster for us) and that we can start taking advantage of other Postgres features, like data partitioning and partial indices in the near future, and you have the reason we decided to make the switch.

This week should be an interesting one, we’re looking forward to getting feedback from our alpha users on how FeedLounge is performing on the new database.

Finally! We’re live with our alpha 5 release.

Some new features of note:

• A “River of News” view
• Atom 1.0 support
• Auto-discovery of feeds (can enter http://feedlounge.com/blog/ instead of http://feedlounge.com/blog/feed/)
• Revamped tagging interface for easier addition and removal of tags
• Revamped item header display (includes author information if available)
• Enclosures are shown as download links in the item content area
• Links to subscribe to comment feeds for items that offer them
• Creation of the feeds list on the server - speeds up initial load
• A new read/unread icon
• Ability to edit feed tags
• Ability to delete tags in the feeds list
• Ability to set the color of a tag
• Back end optimizations
• The page title now prepends “Loading…” when busy, so you can be in another tab and see when it is done loading

Also, I really need to update the screenshots here at some point.

Back when we released alpha 3, I had this to say about our development process:

…our plan moving forward is to push out more feature focused releases on a faster timeframe. Having more bite size releases rather than cooking up a whole meal for each release should help us get specific testing and feedback on these features from our alpha users faster, and should make the releases easier for us to manage.

As you might expect, alpha 5 is (instead of an incremental feature release) a massive rewrite of major parts of the system. Sure, we’re adding a bunch of new features, but we’ve also made some fundamental changes.

Some of these changes are part of our initial plans to helps scalability, some are streamlining the code for easier maintenence, and some are due to things we’ve learned about browser JavaScript performance.

We are planning to push Alpha 5 to the alpha servers soon - it’s almost ready! We’re quite excited about the new features we are rolling out over the next few releases, and quite optimistic about our scalability plans. Right now, it looks like we’ll have another handful of alpha releases where we make a few more major changes/additions, then we’ll be working towards our beta release.

One of the changes I’d like to include in the next alpha release of FeedLounge is a better read/unread icon. Right now we use a page with writing on it or blank.

This is pretty weak in several ways, mainly:

• It looks like a checkbox in Windows XP (depending on your theme)
• Its meaning isn’t immediately recognizable

My first version of the icon was an eye - open or closed. The meaning may have been a little better, but Scott thought it looked too freaky. Here is one of the early drafts of it - he’s probably right.

E-mail programs generally use open and closed envelopes to indicate read/unread status, but there isn’t an equivilent “wrapper” for a feed item. Perhaps we should use a rolled/unrolled newspaper or something like that?

Maybe you folks have some ideas for a great read/unread icon we could use? Feel free to post your ideas and links to your icon designs (16×16 pixels please) in the comments.

A few people e-mailed us or blogged¹ their extreme displeasure at our “Firefox-only-religious-zealot-Microsoft-hating-open-source-crazies” browser policy for the alpha.

We tried to clear this up once already², but we seem to have done a poor job of it.

So here it is one more time - I hope this is more clear:

1. When developing a web application, supporting only a single browser makes it much easier to test and support your application.
2. From experience, we know that it is easier to make a web application that already works in a Gecko browser (like Firefox) work in IE or Safari than vice versa.
3. Firefox has some great development tools (the Web Developer extension, JavaScript console, Venkman, etc.) which save us time and make development easier.

Once we have our core functionality working in Firefox, we will take the time to bridge whatever delta may exist between our Firefox tested code and what is needed to work on additional browsers.

Thanks for understanding.

1. Not picking on anyone, just a recent example. [back]
2. Twice actually. [back]

Since FeedLounge is using so many modern web application techniques to deliver the user experience, we occasionally run across issues that look suspiciouly like browser bugs. We want to make sure these get reported and fixed (either in our code or in the browser’s code), so we’d like to invite a developer or two from all browser teams to join our alpha test.

We feel that this will be a mutually beneficial arrangement:

1. We push the limits of the browser and potentially find bugs that don’t normally get surfaced, report them to you (the browser developer) and provide test and use cases.
2. Bugs that affect FeedLounge get fixed (or we get work-arounds for them) which is a great thing for our users.

If you are a browser developer and are a feed reading fan or are just curious to see what we’ve got going on here, please get in touch - we’d love to work with you.

Right now we use AJAX to store each user’s preferences (current feed/tag/item, which tags are open in the feeds list, etc.) on the server, maintaining a single consistent experience across any browsers and computers someone is using. There are a lot of benefits to this, it’s often great to be able to pick up right where you left off.

However, we’ve also considered taking a different route: storing preferences (for things liks open/closed feed tags, selected item, etc.) in a browser cookie. This would allow you to have a “work” profile and a “home” profile of sorts.

How do you read? Which would you prefer? We want your feedback…

One of our #1 goals for the alpha release was to guage how much hardware we needed to have in place for our beta release. Over the last few weeks, we feel like we’ve gotten the information we were looking for; and it’s a fairly sizeable bit of infrastructure.

We’ve got a lot of things in motion right now:

• We’re in the process of setting up an interim dedicated server for FeedLounge so that we can bring on a few more alpha users and get some final performance metrics without the other box activity cluttering up the numbers.
• We’re finishing our business incorporation, setting up bank accounts, etc. etc.
• We’re getting ready to purchase a number of servers which will all need to be configured.
• We’re deciding on a datacenter to put our hardware into and getting that set up.
• Oh yeah, and we’ve got a bunch more software development on the list too.

While we’re committed to making a free version of FeedLounge available, we’re also committed to making sure the service can sustain itself. We have a few creative ideas in mind regarding revenue that we’ll be talking about more as we get closer to a beta launch. Luckily, there are people out there who are willing to pay for online services (well, at least one).

Getting a service like this ready to go is expensive, both in time and money. Unfortunately, there aren’t shortcuts for either.

While we know that many of you are eagerly awaiting the beta release; we also know that a premature release won’t be good for anyone. Your patience is greatly appreciated.

We’ve been hard at work since the last release: refactoring and optimizing code, fixing bugs and generally making our foundation more stable.

The Alpha 3 release includes over 100 tracked fixes and enhancements, plus a few dozen more that were made along the way. Some of the notable changes include:

• A complete rewrite of the feed refreshing process. It is now multi-threaded and using our smart update schedule. The update process now comes calling with a FeedLounge user agent which includes not only how many subscribers this feed has in FeedLounge, but also when we’ll be back to check again.
• A new toolbar for the feeds list, completing the contextual toolbar implementation.
• Revamped pop-up windows that keep the buttons on the screen at all times.
• Support for Page Up and Page Down keys in the feeds and items lists, and the ability to open and close tags in the feeds list using the ‘=’ key.
• A new preference setting lets you choose the default state of the ’save items until I delete them’ options when subscribing to new feeds and importing OPML.
• OPML export (the link is in Settings - My Account).
• Lots and lots and lots of little changes and fixes - both front and back-end.

Now that we feel we’ve shored up areas of the platform that needed it and spackled a few nail holes, our plan moving forward is to push out more feature focused releases on a faster timeframe. Having more bite size releases rather than cooking up a whole meal for each release should help us get specific testing and feedback on these features from our alpha users faster, and should make the releases easier for us to manage.

Last but certainly not least, a well-deserved pat on the back to Brian who did a great job knocking tons of bugs off the list to get this release ready to go.

Now that there’s been a little wider testing, it seems I was a little overly optimistic when I mentioned we might already have Safari support in FeedLounge. Everything works fine with the layout and the mouse, but the keyboard navigation doesn’t work the same. It’ll be fixed before beta.

If there is anyone out there on the Safari team that would like to join the alpha test and help us track down tough Safari issues, please get in touch.