A week ago today, I gave a talk at PyCon Australia in Sydney. I discussed refactoring in Python, and how to build refactoring tools using nothing but the standard library, building up from concepts to syntax trees to how the lib2to3 module works. The talk finished with the announcement of the open source refactoring framework I built at Facebook, Bowler, which I built using those same concepts. I really appreciated the questions from the audience and in the halls afterwards, and we really enjoyed our time in Sydney. Thanks to everyone that made PyCon so great!
This past Friday, I presented a talk at PyCon US 2018 – the first time I’ve been fortunate enough to attend. The talk was focused on achieving high performance from modern Python services through the use of AsyncIO and the multiprocessing modules. The turnout was better than I could have ever expected, and I was really happy to hear from everyone that stopped by the Facebook booth to ask questions, discuss Facebook engineering practices, or even just say “hello”. Thank you to everyone who made my first PyCon amazing!
A short quote, and timely announcement:
I’m currently preparing a talk for PyCon 2018 on using asyncio with multiprocessing for highly-parallel monitoring and/or scraping workloads. To go with this talk, I’m working on some simple example code that I hope to publish on Github. This will be my first major conference talk, so I’m both excited and absolutely terrified! 😅
I’m looking forward to giving that talk, and will post a video here afterwards!
One of my favorite projects that I’ve created has to be ZNC Push, a plugin for ZNC that generates push notifications for IRC highlights and private messages. It supports a wide array of push services for all of the major mobile platforms, and gives the user deep flexibility in choosing how and when to trigger these notifications.
However, I have very little insight into usage statistics for the plugin. Pushover is the only service that gives me gross usage data, but considering ZNC Push supports six other networks as well as custom push URLs, this only represents an unknown portion of the total usage. But I will continue to draw baseless conclusions from it regardless! Do note that I can’t see anything more than raw volume of notifications; I do not have access to who is sending/receiving these, nor the contents of those messages.
So without further ado, some pretty graphs straight from the Pushover dashboard:
ZNC Push reached a milestone of sending 15,000 push notifications in May through the Pushover service!
Push volume has been growing steadily since Pushover support was added, with a small dip in December and January that is likely attributeable to the holidays for most users. May marked the second occasion that ZNC Push has exceeded the volume “cap”, and Pushover has again graciously increased that in support of this open source project. At this sort of grawth rate, I would expect the next volume cap (25,000/mo) will be reached by October at the latest. I’m excited to see such growth for a small side project with such a niche use case.
Sometimes, it’s amazing to stop and realize just how far this project has come. What started in January of 2011 as a basic module for the Notifo push service, has blossomed into a mature system supporting multiple push services and far more customization options than I had ever envisioned from the beginning. Knowing that it’s being used by so many people, and helping them stay more connected with their online communities, makes me a very happy engineer.
Many thanks to Pushover for granting such a large volume of free notifications to ZNC Push. If you’re not already using them for your push service of choice, I highly recommend them. They have apps for both Android and iPhone, and they continue to improve them and add new features over time.
And as always, my eternal appreciation to everyone who has contributed time and energy to helping me improve the module over the past two and a half years! The project wouldn’t be where it is today without your support.
Eighteen months ago, I wrote an article about my Favorite Android Apps, but a lot has changed with Android and its app ecosystem since then. Developers have started putting more focus on design and usability when creating apps, and of course newer apps have come along that I have come to rely upon. I myself have upgraded to newer devices, with a Nexus 7 replacing my Galaxy Tab 10.1, and a Galaxy Nexus from work has sadly replaced my beloved but aging Nexus S. So what follows are some of my favorite new apps from 2012, and some follow-up on what my previous favorite apps look like in modern attire. All screenshots are taken from devices running Android 4.2.1 (CM 10.1 nightlies).
For the last couple years, this site has been managed and updated using the open source static site generator called Poole. It is an excellent and simple system, comprised of two Python files — the Poole source code and a site-specific macros.py — a simple page template, input documents and static content. This is more than enough for a small static website, but more complicated sites, like blogs, require quite a few macros in order to generate things like archives, tag pages, or even RSS feeds. Over the 18 months that I worked with Poole, my macro file had gotten a bit disorganization, and the limitation of working from only a single template file was starting to strain on what I could do with it.
Now keep in mind, I still like many of the features that drew me to Poole in the
first place, such as the use of a document-centric build process, with YAML
headers for defining page metadata rather than an inline, proprietary format
used by projects like Jekyll. I also enjoy the simplicity of the content
representations, but the lack of an extensible content pipeline is my biggest
complaint. Supporting formats other than Markdown requires adding yet another
hook to the macro file; generating archives and tag pages required hacking in a
multi-phase build using
os.exec(); and inlining another page’s content
post-render was just not possible, resulting in Markdown named-link collisions
when rendering multiple posts to a single page.
I wanted something new. So, for the past few months, I’ve been working on Nib.
The resulting design is based heavily on the concepts of Poole, but built around a primary goal of producing a proper content pipeline that is simultaneously aware of the differentiation between resources and documents, and defines multiple stages where plugins can hook into the process and alter or generate page contents at build time. Indeed, most of the actual functionality of Nib is contained within a handful of plugins, while the main module merely defines a framework for the content pipeline.
In effect, adding support for more content or resource formats should be as simple as adding a new plugin attached to the appropriate file extensions. Advanced content manipulation, generating “virtual pages”, and aggregating pages or documents into multiple locations are all possible as well.
The Markdown plugin is 14 lines of code; the LessCSS plugin is 13; even the blog plugin — which generates the archive pages, tag listings, and Atom feed — takes only 86 lines to do a better job than the old Poole macros that required double the effort.
Today, I’ve thrown the switch. While it looks nearly identical to the old site generated by Poole, the page you are now reading has officially been generated by Nib. The news feed sees the biggest makeover, having changed formats from RSS to Atom, and now offers full content posts. Link-style posts, a format that I’ve been wanting to experiment with, are also a new option. Both features would have required more effort on Poole than I was interested in expending, but the new architecture has allowed me to indulge myself.
Nib is still in an extremely early, and unstable, phase though. It works for the needs of my site, and does come with some basic documentation and a sample site to start from, but it’s far from complete. Near term goals include adding support for an intelligent menu, as well as support for more content and resource formats, like reStructuredText or SASS. Contributions are always welcome though, even at this early stage. Nib is liberally licensed, and I would love to hear feedback from anyone trying to use it. Hopefully it will be useful for someone other than myself.
The past two months have been intense. At the beginning of May, I was let go from BioWare and Electronic Arts in a round of budget cuts for my studio. Since then, I’ve played far too many video games (and beaten multiple titles), slogged through innumerable interviews, redesigned my website from the ground up, and celebrated my seventh wedding anniversary in some of the best summer weather I’ve ever experienced.
Oh, and I got a job at Facebook.
On May 3rd, BioWare informed me that my position in the studio was being eliminated. My terrible luck had struck again. But this time was different; within a few days, a recruiter from Facebook found this website and my open source work, reached out to me, and encouraged me to apply for an engineering role. Four weeks later, after phone screens and an on-site interview, I was offered a position as a Production Engineer. My luck had not only reversed, it had gone off the scale in the opposite direction.
Alongside Facebook, all my other leads paled in comparison. I wanted to stay in gaming, but here was the opportunity of a lifetime, to join one of the biggest and best engineering companies in the world, and the chance to learn from some of the brightest minds in the industry. The position is similar to the role I had at BioWare: writing and maintaining software to manage and automate the vast array of infrastructure that supports the front-end applications and engineers. It was an offer I couldn’t refuse.
Monday was my orientation, and I was immediately impressed by the level of passion, vision and dedication from the entire team. Boz discussed the culture of “hack”, and told a story about building a loft in a war room when it couldn’t fit everyone in just two dimensions. Chris Cox gave an inspiring talk about where Facebook is heading, and I got a surreal sense of wonder when he mentioned our long term goals or the level of impact we have on global society and the way people communicate. It feels amazing to be part of a company that truly wants to change the world and improve the lives of hundreds of millions of people.
Even the training process embodies those same ideals. The first few weeks, every engineer goes through “bootcamp”, attending learning and development sessions, sitting among other bootcamp engineers, and working with veteran teammates on real tasks to get familiar with the infrastructure, tools and codebase. Everyone is given the chance to progress at their own pace, with as much or as little help as needed. I’ve already learned so much in a few days, and I’ve also had the chance to share my own knowledge and experience with my fellow newcomers. And yet there’s still so much left to cover.
Wednesday was my first day at a desk. Taped to the monitor was a sheet of paper printed with bold, red lettering.
What would you do if you weren’t afraid?
I don’t have an answer for that yet, but I will soon.
I’ve been spending time migrating my server from Ubuntu 10.04 to Arch Linux, and in the process I thought very deeply about every PHP application installed on the old server. Five out of the six PHP apps were only there to support the one that really stood on its own: MantisBT, my long-standing bug tracker of choice.
Now, I’ve been a core developer, and de facto release manager, for the project for many years — since I was still in university and getting paid by my then-employer to contribute features and plugins that they wanted to use for their engineering team. Those plugins, like Source Integration, wouldn’t be free without me fighting to license and release them for the community. And I wouldn’t be where I am today without the experience and help I received in turn from the very same community.
I know I haven’t been as involved in the project these days as I would like to be; there is an endless list of features and improvements to make to both the core system and the array of plugins I’ve created for it. Some great community members have stepped up and filled my place at times, while I have at least tried to stay active on the mailing lists and in the IRC channels. I’ve still guided and cut the last couple releases, but I haven’t played a part in shaping the future of the project.
There are multiple competing visions charting new paths for the aging project, with conflicting goals and revision histories, and it really needs a stronger leader to take the reins and guide the project to its next milestones. I’m unfortunately not the person to fill this role, for many reasons. Maybe a few years ago it would have been better timing.
Lately, I’ve come to the realization that I can no longer bring myself to work with PHP for personal projects. I don’t like the syntax, I don’t like where the language is heading, I don’t like how much memory and CPU it requires to run on a web server, and I just spend the whole time wishing I was writing Python code instead.
This blog hasn’t run on PHP or a database for just over a year now, and with my MantisBT install being used mainly for projects I don’t have the time or will to work on, it just seems to be dead weight. Github can serve my needs well enough for the few remaining projects I work on, and without needing a complicated setup on my end. Turning off MantisBT means I no longer need MySQL, PHPMyAdmin, APC, or even mod_php at all. The remaining apps can easily be replaced with external services.
So basically, this is me announcing what I’ve already been practicing for many months now: I will no longer be a developer for Mantis Bug Tracker, but I will remain involved as a mentor for other core developers, or for those seeking some advice on my plugins or creating their own. I won’t be maintaining any of my plugins, but I will look at and accept pull requests until someone else wants to step up as maintainer. I will be removing MantisBT from my site, but will keep a database dump in case I ever need to reference it in the future.
This is not me withdrawing from open source; I have many other projects that I’ve been working on, most of which are written in Python or C++. I find them more enjoyable to deal with, and most importantly, they allow me to break out of the realm of writing web applications. IRC, as old as it is, has been my point of intrigue lately, and is at the core of my current “pet” project.
Regardless of language, you can still find me on Github, where all my toys are available for the public to point and laugh at. And as always, I will answer questions on Freenode as “jreese”, or via email, although there may sometimes be a long delay before I can reply.
Thank you to everyone who’s contributed to MantisBT or its plugins, and thank you to everyone who helped me on the way to where I’m going.
Just wanted to say thanks for the ZNC Push Module you made. I’m using it with Notify My Android and it works great! This should be a standard module in ZNC imo. Stellar work my friend.
It makes me smile every time I hear from someone using my software, knowing that I can make an impact on peoples daily lives, no matter how small it may be. Thank you to everyone who has taken the time to thank me for my work.
Last week, I received an offer from EA BioWare: they want me to join their San Francisco operations team as a Platform Systems Engineer! I still can’t believe that this is happening, but I’m going to be moving to California to work for a game studio. This is practically my dreams come true, and I’m excited and overwhelmed beyond my imagination.
I won’t be working on games directly, but I’ll be working in a devops role to automate and manage servers for games and internal projects. It’s a good fit with my experience in web applications, system administration, and tool development. It also can be a potential stepping stone to a game development position later on.
The next two months are likely to be hectic, as I move to temporary housing in California, and then find an apartment and move into it. Ongoing projects may have to take a back seat until I can get the free time again. But the weather and job will be worth it!
This past Friday, I bought the new Samsung Galaxy Tab 10.1, and I absolutely love it. The new Honeycomb interface looks amazing, and is quite an improvement over what’s available on my Nexus S with Gingerbread. I love that the back, home, and task switching buttons are now rendered on the screen instead of having hardware buttons, and combining those buttons into the notification and status bar means you don’t even lose screen space compared to previous versions. The new application switcher looks really nice, but does seem to have some odd behaviors, like not showing the browser that I just switched away from, or seemingly choosing at random whether to display the list from the top instead of from the bottom where you would expect.
There are a lot of applications that don’t yet take advantage of what’s offered by the Honeycomb APIs, but still tend to work really well. It mainly depends on how well the author designed the application to scale with the user’s screen size and density. An example of doing it wrong is the official Facebook app; it’s still usable mind you, but it certainly looks dumb in process, showing the main menu as a large grid of tiny icons with massive amounts of whitespace between them. It would have gone a long way if they had simply scaled the images to fill the screen.
I’ve been meaning to write about some of my most cherished apps, and seeing them in new form has given me an even better reason to get to it. Some of them really only work well on a phone, and others have only gotten better than ever when given a tablet form factor to call their own. So in no particular order, here are my favorite apps for Android. All prices are rounded up from Market estimates at time of writing, and all screenshots are taken from my devices:
On Saturday, I appeared as a guest host on Podcast 17, for the interview portion of episode #128, to talk about the benefits and challenges of creating open source modifications for Valve’s Source engine, the power behind their classic Half Life 2 series. It got rather sidetracked from the original plan of discussion, and ran longer than expected, but there was some interesting discussion, and great insight from former Valve contractor Tony Sergi.
If you work on the Source engine, or are looking to run or join open source mod projects, I highly recommend you take the time to listen in.
For multiple reasons, I’ve decided to move away from using my LeetCode.net domain for my online identity and begin using NoSwap.com instead. I will be slowing moving over my sites and applications, and will make sure to set up 301 permanent redirects or other forwarding methods as necessary to make sure nothing gets orphaned for the forseeable future.
As for the actual reasoning behind the move, it stems from multiple issues I had with the old domain, such as:
Being able to pronounce my email address and domain to the general public without having to always spell it out and make sure they catch that there are two e’s in the domain.
Switching at the same time from jreese@ to john@ for my address also means I don’t have to worry about people misspelling my last name, which happens so much even my own family still gets it wrong on a regular basis.
Not having to explain what “leet” is, or getting remarks about it being a juvenile choice due to the stigma of “leetspeak”.
It’s a short .com domain, so that people will get it right every time; I’ve had people assume I actually meant leetcode.com, without confirming it before sending a doomed message…
Anyways, in the off chance that you’re a sentient being, and actually reading this, if you happen to catch an error or gap in my redirection efforts, please drop a line to john@ this domain.
I’ve just reached the 1000 commit milestone as recorded by my Ohloh profile. By far, most of them are related to my work on the MantisBT project and associated plugins.
Now if only I could find a job local to Cincinnati… If you are interested in hiring me, or know someone who’s looking for an open source developer, take a look at my resume.
- Six by four
- Eight by three
- Twelve by two
- Six past legal
- Three past drinking
- One before cheap insurance
- Space Shuttle Discovery
- SLR cameras