ActivityPub and MediaGoblin at TPAC 2016 (or: ActivityPub needs your review!)

    Screenshot of ActivityPub Working Draft

    Hello everyone! We have a lot of news to cover, but I’m going to jump right into the thick of it: we’ve been working hard on a new federation (as well as client to server) standard called ActivityPub (formerly ActivityPump). We’ve made tremendous progress, and I was just recently at a face to face meeting at TPAC, the W3C’s big technical conference.

    The good news: ActivityPub is aiming to hit Candidate Recommendation status by October 11th. (That’s less than a week away!) However, in order to enter that stage, we need your review! If you have any interest in the decentralized web, you can help. All you have to do is read the latest editor’s draft and provide feedback. (The earlier the better… maybe a fun weekend project?) You can do this by any of the following:

    • Post issues on the issue tracker (yes, I think it’s ironic and problematic that we’re using GitHub, I picked my battles here, whether right or wrong).
    • Email the social working group comments mailing list.
    • You can email me directly. Email cwebber AT dustycloud DOT org, and include “ActivityPub” in the subject. Note that I will be publishing your comments publicly, probably on the Social Working Group wiki!

    If you have feedback, we want to hear it! (From anyone, but especially from those who are interested in implementing ActivityPub!) Thank you!

    ActivityPub live

    Note, the rest of this post is a bit of a dive into behind the scenes decisions and activity in MediaGoblin and ActivityPub… there’s no need to read this part to submit a review!

    I’m kind of traveling through time in all the wrong order in this post… but it’s worth jumping forward a bit to see the results of what we’ve done. Over the last many months there’s been a huge push on ActivityPub as a standard, as we’ll talk about. But in order to succeed, I also needed implementations, both of client to server and server to server. Well, I’m happy to say that those did come together…

    Pubstrate in action!

    This is Pubstrate, an implementation of ActivityStreams and ActivityPub for GNU Guile. Sorry for the sappy embedded love-note, though what’s nice about this is that it’s the first demo I gave to someone else of the client to server functionality working in Pubstrate. But wait, what was the client?

    Soci-El in action!

    This is soci-el, an ActivityPub client written in… you guessed it… emacs lisp. You can see the rendering of the user’s outbox here as well as the buffers from which the message was composed.

    Pretty nerdy! I don’t expect everyone to be using emacs as an ActivityPub client of course… I hope to see various desktop, web, and mobile clients made available. But, Emacs is the most fun for me, and I was time pressed, so there we are.

    Everyone loves seeing screenshots, but maybe not all of this stuff makes sense without context. What does this mean for MediaGoblin?

    So what’s been going on?

    It seems a recurring meme in MediaGoblin land to say “we’ve been quiet, because we’ve been busy” (or maybe historically on every tech blog ever), but I guess I can’t resist repeating the mantra. It’s true! Though the weight of my focus has been shifted from where I expected it to be. From the last few updates over the last year, you would be right to anticipate that the main thing I would be working on would be merging the federation code Jessica has written and getting 1.0 out the door. That was the plan, and we’re still working towards that, but priorities shifted as we realized the opportunities and time pressures we were under with ActivityPub. After the Social Working Group face to face meeting in June, Jessica and I sat down and talked about what we should do. Jessica had recently started working at Igalia (great people!) and was busy with that and other transitions in her life, so we discussed whether we thought it was most sensible to focus my energy on MediaGoblin or on ActivityPub. It was clear that ActivityPub was shaping into a solid specification, but it was also made clear that the Social Working Group’s charter was running out by the end of 2016. We both think ActivityPub is key to MediaGoblin’s success and didn’t want to see our invested time go to waste, so decided my immediate focus should switch to ActivityPub so it could successfully make it as a standard.

    Which isn’t doom and gloom for MediaGoblin! MediaGoblin development has continued… the community is good enough that people have been able to work while I’ve been busy. I’m happy to say we also appointed longtime contributor Boris Bobrov as co-maintainer to help reduce me being a bottleneck. (Thank you Boris, and congrats!) Other contributors have also stepped up to the plate. I’m especially thankful of Ben Sturmfels for hosting MediaGoblin hackathons and being so responsive to community members. (And of course, there are many others to thank, too!)

    Anyway, I’m not going anywhere, I’ve just been shifting focus to standards work for a bit… but it’s all for the greater good of MediaGoblin. (Well, and the greater federated social web!) Soon, we’ll be putting the work we’re doing on ActivityPub directly into MediaGoblin. When we merge Jessica’s work on federation, we will also retool it so that the first federated release of MediaGoblin will be blazing the trails with ActivityPub.

    Both ActivityPub and I personally got a significant boost by a happy visit from friend and Social Working Group co-conspirator Amy Guy. Amy dropped by for an intense four days of standards hacking and scheming, and came up with several significant ways to restructure and improve the document. With her help, we now have much clearer distinction between the use of ActivityPub as a client to server protocol (think mobile applications and desktop applications connecting to your server) vs the server to server protocol (federation). Both of these are now clearly intertwined in the document, but are distinct enough where they can be used and understood separately if desired. We also more clearly established the connections between ActivityPub and the linked data community by explaining ActivityPub’s relationship with Linked Data Notifications.

    Amy has a gift for composing standards language, something I’m still struggling to learn (but hopefully getting better with over time). ActivityPub is much better with her hard work. Thank you Amy!

    In addition to the standards side of things, in order to get ActivityPub to the next level, we needed to be able to present real live demonstrations of the standard in action. Hence the work on Pubstrate and soci-el mentioned previously. For most of the months before TPAC, I was working furiously day and night to get things ready to show… And then, it was time to head off, and hope it was good enough…


    Live demo image, by aaronpk
    TPAC demo room photo by Aaron Parecki

    Fortunately, all that hard work paid off. The Social Working Group kicked off TPAC with live open-viewing demonstrations of the various standards we’ve been working on. We got in a really solid set of demos from everyone in the group.

    From my end, I managed to demo all the core parts of the ActivityPub spec: I wrote a note in my client (soci-el), fired it off to the server (Pubstrate), where it rendered successfully. Then I explained, well, what if you want someone on another machine to see it? So I had another user on a separate Pubstrate instance, added the user over there to the recipients list on my message, shot the message over to the server, crossed my fingers and hey! The live demo of federation succeeded. (Whew!)

    In the actual main Social Working Group meeting, we hammered things forward pretty nicely. As said, ActivityPub was positioned to move ahead towards Candidate Recommendation by the 11th. Again, your feedback is most valuable at this time!

    Spying image from the campaign video

    Oh, one more thing. On the second day of the main Social Working Group meetings, at Amy’s suggestion (apparently she was impressed when I showed her at her visit), I showed off the MediaGoblin campaign video to the group. Apparently I had never done so previously, so it was really nice to hear the reaction: “Holy cow, this is describing exactly the type of stuff we’re working on in the working group!” Yep, exactly… all that stuff you see illustrated in that video, we’re working on putting into code and standards. The dream lives!

    So, all this Social Working Group stuff… things are happening! We’re not just goofing off!

    TPAC 2016-09-23 goofoff gif by aaronpk
    Super serious TPAC group “photo” by Aaron Parecki

    … er, right. Not just goofing off! :)

    MediaGoblin 0.9.0: The Three Goblineers

    MediaGoblin 0.9.0: The Three Goblineers
    The Three Goblineers” by Christopher Allan Webber (pen and ink) and Morgan Lemmer-Webber (colored pencils). Licensed under CC BY-SA 4.0.

    This release is called The Three Goblineers, because we are finally fully embracing Python 3! You could even think of this release as Py-oneering, which it definitely is. Many traditional web service tools are less-than-ideal for federation and so we’ve had to do a lot of rebuilding and retooling. This release represents lots of intense behind the scenes work to make the user experience smoother, as well as some key improvements for MediaGoblin developers and deployers.

    Federated services use databases in a some fundamentally different ways. We had to make a traditional (rigid) database more friendly for more flexible relationships. The result is similar to the generic foreign key implementation used by Django, but optimized especially for federation. Jessica Tallon did the lion’s share of this work and was aided by Andrew Browning who did extensive testing.

    We also updated how we handle comments and media collections. On a multi-media service, people will naturally want to reply to comments with videos and to ASCII art with songs so we had to make our commenting function support all the available media types. Also, the media collections aren’t just for your personal gallery anymore; they’re also used now in federation and the API as the backbone of a user’s “inbox” and “outbox” feeds. Also, to make life easier on those uploading whole albums of content, if your user has some collections available, these will be presented as a dropdown option while submitting media.

    Selecting a collection during submit

    In the less visible but equally important department, we updated OAuth and created a “graveyard” system. We updated our code to make better and more secure use of OAuth so that my server and your server can be really sure we are actually talking to each other. The “graveyard” system uses tombstone icons to let you know that an object was removed by it’s original poster. This makes it easy for users to remove media they no longer want to share, while creating as little confusion as possible for other users.

                  *                      *
     *                      *                  _.     *
          *           *          *            <  '.
                 *                             )  )       *
                                        *     <_.'   *
        *      *        .-------------.
                      .'               '.                *
           *          |                 |   *
                      |   TOMB OF THE   |       *
                *     |     UNKNOWN     |            *
       *              | ACTIVITYSTREAMS |
                      |     OBJECT      |
                 .^.  |                 |
      _  .+.  _  |~|  |    ????-????    |  .+. .-.  _  .-.
     | | |~| |=| | |  |                 |  |=| |~| | | |"|

    Developers will be happy to know that we are keeping pace with the larger Python community and now have complete support for Python 3. We had early and experimental Python 3 support in 0.8.0 but couldn’t quite recommend it for production use. That work is finished! Now you can choose to hack in Python 3 or Python 2 and your code will be totally compatible with the main codebase. Since Python 2 won’t be supported forever, Python 3 is the future. And we are all about the future.

    We also switched our migration system over to Alembic. Previously, we’d been using sqlalchemy-migrate, but it also wasn’t as good for updating the database layout when you upgrade, and lots of dragons be there. We think Alembic will make it less terrifying to update your MediaGoblin instance so you can stay current and without spending lots of time trying to sort what happened in the database. Alembic is also newer, fresher and better maintained!

    Another thing that makes updating a little persnickety is finding out that the dependencies have changed. We are collaborating and experimenting with the Guix community to fix this issue. Cleaner packaging will help people upgrade without fear, using deployments they can trust. Watch this space for a future post on how to use Guix and MediaGoblin together for worry-free upgrades.

    For users upgrading from previous versions, as always, check the release notes for instructions on upgrading!

    Our work to overthrow the red eminence of the centralized web continues! Between Python 3 support and laying down foundational changes to support federation, this release brings us much, much closer to our long term goals! Thanks to all our Goblineers and Py-oneers without whom this work would not be possible! Give a round of applause to Andrew Browning, Ben Sturmfels, Berker Peksag, Boris Bobrov, Christopher Allan Webber, Daniel Krol, Deb Nicholson, Duncan, jerome, Jessica Tallon, Loic Dachary, Sebastian Spaeth, Tom Fay, and 宋文武! You all helped make it happen!

    And if you want to make it happen in our next exciting release, we’d love to have you get involved! Visit us in IRC; #mediagoblin on! Or sign up for regular updates on our mailing list. Got ideas or questions about our work? Email us at — we look forward to hearing from you!

    Apply for GSoC in MediaGoblin (and Guix/Shepherd!)

    Hello all!

    Summertime is fast approaching, and this means GSoC is fast approaching too. This year we have some interesting potential projects. Check it out, and if you’re interested, apply! You have until Friday (March 25th) to get your application in.

    We just added a new item, and I wanted to call it out specifically: porting federation to ActivityPub. This is not the only potentially interesting GSoC project, but it is a special one; you’d be able to work with Jessica Tallon (the source of federation support in MediaGoblin!) on updating federation to our current standards work.

    One more call-out: this one isn’t MediaGoblin specific, but I (Chris Webber) am willing to mentor it: Adding an extensible event loop to Shepherd (the init system used by Guix). This would quite probably use the under-announced GNU 8sync project. If you’re interested in learning about event loops and how to write them, you might want to watch and even better, read the corresponding sections of SICP which inspire it. This will be a tough one! But if you’re really interested in digging into some cool ideas about concurrency, I’d recommend it. There are some other great projects for Guix as well!

    Both MediaGoblin and Guix welcome all applicants, though both projects also strongly encourage women, non-binary gendered individuals, people of color, and other underrepresented groups to apply. Both projects follow a Code of Conduct (for MediaGoblin and for Guix).

    Time is running short; apply! Detailed proposals are encouraged, and jumping in and experimenting / will greatly enhance your possibility of acceptance in both projects. Join #mediagoblin and #guix respectively on to talk to other developers.

    Happy hacking!

    MediaGoblin 0.8.1: Security release

    Basic Summary

    We have had a security problem in our OAuth implementation reported to us privately and have taken steps to address it. The security problem affects all versions of GNU MediaGoblin since 0.5.0. I have created a patch for this and released a minor version 0.8.1 (see the release notes page). It’s strongly advised that everyone upgrade as soon as they can.

    In order to exploit the security issue, an attacker must have had access to a logged in session to your GNU MediaGoblin account. If you have kept your username and password secret, logging in only over HTTPS and you’ve not left yourself logged in on publicly accessible computers, you should be safe. However it’s still advised all users take the following precautions, listed below.

    Users should check their authorized clients. Any client which looks unfamiliar to you, you should deauthorize. To check this:

    1. Log in to the GNU MediaGoblin instance
    2. Click the drop down arrow in the upper right
    3. Click "Change account settings"
    4. At the bottom click the "Deauthorize applications" link

    If you are unsure of any of these, click "Deauthorize".

    I would like to thank Dylan Jeffers (author of Goblinoid) for finding and reporting this to us in a responsible manner so that we were able patch this.

    Technical Information

    The security issue was caused by the verification of the OAuth verifier code. There the proper checks were not occurring to validate the verifier code matched the one issued to the request.

    This only affected those who clicked the verifier link whilst being logged in and entered a different code. The assignment of the user to the access token only occurs when you go to the authorization page whilst being logged in. If the link isn’t clicked with the user logged in no user will be assigned to the access token and a client attempts to use it will be denied as the endpoints won’t be able to look up the requesting user.

    A patch has been made should you wish to view the fix.

    State of the Goblin: Stripe Open Source Retreat, and more!

    Hello, all!

    It’s been a few months since my last major update so I wanted to fill in what’s going on. As usual, a lot has been happening, and it’s been hard to cover it all as we go. There’s some particularly huge news in this update, including something about funding something oh hey this should help us get MediaGoblin 1.0 out the door, plus something about the standards work we’re doing, something something. So let’s dive in and resolve all those somethings, right?

    Support organizations that support freedom!

    Okay, wait, a brief intermission! We’ll get to the cool MediaGoblin related news in a moment, but we’ve got something very important to cover first. Two organizations I really care about are running funding campaigns. Okay, well, it’s that time of year, and a lot of organizations are running funding campaigns, but these two especially could use your kind contributions!

    Support Conservancy!

    Copyheart by Christopher Allan Webber, CC BY 4.0 or GPLv3 or later, your option.. Source here.
    Become a Conservancy supporter!

    The first is Software Freedom Conservancy. They’re close friends of ours, and do great work. They’re also pushing hard to try to build up a supporter program, and they could really use your help. Conservancy does a lot of work around many things, from running Outreachy (hey, we were lucky to get Jessica Tallon working on our stuff through that!) to enforcing the GPL to hosting a whole lot of useful free software projects. Become a supporter today!

    Support the FSF!

    Next up is the Free Software Foundation. Hey you probably know these folks right? The FSF is the steward of GNU, and MediaGoblin is a GNU project, so success for the FSF is success for MediaGoblin. They’re running their annual fundraising campaign and they could really use your donation. The FSF has the long-standing history of being the anchor of the free software world, and they continue to do great work year after year. Help the FSF continue far into the future… get your donation in today!

    Support Guix!

    GuixSD logo by Luis Felipe López Acevedo. Permission is granted to copy, distribute and/or modify this work under CC BY 4.0.
    Support Guix!

    Did I say two organizations? What’s with three sections? In fact, our friends at the FSF are also teaming up with our friends in the GNU Guix project to run a special campaign to raise funds for new servers. I’m highlighting this for two reasons:

    1. I believe Guix is very important to the long run of “deployability” in free software network services (which you may know I believe to be an extremely important issue…)
    2. Guix is using a similar funding model of going through the FSF as fiscal sponsor. This is the same route we took for the MediaGoblin campaigns and I think is a great way for free software projects, and particularly GNU projects, to go. I’d love to see further examples of success for it in Guix.
    So, go donate! :)

    Okay, whew… sorry about that distraction, but these things are really important! But I know, I know, you came here for the MediaGoblin news. Well, let’s get to that!

    MediaGoblin selected for the Stripe Open Source Retreat

    So this is some big news! MediaGoblin has been selected to be a participant in Stripe’s Open Source Retreat 2016!

    What does this mean? It means that I’ll be moving to San Francisco from mid-January to mid-April 2016, working from Stripe’s office, and Stripe is going to pay me to focus totally on getting MediaGoblin 1.0 out the door and advancing our federation work. This is a huge opportunity for us; getting such ~unrestricted funding is, as anyone who has ever done fundraising knows, enormously difficult. We should be able to use this to bring MediaGoblin to the next level.

    When I filled out the application for this I was interested but skeptical of Stripe’s claim that this would be “no strings attached” funding. I’m happy to say this seems to be true: they’ve paid us to do the work, and Stripe is making no claim to copyright or asking us to change any of our existing policies. Contributing upstream to MediaGoblin happens as usual for the time I’m there, which is great.

    So what do we hope to get out of it? Well, my goal is by the end of the retreat, we’ll have MediaGoblin 1.0 out the door with the basics of server to server federation in place. I’ve also talked with the Stripe team about using that time to advance work on our federation standards work, and if there’s time, some deployment work too. But MediaGoblin 1.0 comes first!

    (In the meanwhile I just booked a bunk bed for $1000 USD per month, which it turns out is cheap for San Francisco housing. Egads! How do people afford to live there? Luckily it’s well covered by the retreat’s stipend! Are you in the area during that time? Maybe we should meet up!)

    I’m very excited about this opportunity. Thanks again to Stripe for supporting our community. I promise that the grant will go to good use, and we’ll have exciting things to report!

    W3C updates

    W3C Social WG, third GMG represented meeting
    W3C Social Working Group December 2015 face to face meeting attendees. I’m hiding in the back.
    Photo taken by Aaron Parecki, CC0 1.0, originally posted to the W3C wiki.

    Our work to standardize federation technology within the Social Working Group continues. Just a few weeks ago another face to face meeting was held at Mozilla’s San Francisco offices (thus giving me an excuse to test-run the bunk bed I’ll be sleeping in for several months for the Stripe retreat). The meeting was extremely productive… in fact I would say it was the most productive time the Social Working Group has ever had. You can read the minutes for Day 1 and Day 2 if you like that sort of thing. But here are some highlights:

    • Most of the work towards making ActivityStreams 2 a Candidate Recommendation document was put in place, and in a couple of weeks this I expect it will achieve that goal. Candidate Recommendation is a big step in a standards process, and AS 2.0 is the heart of ActivityPump, so this is huge for us.
    • ActivityPump, the standard we are pushing for server to server federation and client to server communication has moved to Editor’s Draft state! The objective of moving to First Public Working Draft by mid-January has been set, and we are pushing hard towards it.
    • Because of increased push on moving ActivityPump to this state, I have been added along with Jessica Tallon as co-editor on the specification.
    • The “IndieWeb stack” (for lack of a better grouping) standards have all also advanced to Editors Draft status, including Webmention, Micropub, and Post Type Discovery. These standards are also on track for First Public Working Draft in mid-January.
    • Amy Guy’s work on a “convergence” standard has been renamed to Social Web Protocols and also (are you noticing a trend?) advanced to Editors Draft and is on track to First Public Working Draft in mid-January.
    • I demoed ActiviPy, which went over really well. ActiviPy started out as a method of representing and working with ActivityStreams for Python, but I realized in developing it that since it was using JSON-JD with an implied context to handle ActivityStreams anyway, I could also extend it to support the Microformats vocabulary by using the JF2 context being developed for JF2. So I showed off a demo where I loaded ActivityStreams2 documents, demonstrated the method dispatch system ActiviPy uses (which is fairly interesting but I won’t bore readers with it), but most excitingly, I loaded ActivityStreams and Microformats documents side by side in the same system and then converted them both to linked data! This got a fairly strong reaction from the room, since this was all three of the “directions” of achieving federation we’ve been working towards, with a real live demo of convergence! I was very proud to show this off.

    Maybe most importantly was the “spirit of the room”, and how much this has changed from prior meetings. This group was formed to work on some very challenging domains with the goal of bringing initial participants with some historically very differing backgrounds. But both the last face to face (which we mentioned in the last state of the goblin post) and this one have really done tremendous things towards propelling this group forward, and unbelievably, towards something that might actually be convergence (without requiring that).

    Those who know enough historical detail of this space may be astounded to read that last sentence, but I believe it is true. In a sense, agreeing that convergence was not mandatory helped bring us towards a greater possibility of it. The agreement that ActivityStreams, Linked Data, and the “IndieWeb Stack” (for lack of a better term) were not required to work together, and that we could produce multiple deliverables, has eased that tension in the group and allowed us to work collaboratively. Everyone has worked hard to understand each other. But one person in particular has been doing a stand-up job of trying to bridge the cognitive gap and that person is Amy Guy. This can be seen immediately with her work on the Social Web Protocols document, but I believe she has done a great job otherwise in mapping the space.

    So anyway, optimism can only bring you so far. There’s a ton of work to be done in this space, and we’ll be pressing hard.

    See you at FOSDEM!

    Are you going to FOSDEM? So will I! I’ll be giving two talks in the Guile/Guix room, and I may be giving one more in another room, depending on acceptance or not. It would be good to meet other MediaGoblin community members or enthusiasts of MediaGoblin.

    I should also say that I was really not sure if I could make FOSDEM originally, but a number of people very kindly donated to send me on my way. Thanks very kindly! I’ll be sure to put it to maximum use for our community’s sake. :)

    We’re also thinking of running some sort of dinner (or lunch?) for those who donate between now and FOSDEM to MediaGoblin, so hey, by the way, our donate page still works! ;)

    Goblinoid updates

    Goblinoid, you may remember, is the result of this year’s Google Summer of Code, and is a neat MediaGoblin application for Android. Here’s an update (thank you Laura Arjona and Dylan Jeffers for writing this up):

    Dylan Jeffers, our GSoC 2015 student, is working in an Android app for GNU MediaGoblin.

    Current features include viewing the recent activity feed, comment about media, upload photos from file… taking advantage of the Pump API.

    The code is under heavy development, (repo here), and binaries for each release can be found (along with checksums) hosted at Goblin Refuge, a third party site kindly offered and maintained by SalmonLabs LLC, who also host the MediaGoblin public instance.

    Meanwhile we get the app in the popular free software repository F-Droid (it’s taking a while because it’s the first app in F-Droid built using the Python-based Kivy framework), we encourage everyone to test the app. The most up to date release is always available at and report feedback in our IRC channel or the repository issue tracker).

    We are very excited to make MediaGoblin part of your mobile life, and the Android app development is allowing improvements in MediaGoblin itself too (mainly fixes/improvements in the API-related code and the database and database migrations). Help us with your testing to improve that experience!

    Wrapping up this year, onto next!

    It’s been a busy year. Here are some highlights:

    • We took initiative on the challenges of user-centric hosting.
    • We got out a new release of MediaGoblin and dug in for the hard work ahead of getting towards 1.0.
    • We put your money to good use and funded Jessica to plow ahead with federation in MediaGoblin. This is well on its way, and we anticipate server to server federation to land in the next couple of months. Jessica just landed a massive overhaul to our database structure which was required to make this work, and that should make its way into the next release (coming soon).
    • We became more engaged with the work to bring federation beyond MediaGoblin itself. Jessica and I both joined the W3C Social Working Group and have become co-editors on the ActivityPump specification and have devoted much time to advance these initiatives. This includes building tooling such as ActiviPy which will be critical for putting federation to real use within and outside of MediaGoblin.
    • For the second year in a row, a major MediaGoblin member has received the O’Reilly Open Source Award largely (though not only)in recognition for their work on MediaGoblin. This year I (Chris Webber) received the award and last year Deb Nicholson received the award, which is rather incredible given the list of previous recipients of the award.
    • Both Deb Nicholson and I (but especially Deb) have given talks at prominent conferences and have spoken on podcasts on issues of network freedom.
    • We had a successful Summer of Code project resulting in Goblinoid, a free software client for MediaGoblin, which is Android and Replicant compatible. (It can run on GNU/Linux also!)
    • And as we said, we were accepted as recipients for the Stripe Open Source Retreat! (Well, accepted this year, and the results of that should play out into 2016!)

    That’s a lot of stuff… what a year!

    Well that’s it from this update! See you in 2016, and happy hacking, fellow goblins!

    State of the Goblin: September 2015

    Quick announcement: I’m going to be making two appearances over the next week! First I’ll be giving two talks on September 30th at Red Hat’s Chicago office for the Chicago GNU/Linux User Group on Guix and Federation (both mentioned in this post)! Second I’ll be attending the FSF 30th Birthday Party in Boston on October 3rd. If you’re able to make it to either, do stop by and say hello… I’m expecting both to be a lot of fun!

    Hello everyone! It’s been a while since a comprehensive update of what’s happening in MediaGoblin land. Despite the quiet, there is a lot to report, so let’s get down to business and start reporting!

    O’Reilly Award (again!)

    receiving the award
    Photo taken by Brandin Grams, CC BY 4.0, originally microblogged by Karen Sandler

    First of all, something fun: I was fortunate enough to receive the O’Reilly Open Source Award! (Yes, I know about the terminology mismatch, we’re free software people, but it’s still a great honor, and I was presented the award under the description of my free software advocacy and GNU MediaGoblin work.) The other recipients of the award is quite the incredible group of people, and I’m honored to be listed among them. But here’s what’s really cool: you may remember that MediaGoblin co-founder Deb Nicholson won the O’Reilly Award last year. How’s that as a vote of confidence in the things we’re working on?

    O'Reilly award, on display

    Anyway, if you want a more personal reflection, I wrote more on my personal blog!


    MediaGoblin 0.7.0: Time Traveler's Delight banner

    So right, what about shipping software out the door? Well…

    MediaGoblin 0.8.0: A Gallery of Fine Creatures banner

    Since the crowdfunding campaign, we’ve gotten out two major releases, 0.7.0: Time Traveler’s Delight and 0.8.0: A Gallery of Fine Creatures. I’m extremely proud of both of these releases! We have a lot more to do though on the road to 1.0, and we’ve been directly been putting the funds from the campaign to work to achieve that goal, so let’s talk about that.

    Putting your money to good work: Jessica and Federation

    Dropdown menu for administrative features

    You may recall that we hired on the talented Jessica Tallon to get federation working in MediaGoblin. Jessica recently gave an update on the state of federation. Jessica is doing great work, though as expected, converting MediaGoblin to be a federated project has been no small task (knowing what a big task it was, hope that we could hire Jessica on to do this work was my #1 goal in the last campaign, in fact!). This decision has turned out to be absolutely the right one. Some of the best parts of the last two releases have been adopting the client to server Pump API. Federation has been MediaGoblin’s goal since day 0, and Jessica is helping us to actually get there.

    However (and now I’m going to do a pretty technical deep dive, so you can skip this paragraph if that isn’t your thing), the most complicated aspect to making MediaGoblin into a federated project has had to do with updating the database to handle things while preserving data correctly for existing users. Why is this so complicated? A number of years ago we switched MediaGoblin over from using MongoDB to using either PostgreSQL or SQLite and while I believe this was absolutely the right decision, adding federation made the relational database system we had in place substantially trickier. For the more database-technically inclined, you can see that the ActivityPump API / Pump API require that any ActivityStream type object (in our case, that can be media or comments or even users) be referenceable by any type of activity. Furthermore, our existing comment system simply held that comments referenced media entries, whereas now comments can reference simply anything that is an ActivityStreams object. This means a large portion of our relations in our relational database needed an overhaul, and we needed a way to handle generic relations between tables. (The solution used is not unlike the “generic foreign key” implementation in Django.) There are more technical details on what has been done, but Jessica has been neck deep in this for months, but we believe we’re finally on the home stretch, in which case Jessica can finally knock out server to server federation.

    (I’ve thought that a whole post on database structure lessons learned may be a good blogpost of its own. One thing I’d note is that if jsonb had been an option when our current database design was put together, adopting that would have simplified things greatly, though it would require being PostgreSQL-only. But moving to that now would require a massive overhaul. If you’re starting a new federated project from day 1, maybe keep that in mind!)

    So the summary of all the federation stuff is: it’s complex, but we’re making good progress through Jessica’s hard work. Expect more on this soon, and huge strides in the next release!

    Federation and the W3C Social Working Group

    So something Jessica and I have both been involved in over the last year is the W3C Social Working Group working towards official standards for federated web applications.

    W3C Social WG, first GMG represented meeting
    W3C Social Working Group March 2015 face to face meeting attendees. Jessica’s holding the laptop on the left, and I’m right behind her.
    Photo taken by Aaron Parecki, CC0 1.0, originally posted to the W3C wiki.

    The federation protocol that MediaGoblin has been working towards until this point is primarily based on the Pump API, but this is really just a semi-formalization of the interface for the API. In the Social Working Group we are working towards defining a new standard, ActivityPump, which is based off of the ActivityStreams 2.0 standard. We’re very excited with where this standard is going and feel it’s a clean refinement over the Pump API we’re already working with, while still keeping many of those same conventions.

    W3C Social WG, second GMG represented meeting
    W3C Social Working Group May 2015 face to face meeting attendees. I didn’t make it to this one, but Jessica did, and did a stellar job representing MediaGoblin and ActivityPump!
    Photo taken by Aaron Parecki, CC0 1.0, originally posted to the W3C wiki.

    This has taken a lot of our time, but I believe it the results are worth it. Jessica and I have been attending weekly calls related to this standardization, and have thus far attended two face to face meetings at well. (More accurately, Jessica attended the second MediaGoblin-represented one without me, giving a kick-ass presentation on how ActivityPump works to the group! Go Jessica!)

    As for my own personal work advancing this, I’ll go into this a bit further on in this post!

    Google Summer of Code result: Goblinoid!

    Current screenshot of Goblinoid
    Screenshot of Goblinoid, as it looks now!

    Dylan Jeffers joined us for Google Summer of Code student this year work on a pretty cool project: a MediaGoblin client for Android or… really nearly anything… called Goblinoid! There’s two really interesting features about Goblinoid: one, it’s written in Kivy, a GUI toolkit which emulates the Android look and feel, but is actually can run nearly anywhere Python can run… making it quite portable, yet ideal for mobile computers!

    Mockup of Goblinoid future UI
    Mockup of what Dylan would like Goblinoid to look like in the future!

    So Goblinoid works… it could use more user testing and packaging, if you’re interested in helping with that! But you can already upload images on the go via Goblinoid, and we expect more to come. Give it a go!

    We’ve long been interested in having a client for MediaGoblin which makes use of the Pump API implementation we’ve been working on. Thank you Dylan for making that happen!

    Infrastructure challenges

    This has been a challenging year as in terms of supporting MediaGoblin’s infrastructure. Spammers attacked both our wiki and bugtracker hard, at one point leaving me to take several weeks to fight issues and to try to find solutions. (Unfortunately, for the bugtracker, no great solutions have been found, and we are on a request-an-account basis… not a great situation to be in.)

    Additionally, the primary Gitorious instance went down, where MediaGoblin’s code was hosted along with many, many feature branches from contributors. The MediaGoblin community spent a while debating what we were going to do. A move to GitHub was tempting but is not an option; that’s exactly the opposite of the type of world we want to build. Not everyone in MediaGoblin’s community is comfortable with GitLab, and their primary instance is running a proprietary version. There are some other communities hosting things like Notabug who seem to be run by great people, but we run the risk of running into these same problems all over again… though so we did with most of these other solutions. We could self-host, but there is very little time for extra server maintenance burdens right now. So we moved our git repository over to Savannah and I’m glad to have hosting there by people I trust, though I do also feel that contributors may expect more modern hosting facilities, but we don’t have time to run them ourselves.

    The Gitorious shutdown came around the time of great exhaustion of already dealing with issues related to the bugtracker and left me feeling very burnt out. But it also lead to a great amount of reflection… who am I to feel frustrated with? The Gitorious people graciously hosted our software repositories for some time, and I was not willing to run an instance of my own. Why not? Well, one problem is that if you run your own instance of a free software web application that isn’t federated, you’re working in Yet Another Semi-Free Micro-Silo (TM). But let’s face it, that’s not the real major problem… looking at the frustrations we’ve had with Trac, the answer is obvious: running free network services is a huge maintenance burden.

    But wait… yes, you may be catching a whiff of irony here… if I’m not willing to run a free software web application because I don’t want to take on the maintenance burden of someone else’s software, how can I ever expect MediaGoblin to gain adoption?

    And here’s where I come to a tough, but I think necessary, conclusion: there’s simply no way for MediaGoblin to succeed if the world of deployment stays where it’s at. Something must be done. But what?

    Research into deployment and federation

    Partly affected by the above, summer came around, and I had a talk with MediaGoblin contributors in our monthly meetings. I wanted to take a sabbatical… a sabbatical where I was on break from “direct” MediaGoblin things, so I can advance things that affect MediaGoblin greatly indirectly. (My spouse and I were also going through a big move so this was a good time to do it, I figured.) I wanted to do research into two things: deployment and federation.

    Framing the deployment side of things, Deb Nicholson and I gave a talk that kicked off “userops”, a term that I still feel accurately captures what we’re trying to do (and a talk which I still believe accurately summarizes the challenges we’re facing). Within this context I began exploring options of what can be done to improve deployability.

    The directions I’ve explored, and why I’ve come to the conclusions I have, are a series of blogposts of their own (if you’re interested, I suggest subscribing to the userops mailing list, where I will be posting more as I go). The short of it is that I laid out a set of requirements to achieve easy and maintainable deployments, attempted to explore them with the most popular current tools (Ansible, Puppet, Salt, Docker, etc), and found that it is not possible to build both easy and maintainable systems on top of them based on what I believe users need. This lead me eventually down the path towards Guix, a package manager (and with GuixSD, also a distro) and soon to be deployment system which I am very confident has the potential to solve many of the challenges which make deploying and maintaining systems too exhausting a task for the average user. The software is nowhere near being “easy” at present (I think it’s very telling to say that it’s “the Emacs of package managers / distributions”… one can extrapolate on that in many ways, most of which are correct, except for the Emacs-haters kind… lay off, Emacs haters!) but I think has the potential to become so. An easy to use web user interface has already been demonstrated, and I believe the foundations are good for building a complete and easy to use system for everyday users. But again, to go into detail beyond what has already been said is something that will be explored elsewhere.

    There is a more pressing need for me to have explored deployment at present as well… though I want to explore deployability for the sake of other users, I also need to explore deployability for my own sake… particularly because we promised that we will provide premium hosting in the last campaign we ran! Yes, in case you have been wondering about it, I have not forgotten about this promise. How to fulfill this promise without being crushed by the maintenance burden of hosting? I came to realize that there was every risk that I would spend all my time supporting and maintaining servers that were running MediaGoblin and I would not have the opportunity to any longer be a steward of MediaGoblin itself… which could lead to failure. So figuring out a better path forward on hosting has become a necessity. I’ll explore what this means further in the next section, but first, I should say a word on federation and what my sabbatical lead to on this front.

    As I have said, Jessica and I have been involved in the W3C Social Working Group. Part of the activities of the group have been the definition of the ActivityStreams 2.0 and Pump API standards.

    I set out to explore this a bit more deeply under a repository with the joking and interim name of activitystuff (the Social Working Group is using GitHub for its work, so I’m making an exception there). Along with some other projects, this has contributed to a deeper understanding of how federation should work, which is something useful to take back to MediaGoblin. There are some potentially useful things in that repository (including a partially complete implementation of the JSON-LD API), and it may turn into a full implementation of ActivityPump, though its original and primary goal was for exploration, a purpose it has served well.

    One major event relating to federation has just occurred over the last week and bears note here: a number of Pump.IO community members and myself are now working with Evan Prodromou (who has near-certainly contributed more to the federated web space than anyone else alive) to transition the project from being a project of primarily stewardship under Evan to one of community stewardship and governance. I posted a summary of a recent meeting, and (MediaGoblin contributor!) Laura Arjona put together a community document. In sum, Pump.IO could use your help if you’re interested.

    So anyway, it’s been a busy last couple of months! But it’s time to return to MediaGoblin-ville, so…

    What’s next?

    Well, that’s a whole lot of text above, so how about a bulleted list next? I hear those are easier on the eyes.

    • I’m swooping back into MediaGoblin territory starting next month. However, my initial focus will be on getting MediaGoblin to a deployable point for myself, particularly forward-looking towards making premium hosting feasible. Expect more soon!
    • Jessica is working on MediaGoblin federation. We expect the massive database changes she has been working on to tidy up in the next few weeks, and if all is well, we’ll have server to server federation basics in place by the end of the year.
    • Work towards 0.9.0 will proceed, unless 1.0 lands first! Expect major infrastructure improvements in 0.9.0, and federation to land in 1.0.

    Those are all great things, and I think we are indeed on track towards our goals, slowly but surely.

    There is a more rough side to this: we’ve allocated nearly all of the funds from the last campaign towards paying for Jessica’s work on MediaGoblin, and she’s devoted enough to the project that she’s been working well, well below market rate. (And aside from some travel reimbursements, I have not taken money personally from the last campaign.) I’ll post a financial transparency report soon, but the short of it is: the MediaGoblin funds are running low. Even with Jessica generously working for such a relatively low amount of money, we won’t be able to pay Jessica to work on MediaGoblin much longer, given our present finances.

    But we aren’t giving up! The goals of MediaGoblin and of federating the web are just too important, and so onward we press, into the future! Though MediaGoblin is an ambitious project, I am confident we can achieve the things we have set out to do, but we are constrained by limited resources and time.

    Appreciate what we are doing? Want to help us in our quest to bring network freedoms to everyone? You can help!


    • The simplest way to help? Donate! Though the official campaign is over, you can still donate to MediaGoblin through the FSF!
    • Jessica and I have been contracting so we can pay the bills (doing this allowed us to “stretch out” the amount of time Jessica could spend on the project… useful with all that W3C stuff in progress!). Unfortunately, while the people we have been working with are great, the contract we’ve been working on is coming to an end. Are you looking for contractors or part time workers who are capable of developing high quality free software and providing community leadership? Jessica and I are both interested in working in such cases… feel free to email me at: cwebber AT dustycloud DOT org

    We’re committed to making a better, decentralized web. I hope this piece cleared up where we’re at and where we’re going. I believe we’ve got exciting times ahead… till next time, goblins!

    The state of Federation

    It’s been a long time since there has been any news on the state of federation, so here’s an update on where MediaGoblin’s at and some technical aspects of federation. We’ve been working with the W3C Social Working Group to define the future of federation, and part of my work there has been to work on the ActivityPump standard. There’s more to say on that and why we’re investing time there, but this blogpost will mostly be about MediaGoblin and federation from a technical perspective.

    Over the last year I’ve been building up the foundations of federation, creating the necessary APIs and infrastructure needed. I am currently finishing the last of that, modifying the models to handle federated data. This is a challenge as the models were designed without federation in mind. There is a lot of new data that is being stored as well as remote and local versions of the models.

    Once I’ve finished working on the models I’ll be starting the work on actually getting the media and comments federating across instances.

    The core of the federation and APIs are "Activities", these are created all the time right now in 0.8.0. Every time someone posts media, comments, tags or updates anything. These serve two main purposes:

    • Command language: instructing the server to do something
    • Log: telling clients and other servers what has been done

    The activities at their core at subject predicate object, for example:

    Chris posts an image

    The activity also usually contains the audience targeting data (i.e. who it’s for). These are represented in JSON with elements being nested as other JSON objects, lets take a look:

        "id": ""
        "verb": "post",
        "objectType": "activity",
        "actor": {
            "id": "",
            "displayName": "Christopher Allan Webber",
            "objectType": "person"
        "object": {
            "id": "",
            "objectType": "image",
            "url": "",
            "fullImage": ""
        "to": [
                "id": "",
                "objectType": "collection"

    This might look overwhelming but it’s the same basic subject predicate object sentence above. Chris (the actor) is posting (the verb) an image (the object), it also contains some extra information like IDs so they can be referenced and the audience which is a a collection of people that follow Chris.

    Once we have this activity, what actually happens, you ask? Each user has an inbox and outbox like you do with email. Step by step walkthrough of what would happen when someone posts an image from a desktop or mobile client:

    1 Client to server

    back a preliminary object to use in the API.

    1. The client creates an activity that at minimum would look like this:
        "verb": "post",
        "objectType": "activity",
        "object": {
            "id": "",
            "objectType": "image",
        "to": [
                "id": "",
                "objectType": "collection"
    1. This activity is then posted to the client’s inbox.
    2. The server receives the activity, creates an ID, and attaches the actor and other metadata, then saves it.

    2 Federation!

    where we’re all on our own servers. This is what the server (once it’s finished) will do once it gets the client’s request:

    1. The server looks for whom it’s sent to (the "to" property) to get a list of people that the object has to be sent to.
    2. The activity is then sent to each persons inbox.

    As one of Chris’ followers you can now open up the website or any client and see Chris’ image. Commenting on the image will produce an activity which is sent to Chris, who subsequently sends it as an update to everyone else.

    Phew! we’re done, that’s how the API and federation works in MediaGoblin. We’ve successfully posted an image from a client and had it sent around and hopefully you’re able to see how a comment or something else would work in this system too.

    This is just a high level overview, there are many technical problems such as access control (who’s able to view content), verification of objects (checking what we’ve been sent is correct), etc. Federation doesn’t come easy but we think it’s well worth it.

    MediaGoblin 0.8.0: A Gallery of Fine Creatures

    | tags: release

    MediaGoblin 0.8.0: A Gallery of Fine Creatures banner

    We’re excited to announce that MediaGoblin 0.8.0, “A Gallery of Fine Creatures”, has been released! The biggest news is that the client to server API (making use of the future federation API) is much improved! That means that users no longer have to depend on a browser to access MediaGoblin. You can access and post to your MediaGoblin instance via any of several compatible clients, like Pumpa and Dianara (or write your own using PyPump)! The grand goal is a generic (and ubiquitous) client protocol that will work with lots of different served applications that use the pump standard. Eventually, any Pump API compatible client will essentially be a MediaGoblin client and a Pump.IO client. We expect more client types to be added very soon!

    Part of the process of world domination via federation means that we’re now able to support serving content to multiple client types through a single protocol.

    For example, here’s a user uploading an image using the Pump.IO client, Pumpa:

    Pumpa uploading image to MediaGoblin

    And here’s that same image, now uploaded to MediaGoblin!

    Image from Pumpa now on MediaGoblin!

    In a nutshell, the client to server part of the API/federation equation is working. We are still working on server to server federation that will enable us to share comments, tagging and all the other things that can happen to your shared content on someone else’s server that you may (or may not) want hear about.

    By the way, if you’re using MediaGoblin with Apache (rather than Nginx or some other setup), you’ll need to add “WSGIPassAuthorization On” to your config or the API won’t work. You can look at this wiki page for reference.

    Speaking of updating and getting with the times, we are officially offering preliminary support for Python 3. Most of our features work without Python 2 installed, so welcome aboard futurists! As always, if you spot something we missed, we’d love to hear from you via our bugtracker, mailing list or IRC channel.

    Also on the upgrade list for this release, GStreamer! We’re now using version 1.0 which adds a nice new thumbnailer and includes much improved video transcoding support. If you didn’t get thumbnails before, it wasn’t you. We encourage you to try again and see if it works for you now!

    Video thumbs, rendering nicely
    Thumbnails from Venom’s Lab, licensed under CC BY-SA 3.0

    Obviously, the future demands cleaner packaging. Configure and make support is now the default… welcome to fewer steps for installing your MediaGoblin instance, which will be critical as we build packages for Fedora, Debian and any other distro that wants to help its users host the federated future of the web.

    We’ve also switched away from Transifex, which had become proprietary (boo!) to an instance of Pootle that many of our fellow GNU projects are now using for translations. If you’ve been looking for a fully free translation tool, Pootle may be just the thing you’ve been looking for!

    And finally, we fixed the footer. It is now forced to the bottom of page, instead of floating in the middle of short pages which everyone agrees was sub-optimal.

    Next up is server to server federation. This is what Jessica Tallon has been working on full-tilt — thanks to everyone who pitched in on the MediaGoblin campaign. (Though the campaign is over, you can still donate!) Both Chris and Jessica have been participating in the W3C Social Working Group on a federation standard. (As well as a general purpose client to server API… you better believe the increased client support coming out of this release is related!) The design is very closely related to the existing Pump API spec we’ve been using. You can read the current draft of the standard here. We’ve got more news on the way, including on that federation front. Expect more news soon!

    Thanks to everyone who is helping us make the future of the web happen everywhere! This release wouldn’t have happened without the help of these people: Alon Levy, Asheesh Laroia, Andrew Browning, Berker Peksag, Boris Bobrov, Christopher Allan Webber, Deb Nicholson, Ineiev, Jaakko Luttinen, Jakob Kramer, Jeremy Pope, Jessica Tallon, Jim Campbell, Laura Arjona, Meg Ford, Rodrigo Rodrigues da Silva, and Ben Sturmfels. You all rock!

    Want to put in your own help towards the future of federated, awesome media publishing? Join us! Visit us in IRC (#mediagoblin on or sign up for regular updates on our mailing list Got ideas or questions about our work? Email us at press AT mediagoblin DOT org — we look forward to hearing from you!

    Userops: Deployment for the People

    Deb Nicholson and I both recently gave a talk at FOSDEM 2015 called “Can Distros Make the Link?” (A recording is here, and my slides are here, hit “s” for speaker notes or read the org-mode source if you prefer.) The main purpose of the talk was that packaging libre network services/applications for distros is important, but distros in their present forms aren’t really enough to solve the deployability problems and pains that anyone trying to run their own libre servers knows. I had a bit of a worry that this thesis would upset part of the audience (it was in the distros room, after all) but it turns out that everyone seemed to agree and be on board.

    Many audience members even encouraged us that this conversation needed to continue beyond FOSDEM, and there was discussion of hosting a mailing list to continue the conversation. As usual, everyone had various ideas of where to host it, but the audience seemed to feel that MediaGoblin’s servers were fairly neutral ground, so we announced that we would put up a mailing list and announce it here when we got the chance.

    It’s a bit delayed, but I’m happy to announce the launch of the userops mailing list! If you’re interested in talking about making deployment easier for every-day users, please consider joining the conversation there. (Oh, and we also have an IRC channel: join #userops on, if you’re the IRC type!)

    Why the name “userops”? As you may have guessed, this is a pun on the term “devops”; the idea is that we also care about configuration management and deployability, but we aim for a different audience. Devops, as the name implies, focuses on liberating developers in the world of deployment, particularly developers who have to deploy a large number of machines for $LARGE_CORPORATION at their job. Userops, on the other hand, aims at liberating users in the world of deployment. You shouldn’t have to be a developer to take advantage of network freedoms and run network-oriented free software. After all, the free software world generally agrees that it makes sense that users of desktop software should not have to be developers, and that “user freedom” takes priority over “developer freedom”… the freedom of $LARGE_CORPORATION, while not something we object to, is not really our primary concern. (Though of course, if we build solutions that are good enough for end-users, corporations will probably adopt them, and that is fine! It just isn’t our focus.)

    (Oh, and in case you stumble upon it, “userops” was originally a name I had for one of my personal projects experimenting with deployability, but a friend of mine convinced me that the term was too useful to be constrained to one particular piece of software, so I’ve renamed that project! Everyone is now free to use the term “userops” to refer to the vision described above.)

    We believe that “userops” is now more important than ever. These days, it is not just enough to use free software network services, one must have the ability to deploy and make use of that software. (For many of this, the timeliness and urgency of this is seen with the turmoil for many free software developers figuring out where to go now that the hosted version of Gitorious is being shut down.) And as you may have guessed, we’re well aware of how true this is not of just “all that other libre network services”; MediaGoblin requires quite a bit of technical skills and resources to run. We’d like to improve that, but we think there are some real challenges that are beyond what MediaGoblin can do as MediaGoblin itself: things need to happen on another layer (or layers) too. Hence “userops”!

    If this is something you likewise care about, and especially if it’s something you’re working on or thinking about (or would like to), consider joining the conversation!

    MediaGoblin 0.7.1 released

    | tags: release bugfix

    MediaGoblin 0.7.1 has been released! This is a bugfix release building on MediaGoblin 0.7.0.

    Upgrading is highly encouraged to those running PostgreSQL databases especially. There were some issues in the previous release that lead to users of PostgreSQL to see random errors. We had some problems related to a couple of (non-critical) features not handling transactions well… these have been disabled for this release, but will likely be re-enabled by 0.8.0’s release.

    Thanks to everyone who made this release possible: Andrew Browning, Christopher Allan Webber, Jessica Tallon, Low Kian Seong, Matt Molyneaux, and Odin Hørthe Omdal. Thanks so much!

    Please see the release notes for details. Happy goblin’ing!

Page 1 / 8 »