Forum:Weird Gloop week 2 update
 |
This page or section is an archive. Please do not edit the contents of this page. This thread was archived on 8 January 2019 by TyA. |
Hey everyone! We’re now a week into the launch of the new wikis, and I want to give a quick update about where we stand. It’s been quite a busy time, and there’s a lot to talk about.
In short: we’ve finished the most critical part of leaving Wikia (namely, launching the new site), and that was a pretty major accomplishment -- but there’s still a lot left to do to make the site awesome, and move the traffic over.
I’m posting this thread on meta rather than Yew Grove, because it affects both wikis. This is the first thread on meta, but it won’t be the last -- in the future, I’d like it if a lot of general discussions like this happen on this forum, with links from YG and OSRS’s Watercooler (and other wikis, should we choose to support them).
If people appreciate this sort of information, I imagine we will do similar informal updates/discussions in the future.
Wikifest and RuneFest
In preparation for the new wiki launch on October 2nd, eleven of the wiki’s brightest minds (and me!) got together in Cambridge, UK, to finalize the launch, do some merging, and do some quality control on the new site. Thank you to everyone who took a week away from real life to make sure the launch went smoothly, as well as everyone else who couldn’t make it physically, but still put in loads of work.
We spent the first three days of the wikifest working through a lot of the technical blockers for the launch, and after a really productive meeting with Jagex (more on that later!) on Monday night, we did the final merge of all of our changes onto the latest content from the old wiki.
The actual launch happened at about 3pm on Tuesday, and it wasn’t exactly smooth -- we launched with only one web server instead of two, and underestimated the power of the newsposts and RuneScape broadcast, which created a spikier load than we anticipated. This resulted in a not-great experience for the first half-hour or so, but thanks almost entirely to Kitty and Jayden, we recovered and ended the day well.
On Wednesday, Spine and I were on the Old School RuneScape livestream for a short time to talk about the new wikis, and we talked to Ash for some time afterwards about wiki integration, accessing drop rates, how ladders work, and weird gloop (of course).
We went down to RuneFest later in the week, and in addition to all the regular festivities, we had a great time on the RuneFest livestream talking about the new wiki and demo-ing a couple new projects (we managed to only show the parts that weren’t horrendously broken, yay!). It was fun talking with all of the folks at Jagex that helped make the fork happen, as well as everyone (JMods, streamers, everyone else) who use the wiki all the time and were super supportive of the new site. It makes me really hopeful that we’ll win the traffic battle sooner rather than later.
I can’t overstate how cool it is to actually meet all of these people we’ve gotten to know virtually over the last decade. I’m hopeful we can make this sort of get-together a regular thing, with more people -- it’s looking like the next one might be on the granite shield island…
Coordination with Jagex
We had a meeting last Monday with Cam, Gambit, Matthias, Lenny, and a guy from the analytics squad whose name I don’t remember (sorry analytics guy! you’re still loved). Points of discussion:
Wiki integration
Jagex is very interested in adding ways to access the wiki from inside RuneScape and Old School RuneScape. Discussions on this topic have centered around two main areas: searching (e.g. the /wiki command in Guild Wars 2), and entity-based linking (e.g. clicking on a specific item/NPC and having it go to the right article/section on the wiki).
We have a meeting next week with Cam, Dave Osborne, Ash and others TBD to figure out what our capabilities are, and how much of the entity mapping can be offloaded to us instead of Jagex. I have a feeling they would be very unhappy with me if I made any promises here, but based on the people involved (and the fact that the project already has an active project manager) it’s clearly something they’re taking seriously as a project.
Localisation and other wikis
Jagex is very interested in the possibility of us supporting localised (non-English) wikis, as well as the RuneScape Classic wiki. As before, this is subject to our approval (nobody’s forcing this), and certainly wouldn’t happen until we were in a more stable technical state. Discussions focused on:
- the current pt-br wiki on Wikia (who we’re close with)
- the German wiki at schnupptrupp.org (I had a great chat with one of their two admins, Bowserkor, at RuneFest)
- the French Wikia wiki being run by the guys from FrancoScape
We discussed with Jagex the technical benefits of having common image and data sources across all of the wikis, and we think this is a pretty compelling argument for working together. We’re starting to talk to the main people on these wikis and figure out to what degree working together will be beneficial for all parties.
Personally, I think the biggest benefit of working with these other wikis is bringing their really gifted editors/admins more directly into our community -- for example, Sandro from the pt-br wiki has done a huge amount of the work on the interactive maps project, and the Schnupptrupp guys have been doing a way better job than us at Grand Exchange prices in MediaWiki for, like, 5 years.
Incentives for editing
Subject to community approval, Jagex is open to two options to incentivize people to edit:
- We have budget approval to give out a certain number of bonds for contributing to the wiki. It’s not exactly clear how these would be used: there are some data-gathering projects (like disassembly or RuneMetrics) for which there’s a natural way to distribute rewards (we’ve already done this on our own dime before), but it’s less clear how to do that in a clean way for wiki editing, cleanup and project contribution. One place we can take inspiration is the Guild Wars 2 official wiki, which has a project called On Wiki of Gold that rewards a small amount of gold in exchange for contributing to an article or set of articles. I need to confirm the exact number, but I’m pretty sure we have approval for 10 bonds per week, which we could split across the games.
- We also have tentative support for some sort of cosmetic (or perhaps not entirely cosmetic) reward for either 1) linking your account with the wiki, or 2) editing the wiki. I think we’re still trying to agree on how to best leverage that opportunity to improve the editing community, without all of the internal controversy that has surrounded the Wikian title. If we wanted to relate this to account linking, there’s a significant amount of technical work that would need to happen to make account linking between RuneScape and the wiki work, and we would need to make a very compelling case that this kind of linking is worthwhile (in terms of personalization, or data sharing, or something else). I don’t think we’re quite there yet.
Again -- at the end of the day, it’s our decision whether we want these incentives. My personal view is that both of these could be massive wins for the editing community, but we can have that discussion at a later date when it’s clearer on a technical level what Jagex can offer.
Game data sharing
Again, still very early in the process, but I think there’s a very good chance that we can get some really deep server-side information from Jagex, including monster stats, drop tables, skilling success rates, bonus XP ordering, and more. This would be huge on a number of levels, and I’m curious if anyone has suggestions for things I’ve forgotten about that we might want.
Technical
Even after the launch, the sysadmins (Jayden and Kitty) have been putting in a shit ton of work to iron out all of the issues we’ve been facing. Here’s just a few:
- Building the link tables and Semantic MediaWiki tables so that dynamically generated content (money making guide tables, dropping monsters, etc) actually get populated
- Patching our new version of DPL to properly work with our old use cases
- Fixing problems with image thumbnailing
- Modifying our robots.txt to make sure that Google treats our site with the tender touch it deserves
- Quickly resolving multiple high-pressure outages
It would be hard to exaggerate the effort it’s taken from both of them to get us here. The success of this project, more than anything else, comes down to their efficacy and dedication.
Outages
We have had some difficulties keeping the site up for the entirety of the last week. This is to be expected: while we’ve had a lot of time to practice, there’s no substitute for actually scaling up to deal with major load.
According to my calculations, we’ve had 4104 seconds of real downtime in the last 8 days. That’s good for about a 99.4% uptime -- not great, but not bad for the first week. Of those 4104 seconds, 3234 of them were in three separate incidents due to an overly aggressive database backup, which has since been scaled down to not compete with the webservers for access to the database. Another 520 seconds’ outage yesterday was due to an edge case in how one of our caches handled forked processes when it was close to 50% memory usage -- this has since been reconfigured and we expect the underlying issue is resolved.
There’s also been periods of time where only one of the servers was running, potentially resulting in a degradation of load times, but most likely not actual downtime. The biggest of these were caused by a disk quota issue on the web servers that has since been resolved.
We obviously haven’t seen the last outage, but I think we’re moving closer and closer to a point where the trouble spots are known to us, and we’ve taken steps to prevent those from happening again. We’ve handled the downtime without arousing too much vitriol from the community, so I consider that a win.
In the longer term, there are a number of improvements we need to make to reduce downtime and impact -- obvious things like better resource monitoring will be helpful, and Kitty had a really interesting suggestion to utilize Cloudflare’s “always on” policy to redirect to Google’s cached page version in the event of an outage.
Remaining big tasks
There are still some technical hurdles to jump over before we’re really going to be in a stable position. These include:
- Thumbnail syncing issues
- Caching/Redis synchronization issues
- Moving Grand Exchange prices over to an extension instead of vanilla MediaWiki
- Supporting cookie-based theme switching in the backend
- Maximizing our usage of Cloudflare, with an eye towards possibly moving from OVH to AWS in the future if we can reduce bandwidth substantially
Traffic recovery
This is probably the part you’re most curious about. Currently, we’re tracking at about 25% of the previous traffic of the old site. I don’t know whether this seems high or low to you, but I’m pretty happy with that as a first-week milestone. Remember that approximately 70% of visits to the old wiki started with a Google search, and in general we’re not winning on Google yet.
The way I see it, the new wiki has already gotten buy-in from everyone who has their ear to the ground and pays any attention to RuneScape (whether that’s newsposts, Reddit, Twitter…). If you take a look at the links that are getting shared there, they’re overwhelmingly to the new sites, and without looking at the actual site traffic, you’d have a hard time believing anyone uses the Wikia wikis.
Unfortunately for us, a lot of people just aren’t paying attention yet. In media studies, there’s this idea of the 90%-9%-1% rule, where 1% of the people actively produce content, 9% occasionally contribute or make small comments or fixes, and the remaining 90% are just lurkers that just consume without actually contributing anything, and aren’t so actively engaged. I’m not convinced those are the most appropriate percentages for our situation, but the idea remains: we got buy-in from the 1% in the Yew Grove thread, and we now have the support of the 9% that are involved with Reddit and Twitter. It’s that last 90%, that to us barely even exist besides as impression numbers, that we’re going to have the hardest time getting.
It’s tempting to say we shouldn’t care about them, because we have the entire active community on the new site. I certainly see the wisdom of that, but at the same time, it’s going to be important to Jagex in the long run that we get those other readers moved over...eventually. This project won’t be a complete success unless we get both the editors and the traffic.
With that in mind, here’s an overview of what we’re doing to gain the traffic back on the new wikis.
I mentioned earlier that before the fork, about 70% of our visits started with a Google search. (In comparison, about 20% started organically, 3% from the OSRS wiki, and the remainder spread across other search engines and Reddit and YouTube). Unfortunately for us, people are so used to the wiki being the top result for a lot of searches, they’ll ignore our site search entirely and just use Google to get to the next page they’re looking for. This is obviously not ideal.
We’re doing everything we can to get Google to react ASAP to the new changes. We’re now one of the top results for “RuneScape Wiki” (along with three “leaving Wikia”-related posts), and the OSRS Wiki is coming along nicely too -- although the key here is the content searches. Google is currently indexing the entirety of both of the wikis, and we expect to start showing up as the top result for new content in the next couple of days. Jayden, Kitty, Gaz and I are monitoring Google’s search console to make sure we’re doing everything in our power to adhere to Google’s guidelines and get good search positions for our competing articles. My feeling is that once we are one of the top couple results for any particular search, the searchers will really start moving over. It’s not absolutely essential that we’re number one ASAP -- just get us on the first page and I’m happy.
I don’t want to give any illusion that this will happen overnight. I expect we will beat the old wikis on new content fairly quickly, I would be really happy if we managed to be competitive with Wikia on old content in the 3-6 month range. This is really, really hard, but we’ll get there eventually.
The Reddit community, from the moderators all the way down to the trolls, have been incredibly supportive (in some cases a bit TOO supportive) of the new wikis. The mods have set up AutoModerator to shame anyone who posts a Wikia link, they’ve sticked our threads, they’ve given us flairs...they’ve been awesome. The trolls on /r/2007scape vandalized the old OSRS wiki to the point that Jagex more or less asked *us* to go clean it up, so we had to go ask them to politely knock it off. People have been making all sorts of extensions and redirect scripts to help people avoid the Wikia wikis. Point is, the community is overwhelmingly on our side here.
Let’s make the most out of that. When a new update comes out, post the main wiki article on Reddit and watch it zoom to the top. If you’ve worked on some big new article or feature, post it! People love that shit. Even if you’re just trying to get support for some new thing you’re working on, just go ahead and mention it on Reddit. I think we’re dramatically underutilizing the subreddits as a method for getting people on board with the new wiki, because it not only helps us out big-time that one day, but it builds up our profile for Google in the very long-term.
Newsposts
This hasn’t happened yet, but in the future we’re going to try to get links to the main wiki pages for content (e.g. the 2018 Hallowe’en event) from the official newsposts. We think this is going to be super useful not just for organic traffic, but also for giving us an immediate advantage on Google -- in the absence of a dominant Wikia article about a topic, the top result is usually the official newspost, so getting a link from there is going to go a long way.
Twitch and YouTube
Until RuneFest, I was pretty unfamiliar with the Twitch and YouTube sections of the RuneScape community. After talking with some of the streamers and YouTubers about the new wiki and hearing how enthusiastic they all were about it (particularly Faux -- thanks bud!), it’s clear to me that Twitch and YouTube should be part of our traffic strategy.
The ~100k people that watch a random Torvesta video are, by and large, a very different crowd than the ones who are active on Reddit or interact with us on Twitter. They’re a huge part of that 90% I keep talking about, that probably don’t even know there’s a new wiki. If we can get as many of these guys as possible to talk about the new site for a bit (or just complain about Wikia, because honestly that’s probably more entertaining), then we’re reaching a much wider audience than before. I’m exploring the best way to make this happen, possibly in collaboration with Jagex.
You may also know that I’ve been pushing to get some of the Old School wiki people to start streaming their editing (especially things that are partially ingame, like taking images or working on a new update). Based on our Wikifest streams from last week, I have a feeling that this sort of thing could be surprisingly popular, and get a lot of people interested in wiki editing that would have zero involvement otherwise.
Ingame integration
We’ve already gone over this in a fair bit of detail earlier in the post, but if we had meaningful ingame integration with the wiki, it would totally overtake Google as the most popular route for reaching the sites -- my estimate is that in a fully operational scenario, about 50% of wiki visits would start with an ingame interaction. This is why I’m not so existentially worried about the Google search results not coming super quickly.
Wikia editors
We’ve done a phenomenal job so far at moving all of the Wikia editors over to the new site -- when I can sleep 8 hours, wake up, and the only edit to the Wikia wiki is this mess, I consider that a success.
Let’s just keep it up and monitor RecentChanges on the old wikis for anyone who’s still unaware of the move.
Projects
I’ve alluded to this in other places, but the single biggest thing we can do to differentiate ourselves from the old wiki is to work on big projects that are highly visible to the community. Some things that have been boppin’ around in my head:
- Dark mode and general support for localStorage-backed theme switching, even for logged out users
- Doogle Maps: interactive RuneScape maps to replace our static images, optionally leading to a much more feature-rich Google Maps-style UI, including pathfinding.
- Numerical drop rates: leveraging RuneMetrics data and possibly third-party OSRS client data (and perhaps even direct help from Jagex), we can now start to add actual drop rates to monster drop tables on the wiki.
Any of these would be awesome to finish, because then it becomes a thing that people start to miss when they accidentally hit the old wiki, and posting about it on Reddit is basically an automatic full day of “holy shit!” attention to the new site.
I’m curious what ideas other people have for major new projects, now that we’re unconstrained from Wikia’s limiting architecture.
Board elections
(If you’re not interested in joining the Weird Gloop board of directors, you can stop reading this section)
We don’t have exact dates set on this yet, but a friendly reminder that we’re starting to get the ball rolling on elections for the Weird Gloop board of directors. See this page for some under-construction (and subject to change) information about the purpose, process and requirements.
We’re going to begin our call for candidates some time soon. Anyone interested in joining the board (including and especially non-admins) should think about what they’d like to say, and what role they see the board playing in the future of Weird Gloop.
Discussion
Support - What are we supporting? ʞooɔ 20:47, 10 October 2018 (UTC)
Comment - great update Cook :) One other thing to mention is that there is now a notice up on wikia stating that their RS wiki will be moving to fandom.com. I can't pretend that I know how this works, but hopefully it will not be favourable to their SEO. IsobelJ (talk) 21:03, 10 October 2018 (UTC)
Seems good — I appreciate this sort of information and would like to see similar informal updates/discussions in the future. --laagone talk 21:28, 10 October 2018 (UTC)
Comment - Nice. ~ty 01:16, 11 October 2018 (UTC)
Support - I am very interested in this information (else I wouldn't have read it, and probably not posted this, so I suppose that is kind of a redundant statement). ⚞Myles Prower⚟ 03:48, 11 October 2018 (UTC)
Nice update. I like updates. Some good ideas with the incentives for editing too - some OS bonds might be nice, especially since that's the more played game justsayin ;-) Adragon111 (talk) 05:41, 11 October 2018 (UTC)
Woot - We're getting there, but still a long road ahead of us. // Salix // Talk-to Salix // 19:03, 12 October 2018 (UTC)
Support - Thanks for the update. –sumone10154(talk) 22:50, 17 October 2018 (UTC)
Thank - Appreciate the update, I like reading about this sort of stuff, and the wiki's progress in particular. {{SUBST:Template:Signatures/Cycloneblaze}} 20:16, 18 October 2018 (UTC)
For users without Google Analytics access, could you do an update on traffic for the two months since? --Iiii I I I 01:00, 27 December 2018 (UTC)
Closed - Discussion has died. 
02:28, 8 January 2019 (UTC)