I encountered a pretty scary error on the WordPress Multisite network I manage yesterday, which turned out to be something relatively minor. I thought I’d blog it in case someone else found themselves in a similar situation and began frantically Googling… which is what I was doing last night as my wife (thankfully) handled getting the kids to bed.
We think we’ve been having some brute force attacks on our server over the past few months, and while our server mostly weathers them, we had one yesterday morning that required a manual restart of the virtual machine where our system is housed. We were back online in no time and all looked good. I then had a pretty busy day, and wasn’t spending much time on the system, but got no notices that there were problems (usually I’ll hear from users if something’s amiss).
Towards the end of the day, on either side of my commute, I saw the following symptoms emerge:
I noticed that it was taking a long time to access my dashboard even though the site load was normal. I was though still able to eventually get in.
On my way out of the office, I noticed that the BuddyPress admin bar was visible even though I had set it to be hidden for logged out users. I noted to take a look at this when I got home. When I did, I found that I couldn’t change that or any other BP setting… when I clicked “save,” the setting reverted to “not hidden.”
While going back and forth between the front and back ends, I suddenly lost the super admin capability on my personal account, though I was still able to access the super admin menu with the “admin” user.
I then noticed several plugins — including BuddyPress, WP-Super Cache — had been suddenly deactivated, and I couldn’t reactivate them.
Then, I noticed that no network themes were available in the Appearance>Themes menu, and that the “enabled” setting on all themes on the wp-admin/ms-themes.php page had been toggled to “no”
The home site went white, probably because BuddyPress was no longer enabled and that site runs a BP theme
All subdirectory sites seemed fine and were accepting posts.
I’ve never seen this collection of problems before, and began to worry that perhaps the system got hacked…although the pattern of symptoms suggested something was interfering with the database. I first deleted all plugins (except the shar-db plugin), but doing so changed nothing.
Then, I thought, if this is really bad, I need to make sure I have a current MYSQL dump. When running the dump, I got this error:
mysqldump: Got error: 145: Table ‘./wpmu_global/wp_sitemeta’ is marked as crashed and should be repaired when using LOCK TABLES
Ah… crashed database table! I repaired that table via PhpMyAdmin, and, voila, problem resolved.
It seems as though the hard reset that the system required in the morning corrupted the database, and led to the gradual emergence of these symptoms.
Live, learn, and make sure you check your database.
This past Spring I was pleased to moderate a panel at the Baruch Teaching with Technology Conference featuring three of Baruch’s most accomplished blogfessors: Mikhail Gershovich, whose Fear, Anxiety, and Paranoia course site made wide-ranging use of Blogs@Baruch; Paula Berggren, who’s done some of the most focused and interesting work on the system; and Zoe Sheehan Saldana, who’s a two-time reigning Blogfessor of the Year.
The session was well-attended and full of energy, and I think we touched on most if not all of the issues implicated in administering an online publishing platform at the College including pedagogy, resources, administration, and learning outcomes. BCTC was generous enough to record audio of the presentation and to post it to iTunes U, and it’s available below for your listening pleasure. For those of you who wonder what Blogs@Baruch is all about or just what it is I do around here, the audio below should answer some of your questions.
If you’d like to download this to your portable device for mobile edification, you can get the file here (if I link Cacophony will turn the link into an audio player): http://cac.ophony.org/audio/teachingwblogs.mp3.
I’m preparing to roll BuddyPress out on Blogs@Baruch later this month, and I’ve grown a little concerned about the implications of doing so. I thought I’d write up some of my concerns and see if the Internets has anything wise to say about them.
Our goal in using BuddyPress is to try to draw out and congeal an academic publishing network out of the various work that’s being done across the system. We hope to give students a platform to track their work over their careers at the College, to make connections with students with similar interests, and to cultivate a profile in a space they’re somewhat familiar with that we can support and that they can build as they desire. But I’m anxious about a few things.
First, we already have more than four thousand users on Blogs@Baruch, and the vast majority of those accounts were created for course-based blogging. I’m uneasy about turning on profile pages for users who never used the system for that purpose, without their knowledge. My current plan is to send an email out to all users when we turn on BP with instructions about granular control of profile pages. But, as far as I know, that control can only be so granular: with BuddyPress Profile Privacy you can set privacy on a field-by-field basis, but you can’t lock a whole profile page down. I’m hoping Jeff Sayre’s Privacy Component, which apparently is nearing a second beta, can help solve this problem. We’ll be registering incoming first year students for Freshman Seminar and instructing them on how to use the system beginning in August, and we’ll keep Profile pages set to “open” for new users from that point forward (we’ll be updating our woeful Terms of Service as well). I think it might make sense though to lock-down already existing accounts and outreach to those users with details about BuddyPress’s purpose and instructions on how to manage their profile privacy. I’m uncertain about this, though, both the ethics and how I’d manage this technically.
Second, I’d like for the primary engine of Blogs@Baruch to continue to be course-based blogging. BuddyPress, however, elevates the social networking function to equivalence with the blogging functionality of a WP-MS installation. We’re not building ePortfolios like our friends at Macaulay and don’t have the resources to closely support the development of profiles on a system as big as ours. And I certainly want to avoid the creepy treehouse factor, which is an issue with incoming Freshman. I just want students to use BuddyPress@Baruch to connect with each other around interests and academic work. So there are a few spots where I’d like to make some choices or changes that could nurture that understanding; for instance, I don’t think I’ll have a link to the members directory from the front page (but have it publicly accessible via internal links); I’ll hide the BuddyPress admin bar for logged out users; and, I’d like to hack BuddyPress so that upon log in, instead of landing at the front page of the home blog, users land at the Dashboard for their primary blog. Any other ideas?
Third, I have to revisit our registration process. In most classes, we use DDImportUsers to bulk register new users. Our most technologically capable faculty members can handle the intimidating two-step of a “self-registration” and the addition of Andre Malan’s “Add User to Blog” widget. Now, with BuddyPress functionality turned on, registration can become more complicated and require more information, which is fine for self-registering users but potentially problematic for those who are bulk-added. The bulk process also only creates new accounts, which I’ve been struggling with for some time; existing users need to be added to new sites individually, and to do so you need both a username and an email address (if I had my druthers, the DDImportUsers plugin would be able to check a list of newusernames|newemailaddress against the user_email field in the wp_users table and if a email address exists, add the user with that address to the individual site… and then to go on to register all the new users).
As the system grows, this is becoming a bigger problem since every semester a higher percentage of Baruch students have accounts on the system and find their way into new classes that use it. In an older version of WPMu you were able to add users to individual blogs simply with an email address, which was preferable because the cross-referencing is a pain. But that pain is balanced on the other side by the agita that would be caused if nervous first-time blogfessors are made to manage a multi-step registration process. In the past, I’ve taken the pain on in exchange for the benefit of drawing more users onto the system, and it’s been a good trade. I’m not sure yet how BuddyPress fits into this equation and how it will impact my overarching goal of easing the registration process, but wanted to get the issue out there. In the long term we’re looking at LDAP integration, but we’re not there yet. One solution is BP Group Blogs; but that creates additional steps in the registration process and we still want to make things as sleek and streamlined as possible.
These are my concerns for now, and I’m sure there’ll be more to come… any feedback, questions, and exchanges from out there in the wild are welcome and greatly appreciated.
For the second straight year, we’re awarding the Blogfessor of the Year Award to Zoe Sheehan Saldana, of Baruch’s Fine and Performing Arts Department. The award comes with priority support from the Schwartz Institute on all online publishing endeavors. Of course, Zoe already has that because she’s so awesome.
Zoe developed three sites on Blogs@Baruch this academic year. Last Fall, she did a Do-it-Yourself Publishing site that used FeedWordPress to syndicate nineteen individual journals where students documented making their own books from scratch (some digital, some not).
This Spring, she used a site in her Basic Graphic Communication course… here’s a description of her course and how she used her course blog from her “About” page:
This course introduces the graphic design process and methodology. Conceptual and creative thinking is stressed and understood through problem-solving assignments based on research, readings, and classroom demonstrations. The student is introduced to graphic design principles and exposed to historical and contemporary models and current standards of advertising and design. The Macintosh computer is included as the primary graphic design environment. This class is a prerequisite for all advanced Graphic Communication courses. Complete course guide available here, as a PDF file.
This blog is a venue for presenting, exploring, and discussing work, ideas, and topics pertaining to the course.
And, finally, together we developed a site for the Focus on Photography Exhibit which served initially as a processing space for members of the Baruch community to submit photos that they wished to be considered for a physical exhibit (which opened last week at the Mishkin Gallery). The site’s since evolved into an online companion displaying close to 200 images submitted by Baruch students, faculty, and staff. The submissions process used the TDO Mini Forms plugin to collect information from applicants, allow them to upload their images, and then it published those images to password protected pages where the exhibit judges could asses them. After decisions had been made about which images were accepted for the physical exhibit and which were not, Zoe hacked the Monotone WordPress theme (ideal for photo blogging) to create the online exhibit, which will live beyond the one at Miskhin. The amazing photographic ability of Baruch folks is a topic for another post, but I encourage you to take your time and click through the exhibit to see the fantastic images these folks have captured.
What’s so great about Zoe, beyond her gracious personality and charm, is that she’s exactly what an educational technologist like me needs to get better at what I do: someone who asks questions that I don’t know the answers to, patiently awaits the answer, and works to arrive at a consensus around what can be done with the tools, time, and resources available. She’s a great collaborator and a creative teacher. And, as she showed in talks she gave at last year’s CUNY WordCampEd and this year at the Baruch Teaching and Technology Conference, she has a strong grasp of the pedagogical, political, and philosophical impulse behind what we’re trying to do with educational technology at the Schwartz Institute. As her course blogs and her own art show, she’s an O.E.: Original Edupunk, and both Baruch and the Schwartz Institute are lucky to have her around.
Several units at Baruch College, including the Schwartz Institute, are planning an initiative for the next two academic years: Performing Diasporas: Identities in Motion. The broad goal of the project is to raise the profile of the Baruch Performing Arts Center while more deeply integrating the performing arts into the curriculum and the life of the College. We are finalists for a Creative Campus Grant, a competition funded by the Doris Duke Foundation, and organized by the Association of Performing Arts Presenters. The project will proceed even if we don’t get the grant (winners will be announced in August), although the programming will be more robust with the additional resources.
Performing Diasporas is centered around artists-in-residence — in 2010-2011, Maya Lilly; in 2011-2012, Randy Weston; and, both years, Mahayana Landowne — each of whom’s work engages questions of group and individual identity formation. These artists will perform throughout their residencies, and also lead and participate in workshops. Much of the programming, however, will be directed at incoming students. The first year experience for the next two years will revolve in large part around exploration of the project theme: the Freshman Text will be about diasporic identity, the artists-in-residence will perform at August’s Convocation, and significant components of Freshman Seminar and the curricula of selected Learning Communities will be devoted to the theme.
As part of the Steering Committee planning this project, I’m especially excited by a few particulars. Too often the administrative labor of higher education falls into silos whose work is narrowly focused and lacks programmatic coordination with other initiatives at the College. This project is structured to counter that impulse by drawing several partners into a collaborative effort to inject consideration of both the arts and the themes of identity and diaspora into the curriculum. Obviously, this will most directly impact our first year students. But it’s also good for everyone at the College for the various moving administrative parts to find synergies. The project will raise the profile of BPAC, inject the first year experience with a variety of new ideas, and dovetails nicely with Dean Jeff Peck’s Global Studies Initiative.
The project also will also help lead Blogs@Baruch into its next phase. Last Fall, we began supporting Freshman Seminar. 1200 first year students wrote more than 6500 blog posts to 60 weblogs, all of which were aggregated ultimately into a single space. FRO Blogging was a success, if solely because we were able to pull it off with little time to plan. Feedback from last Fall’s students and the Peer Mentors who led the seminars suggested the desire for more creative leeway and fewer required blog posts (students were expected to author at least six reflections on enrichment workshops they attended over the course of the term). The feedback also showed appreciation for the social component of the project; students used their blogging to get to know each other and to form community, something that’s always a challenge at a commuter campus like Baruch.
We’ve redesigned FRO Blogging to incorporate this feedback and to intersect with the goals of Performing Diasporas. There will be three specific components to FRO Blogging in Fall 2010:
Students will be required to write blog posts at the beginning and end of the semester reflecting on their adjustment to college and, in the middle of the semester, will post monologues about their own backgrounds that they develop with their Peer Mentors (who will receive training). Selected monologues will be shaped and then performed by professional actors at an end-of-the-semester event: “Baruch’s Voices.” In Spring 2011, students who are interested in performing their own monologues will workshop them and then perform at a series of Coffee Houses.
Each seminar will be asked to develop its blog over the course of the Fall semester. We will push this process along by crafting prompts that are distributed weekly and that encourage students to reflect upon and share their own stories. Peer Mentors will guide the process, with assistance, and students will be nudged, but not required. At the end of the semester, the most fully developed sites will be recognized with an award. This is an experiment in voluntary buy-in, and we realize that student investment of effort will be uneven. Yet, the constraints of a non-credit course make this approach necessary, and the goal is less to have students develop polished public spaces than to get their feet wet thinking critically about how to present artistic and intellectual material on the open web.
Finally, I’m excited to note that we’ll be rolling out BuddyPress this Fall, which will add a social networking layer to Blogs@Baruch, and afford students additional opportunities to connect with and get to know one another.
Ultimately, what I like most about this project is that it treats our students as creators and makers of knowledge, not merely as consumers. Baruch students are among the most interesting students in the world, and yet few of them seem to realize this (in fact, that’s one of the things that makes them interesting). Performing Diasporas, because it will draw our students inside productive processes and creates multiple opportunities for them to see and share the art in their own lives, is going to be something special to watch.
The Baruch College community has begun to see Blogs@Baruch not just as a blogging platform or substitute course management system, but also as powerful tool for meeting a wide range of self-publishing needs.
A variety of constituencies at the College have begun using the system for a range of internal and external communication. We have some fantastic librarians at the Newman Library, and they’re using Blogs@Baruch for a Reference Blog, an Idea Lab, and a Graduate Research Blog. They’ve also begun using CommentPress to discuss a Library Planning document. The Institute shares many interests and goals with the College’s librarians, and we have so much to learn from them. I’m particularly interested in collaborating with them to explore the role of technology and self-publishing in cultivating digital literacies among our students. This semester’s conversations were a great start.
The Baruch College Honors Program has begun using Blogs@Baruch this semester for a number of projects. They’re now hosting their homepage on the site, taking advantage of WordPress’ elegant content management features, and offering the staff an easy way to stay in contact with students (current and prospective). Also, first year Baruch Scholars have been given their own blogs to cultivate over their careers here, and their posts aggregate here. This is envisioned as a kind-of low stakes eportfolio project: give the students the space, and encourage (but don’t require) them to explore it. Another interesting Honors publishing initiative is the Change For Kids blog, where students working as reading tutors in a number of New York City elementary schools are blogging about their experiences, taking advantage of the opportunity to collaboratively reflect on and work through the challenges of working with children. Kudos to the Baruch Honors Program!
Frank Fletcher, the Executive Director of the Graduate Programs at the Zicklin School of Business, has been spearheading the business school’s move towards self-publishing. Frank has been encouraging his colleagues in Zicklin to explore a variety of initiatives on Blogs@Baruch over the past six months, and is now publishing to Lexington 24:25, where he’ll highlights developments in the MBA program and “identify emerging needs and trends in management education.” We look forward to supporting Zicklin, particularly in their efforts to connect Baruch students with potential employers and alumni.
Three journals are now hosted on Blogs@Baruch. Lexington Universal Circuit: A Journal of Economics and Politics is edited and authored by Baruch undergrads, launched last month (see details here), and we look forward to seeing that project continue to evolve. Dollars & Sense, which used to publish the selected journalism of Baruch students once a year as a beautiful (but costly to produce) magazine, now publishes on a rolling basis, for free, using Blogs@Baruch. While I myself miss the bound hard copy version, and see this transition as a microcosm of the larger troubles facing journalism, I’m happy that the faculty members who oversee the project– Josh Mills and Andrea Gabor– see the opportunities that are made available by self-publishing. For instance, student work produced in the fall doesn’t need to wait until the spring for publication; a wider range of work can be featured; and it’s now easier to share the work of our students with a much broader audience. Finally, iMagazine, the journal of student writing overseen by the Baruch College Writing Center, is in the process of migrating to Blogs@Baruch; stay tuned for a launch early next calendar year at this url.
These are just a few of the most exciting non course-based uses of Blogs@Baruch; there are others in the planning stages that promise to take advantage of the power of this publishing platform to create unique opportunities for members of the Baruch community to interact with each other and audiences beyond the campus. One is our plan to support selected student bloggers who’ll be tasked with chronicling their lives at the College for a broader audience. I’ve often said that we have the most interesting students in the world, but few of them know just how interesting they are. Blogs@Baruch, by providing multiple paths into the work our students and faculty are doing, makes the case more powerfully than I ever could.
Blogs@Baruch was used in approximately two dozen courses this semester, in disciplines that included Fine and Performing Arts, English, Sociology/Anthropology, Journalism, Library Information Systems, Communication, History, and Management.
One project in particular embodied the excitement some faculty members and students bring to their work on Blogs@Baruch. Professor Shelly Eversley, in the English Department, had her American Literature students produce pod and vodcasts that analyzed texts they had encountered over the course of the semester. Buoyed by Cogdog’s “The Fifty Tools”, I did an hour in class on free digital story telling tools (including Voice Thread, Yodio, Gabcast, and Podcast People), and also gave some advice on how to construct a story that balanced narrative, analysis, and style. The students produced amazing work, which they collected here in advance of their voting for the initial American Literature Podcast Awards (the ALPs). They ended the semester with an awards ceremony, and have continued to post their thoughts about the class to the blog in the week since.
Here’s two of my favorite videos from the class:
Prof. Eversley’s project exemplifies the useful energy that multimedia tools can help students invest in their coursework. These projects are not substitutes for the critical engagement with a text or a canon that some might argue can only be attained through writing an essay; rather, they are additional paths towards that engagement. These students were excited about showing off their work, used the city as a laboratory and an archive, helped each other master the technology, and showed deep engagement with their chosen texts. This is good teaching and learning, and we’re happy to support any faculty member who challenges herself and her students to use a variety of tools and literacies in their effort to produce knowledge.
Kudos to all of our intrepid faculty and their students for providing us with yet more examples of innovative pedagogy on Blogs@Baruch. We look forward to Spring 2010, and in particular two film courses that will be taught on the system. Blogfessors, come on down!
Approximately 1200 incoming first year students at Baruch participated in the first phase of our experimental integration of Blogs@Baruch into the Freshman Orientation Seminar. They wrote to blogs in approximately sixty individual sections, and their posts were syndicated on the FRO Motherblog.
As I noted a couple of months ago, we had severe constraints in launching this project, so we focused primarily on the technological implications of getting it off the ground. We didn’t have sufficient time to either develop a well thought-out curriculum or to work with the Peer Mentors who oversaw the sections to help them pedagogically manage the work of their students. We might have had we gone with a pilot project, but for various reasons that suggestion was scuttled, and we proceeded full-bore.
These caveats aside, I think the project was a resounding success. It’s generated a staggering amount of data and also some important questions for us to address, and also helped us see what’s possible with more thoughtful design and oversight.
More than 6200 posts have been authored by first year students and aggregated into a single space. The vast majority of these posts are student reactions to a variety of “Enrichment Workshops” that they were required to attend. As you might imagine, many of the posts are more descriptive than analytical, and some come across as check boxes to be completed on the way to a requirement. The best posts, however, evidence deep and enthusiastic engagement with the workshops or with other elements of transitioning to life at Baruch.
We’ve already begun to discuss with our colleagues Mark Spergel and Shadia Sachedina how we can encourage posts that students are excited to write and also to read and comment upon. We plan to come up with a range of models and prompts that students can choose from that intersect with some of our broader goals for the project: cultivating digital literacy in our students (I plan to talk and think more with Boone Gorges about this), easing their social and intellectual transition to college, and helping them more nimbly and thoughtfully integrate social media into academic work. I envision a series of assignments that build towards these curricular goals, while also generating the kind of shared reflection that our colleagues in Student Life want to see. I also think we have the great opportunity to show off what interesting lives our students lead. This is a unique institution, and blogging in Freshman Seminar can show the world just what Baruch College and CUNY are about.
The Peer Mentors are key to this improved design. We’ll expand the training that they get so they’re better prepared to guide their charges. Next semester, four sections of Freshman Seminar are running, so we finally get to run that pilot project we originally envisioned, though with the implications of scaling the thing up already known. In the summer we’ll likely do some outreach directly to incoming students before school starts so that they are aware of this component of Freshman Seminar, and can hit the ground blogging.
As we plan a new design, we’re trying to figure out how we’re going to make sense of all of the data we’ve collected. It’s difficult, though not impossible, to design an assessment of data that’s been collected without assessment forefront in mind. Ryan Androsiglio, a psychologist in the Baruch Counseling Center, is helping us look at the project to see what questions can reasonably be asked of it.
We were able to perform a much less formal assessment of the program by soliciting feedback from Peer Mentors and First Year Students themselves. Both groups were between lukewarm and mildly-positive in their feedback, and each desired more leeway in what was blogged about and how. The Peer Mentors I spoke with were quite clear that the strongest component of the project was the social cohesion it encouraged among the students in their seminars.
For a commuter campus like Baruch, FRO blogging has become a powerful tool simply because it creates more opportunities to interact. To encourage this, we’re seriously considering integrating BuddyPress into FRO 2010.
The social benefits of FRO blogging are already crystal clear; we now need to work on defining reasonable curricular goals, and a plan to implement them.
We’re winding down another eventful semester on Blogs@Baruch, and over the next few days I’d like to offer some reflections about where we’ve been and where we’re going. Our usership has tripled, and we’ve also expanded to serve a much broader range of constituencies at the college. This broadening and deepening has taught me much about the opportunities and challenges of supporting Baruch’s use of this powerful open source publishing platform.
Mikhail Gershovich accepts the Mike Ribaudo Award at the 8th Annual CUNY IT Conference
Two events over the last ten days drew into sharp focus what we have accomplished and also some of the challenges we face. At the 8th Annual CUNY IT Conference, the Schwartz Institute was awarded the Michael Ribaudo Award for Innovation in Technology. Mikhail, Suzanne, Tom, and I were recognized along with administrative teams from John Jay and the CUNY First project, as well as our good friend Matt Gold, Project Director for the CUNY Academic Commons. The Commons is like a sister project to Blogs@Baruch, since we’re using the same software, and we share ideas, labor, and a philosophy about what support for technology at the university level should entail.
It was an honor to be recognized for our innovations and, especially, to share the honor with Matt, since it signaled to the broader CUNY community that the work we’re undertaking is not only viable, but forward-looking and vital to the work of the University. At the risk of sounding like an ingrate, though, I noted that the certificates we received read that this was an “Information Technology” award. I’ve made the point before, and will make it again: instructional technology is not information technology. This is actually acknowledged in how the Ribaudo is awarded, as it’s split between the two areas (even if the split is not represented on the certificate). This is more than a semantic argument: we need to encourage our communities to understand the differences and to constantly reexamine how the University’s information technology architecture relates to and interacts with the deployment of technology in the service of teaching, learning, and scholarship.
It’s always nice to get an award, and last week brought hearty congratulations from inside and outside the Baruch community. In the midst of these pats on the back, however, I learned a little bit more about the difference between information technology and instructional technology. At approximately 7pm on Wednesday evening I happened to look at one of our blogs, and saw the dreaded:
The error appeared on all subdirectory blogs, while the main blog was completely white. I logged into the command line, verified that MYSQL was running, and saw that the load on our server was fine. The documentation I was able to find suggested either a MYSQL problem or a plugin conflict; I deleted all plugins, with no improvement. Now, instead of the “Error Establishing a Database Connection” I was getting what geeks refer to as the “White Screen of Death” across the entire installation. Having exhausted pretty much the extent of my command line knowledge, I sent out emails to our contacts at BCTC, and waited for a response.
A couple hours later, I was contacted by a sysadmin at BCTC; he had gamely returned to work on his way home from the gym to take a look at our server. He immediately noticed that the directory that holds Blogs@Baruch was about 98% full. We knew that we were approaching space limits, but I had (mis)calculated that we could make it to the end of the semester (when we’ll be moving the entire installation over to a new server). I was puzzled, however, because we had this issue once before and it didn’t cause an outage– it just caused an error in our database backups that resolved as soon as we opened up space. I hoped opening space would clear up our problem, but it did not.
We both thought that the database needed to be repaired, but neither of us were comfortable issuing the repair commands. The admin at BCTC contacted MYSQL, and got assistance repairing and then restarting MYSQL. 1 am, no improvement. We’d have to wait until morning.
At 6 am I took another look at the server to see if I had missed anything, and began to respond to users who were emailing about the site. I posted a query to our premium support forum with Automattic describing the problem, and got a quick response from Donncha, the lead developer of WPMu. Unfortunately, my question included a distracting error that I found in the log that was caused by a bad Phpinfo file I had put on our server (in my haste I wrote the file in Text Edit at home, which put additional characters into the file that I wasn’t able to see). Donncha thought we might have been hacked, and asked me to check our .htaccess files, which looked ok. I caught my mistake, and explained it (along with a note apologizing for not being a system administrator). Apparently I wasn’t clear, because Donncha kept pursuing the PHP error… we weren’t communicating well. He suggested I use error_log() to track down where the PHP problem was.
In the meantime, emails and phone calls from users were flowing in, and I did my best to explain to as many as possible that we were investigating the problem and should be live again soon. Internally, though, I wasn’t so sure; we had exhausted our knowledge and the knowledge in the free forums, and the premium forum to which I was posting wasn’t yielding results. Jim Groom suggested we contact Ron and Andrea Rennick, who I refer to as the “WPMu Wonder Couple,” to see if they might be able to help us out.
Within 3 hrs of Jim’s suggestion, BCTC had vetted Ron and granted him temporary access to our server; he located and fixed the problem in about 20 minutes. In the meantime, Barry Abrahamson, who runs the servers for WordPress.com and also posts to the premium support forum, had offered to do the same.
Turns out the problem was one that I had caused while trying to fix the space issue. When I deleted the plugins in mu-plugins, I failed to delete the Supercache file that sits outside of the plugins folder, inside of wp-content. I also deleted the existing cached pages. Ron concluded that:
Once you ran out of disk space, pages expiring in supercache were being refreshed as empty files. Eventually nearly all of your pages were cached as empty files. I disabled supercache by renaming advanced-cache.php in wp-content. MU checks for the file and includes it in the processing if it exists.
He later added:
I did some testing locally and reproduced the white screen by deleting the contents of the cached version of the index.
Here’s the rub: we got through it. Ultimately this was two small problems masquerading as a big one. We ran out of space, then I failed to properly disable a powerful plugin running on our system, which disabled the entire install. We were down less than 20hrs, and that was only because I wasn’t systematic enough to pick up on the way Supercache works. To a certain extent, something like this was inevitable. All sites go down, even the Big G. It’s the risk you run when you work online, and reasonable end users can accept it– it helps if those running the site aspire towards transparency.
The outage confirmed my belief in open source applications, and particularly the communal ethos that (often) animates them. Three friends: Boone Gorges, Jim, and Zach Davis, offered assistance as soon as they learned of the problem, and moral support because they’ve each been in similar situations. The offers of hands-on help were reassuring, but I didn’t really need them because I was already in contact with the three most knowledgeable WPMu people in the world.
The outage also reminded me that being able to type stuff at the command line and get stuff in return does not make one a system administrator. I’m a humble educational technologist, and I depend on information technology to get my work done. When the lines are blurred– and I blurred them here more out of necessity than conceit– trouble may ensue. Had I been able to look holistically at the problem and troubleshoot it methodically, I probably could have caught the error. But inexperience and the pressure of supporting 3k+ users clouded my vision and convinced me the solution to the problem was out of my reach. These are valuable lessons to carry forward on this project.
Within an hour of Blogs@Baruch going backup, Baruch College’s enews arrived in my mailbox, containing a congratulations to the Institute on the Ribaudo Award. I clicked on a link and landed happily at our pretty little homepage, which was humming nicely along. When I closed my laptop, I still managed to feel pretty good about the week.
PS: I’ve learned that the following cultural artifact can help one oversee an enterprise publishing platform:
This semester, we’re managing our largest lift on Blogs@Baruch yet. In addition to an increasing variety of projects that I’ll blog about in the coming weeks, every Freshman Seminar at Baruch currently is blogging. That’s roughly 60 sections, populated by over 1200 students.
Baruch Freshmen at Convocation, September 2009. Click to see photo in its original location.
Each Seminar is directed by a Peer Mentor, a talented upper level Baruch student responsible for helping newcomers adjust to life at Baruch. The seminars meet every other week, and Freshpersons are required to attend lectures, panels, exhibits, seminars, and trainings, distributed across six “enrichment” areas over the course of the term. Then they’re supposed to blog about their experiences, and discuss them when they meet with their classmates.
Launching the project was a bit of bear, as we had to create the blogs, get the users registered, tie the whole deal together, and give some training to the Peer Mentors, who are crucial to the project. Ultimately, I created a custom theme (built on Carrington Blog), with certain core components to which each section would have access– a List of Seminars and Peer Mentors, a Guide to Blogging for Freshmen (produced by the Office of Student Affairs, who directs FRO), a description of the six enrichment areas, and a Google Calendar that displays upcoming events. I then created a Mother Blog, which syndicates posts from across the sixty sections of FRO, using the FeedWordPress plugin. The Mother Blog collects and stores all of the posts in one place, allowing faculty and administrators to look in on the writing that’s happening in FRO. Students are thus contributing to small discussions in their seminars, and also to a broader discussion among all Freshmen.
Thus far, they’ve been writing quite willingly. In the fewer than three weeks since this thing was launched, we’ve aggregated about 900 posts; at the pace we’re going, we should reach well more than 4000 unique posts by the end of the semester. That doesn’t even begin to address the commenting, which has varied in intensity across the individual blogs. Unfortunately, we do not have the ability to mirror comments between the original location of the post and the space where it is republished… if we did, and we hope to be able to do that soon, the level of dynamism would increase.
Needless to say, we’re looking at an awful lot of writing, and we’re trying to make sense of it in a few ways. We’ve created categories on the Mother Blog for each of the six enrichment areas so that posts directly pertaining to them can be easily sorted. This will allow the two administrators who oversee FRO– Mark Spergel, the Director of Student Orientation and Freshman Year Incentive, and Shadia Sachedina, the Associate Director of Student Life– to get student perspectives on the wide range of extra-curricular programs the school offers. Further, simple searches will allow certain segments of the Baruch community to see what students are saying about them. For instance, many of the early posts offered student perspective on tours of the library. Our librarians have already begun searching for “library” and “library tour” on the FRO blog to read student responses. Several blog posts have engaged Sherman Alexie’s Reservation Blues, the Freshman text.
Other searches hold the potential to help identify students with like interests: “photography,” “history,” and “football” all offer returns. Such a use of the FRO Mother Blog suggests another function that this project can play, perhaps more effectively in future iterations: social networking. As a commuter campus, we constantly struggle to help our students see themselves as part of a community, and FRO attempts to address that tension. Integrating Blogs@Baruch into FRO makes that attempt much stronger, as students can more easily find, connect, and engage with their classmates through our platform. Next year, I’d love to get BuddyPress working in this project to foreground the social networking component… but, one step at a time.
At the end of the term, we’ll have, easily collected and archived, multiple writing samples from the majority of incoming students. With some more thinking and organization, this holds great potential for assessment, integration into writing instruction, early intervention, and assistance for ESL students. Ultimately, this project allows us the opportunity to further the core missions of Blogs@Baruch: increasing the amount and variety of writing that our students do, and nurturing critical thinking about the use of digital tools throughout the Baruch College community. Given the hectic nature of our launch this year, we weren’t able to spend enough time thinking collectively about the general education opportunities embedded in this project. I had argued that we should do a pilot with 20% of the sections so that we could be sure to more closely support our users and think more intensively about the implications of what we’re doing, but for various reasons, a small-scale pilot wasn’t feasible. But when we do this again, we know that the canvas works, what the challenges are in the mechanics of the thing, and how to improve our planning. We’ll be able to make a more significant investment in helping the Peer Mentors better understand the possibilities and implications of doing college work on the open web, crucial knowledge that they can then pass on to all Freshpersons.